Back from the Cloud and Datacenter Conference Germany 2017

I just got back from a very successful Cloud & Datacenter Conference Germany, 2017 edition. I took some vacation days to go educate myself with the help of my peers and the excellent speakers. The people have a wealth of expertise and experience in real world solutions that address the challenges we face today. All this without marketing or too ego. Just pure knowledge sharing about the facts and realties. Take a look here at the wealth of industry experts that were attending and presenting.

image

The attendees were amongst the better educated customers & partners you encounter in the field. This is great as this gives everybody good feedback and insight in the challenges we all face. I’d say there are definitely very pragmatic patterns in how businesses deal with the evolving cloud & datacenter ecosystem successfully. Cyber security is also part of that. I’m happy to see the insights shared with us by an expert lite Tudor Damian reminding us to always keep security in mind and showing us and that Microsoft is indeed making serious efforts to protect us in an IT environment they approach with the assumption that Ii t is compromised.

I did my part for the conference with a session on Failover Clustering Evolved (in Windows Server 2016) as well as with a presentation called SMB Direct – The Secret Decoder Ring for the Hyper-V Community day event the day before the conference. That was also awesome and I had a great response to the session and interest in our experiences with RDMA. Oh yes, I also got some hands on training in filming to the delight of my fellow MVPs Winking smile.

If it’s up to me, I’ll be back in 2018!

PowerShell Script to Load Balance DNS Server Search Order

Load Balance DNS Server Search Order

DNS servers need to be configured correctly, operate perfectly and respond as fast as possible to their clients. For some applications this is critical, but many have a more relaxed attitude. Hence a DNS Server has a full second to respond to a query. That means that even when you have 2 DNS servers configured on the clients the second will only be used when the first is not available or doesn’t respond quickly enough. This has a side effect which is that moving traffic away from an overloaded DNS servers isn’t that easy or optimal. We’ll look at when to use a PowerShell script to Load balance DNS server search order.

DHCP now and then

The trick here is to balance the possible DNS servers search order amongst the clients. We used to do this via split scopes and use different DNS servers search orders in each scope. When we got Windows server 2012(R2) we not only gained policies to take care of this but also DHCP failover with replica. That’s awesome as it relieves us of much of the tedious work of keeping track of maintaining split scopes and different options on all DCHP servers involved. For more information in using the MAC addresses and DCHP policies to load balance the use of your DNS servers read this TechNet article Load balancing DNS servers using DHCP Server Policies.

Fixed IP configurations

But what about servers with fixed IP addresses? Indeed, the dream world where we’ll see dynamically assigned IP configuration everywhere is a good one but perfection is not of this world. Fixed IP configurations are still very common and often for good reasons. Some turn to DCHP reservations to achieve this but many go for static IP configuration on the servers.

image

When that’s the case, our sys admins are told the DNS servers to use. Most of the time they’ll enter those in the same order over and over again, whether they do this manual or automated. So that means that the first and second DNS server in the search order are the same everywhere. No load balancing to be found. So potentially one DNS server is doing all the work and getting slower at it while the second or third DNS servers in the search order only help out when the first one is down or doesn’t respond quickly enough anymore. Not good. When you consider many (most?) used AD integrated DNS for their MSFT environments that’s even less good.

PowerShell Script to Load Balance DNS Server Search Order

That’s why when replacing DNS Servers or seeing response time issues on AD/DNS servers I balance the DNS server search order list. I do this based on their IP address its last octet. If that’s even, DNS Server A is the first in the search order and if not it’s DNS Server B that goes in first. That mixes them up pseudo random enough.

I use a PowerShell script for that nowadays instead of my age-old VBScript one. But recently I wanted to update it to no longer use WMI calls to get the job done. That’s the script I’m sharing here, or at least the core cons pet part of it, you’ll need to turn it into a module and parameterize if further to suit your needs. The main idea is here offering an alternative to WMI calls. Do note you’ll need PowerShell remoting enabled and configured and have the more recent Windows OS versions (Windows Server 2012 and up).

cls
#The transcipt provides a log to check what was found and what changed.
Start-Transcript -Path C:\SysAdmin\MyDNSUpdateLog.txt #
$VMsOnHost = (Get-VM -ComputerName MyHyperVHostorClusterName).Name

foreach ($VM in $VMsOnHost)
{
    Invoke-Command -ComputerName $VM -ScriptBlock {

    #This function checks if the last octet of an IP address is even or not
    Function IsLastOctetEven ($IPAddress)
        {
             #$FirstIP
             $Octets = $IPAddress.Split(".")
             #$Octets[3] #0 based array, grab 4th octet

             #See if 4th octect is even
             $Boolean = [bool]!($Octets[3]%2)
             if ($Boolean)
             {
                 Return $Boolean
                 #write-host "even"
             }
             else
             {
                 Return $Boolean
                 #write-host "odd"
             }
        }

        $OldDns1 = "10.15.200.10"
        $OldDns2 = "10.15.200.11"
        $NewDns1 = "10.18.50.110"
        $NewDns2 = "10.18.50.120"

        $NicInterfaces = Get-DnsClientServerAddress

        foreach ($NICinterface in $NicInterfaces)
        {
                #Here we filter out all interfaces that are not used for client/server connectivity.
                #Cluster Interfaces, HeartBeats, Loop back adapters, ...
                #We also filter out IPv6 here as this is for a IVp4 environment.
             if($NicInterface.InterfaceAlias -notmatch "isatap" -and $NicInterface.InterfaceAlias -notmatch "Pseudo" `
                -and $NicInterface.InterfaceAlias.Contains("Local Area Connection*") -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("KEMP-DSR-LOOPBACK") -ne $True `
                -and $NicInterface.InterfaceAlias.ToLower().Contains("Heartbeat".Tolower()) -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("NLB-PRIVATE") -ne $True-and $NicInterface.AddressFamily -ne "23")
             {

                $Output = "Hello from  $env:computername" + $NICinterface.InterfaceAlias
                write-Output $Output            
           
                $Output = $NicInterface.InterfaceAlias +": DNS1=" + $NicInterface.ServerAddresses.GetValue(0) + " & DNS2=" +  $NicInterface.ServerAddresses.GetValue(1)
                write-Output $Output

                If (($NicInterface.ServerAddresses.GetValue(0) -like $OldDns1 -or $NicInterface.ServerAddresses.getvalue(0) -like $OldDns2) -and ($NicInterface.ServerAddresses.getvalue(1) -like $oldDns1 -or $NicInterface.ServerAddresses.getvalue(1) -like $OldDns2))
                {
                    #If the IP address is DHCP assignd, leave it alone,
                    #that's handled via DHCP policies on the MAC address
                    $GetNetIPInfo = Get-NetIpAddress -InterfaceIndex  $NicInterface.InterfaceIndex
                     if ($GetNetIPInfo.PrefixOrigin -like "DHCP")
                     {
                        $VM                   
                        write-output "DHCP address - leave it alone"
                     }
                     Else
                     {
                         $IPAddresses = $GetNetIPInfo.IPv4Address
                         $FirstIP = $IPAddresses[1] #1 based array
                 
                         if (IsLastOctetEven($FirstIP)){
                            $VM
                            write-output "EVEN 4th IP octet => so DNS search order becomes $NewDns1 , $NewDns2"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns1,$NewDns2)
                         }
                         else
                         {   
                            $VM
                            write-Output "ODD 4th IP octet => so DNS search order becomes $NewDns2 , $NewDns1"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns2, $NewDns1)
                         } 
                         $NicInterface |  Select-Object -ExpandProperty ServerAddresses    
                     }
                }
                else
                {
                    $VM
                    write-Output "Existing DNS values not like expected old values. They are propably already changed"
                }        
            }
        }
    }
}
Stop-Transcript

Continuous available general purpose file shares & ReFSv3 provide high available backup targets

Introduction

In our previous two blog posts on Veeam and SMB 3 we’ve seen how and when Veeam Backup & Replication can leverage SMB Multichannel and SMB Direct. See Veeam Backup & Replication leverages SMB Multichannel and Veeam Backup & Replication Preferred Subnet & SMB Multichannel.The benefits of this are more bandwidth, high availability, better throughput and with RDMA low latency and CPU offload. What’s not to like, right? In a world where the compute and networks need keeps rising due to the storage capabilities (flash storage) pushing the limits this is all very welcome.

We have also seen earlier that Veeam B & R 9.5 leverages ReFSv3 in Windows Server 2016. This provides clear and present benefits in regards to space efficiencies and speed with many backup file related operations. Read Veeam Leads the way by leveraging ReFSv3 capabilities

When it comes to ReFSv3 in Windows Server 2016 most of the focus has gone to solutions based around Storage Spaces Direct (S2D). That’s a great solution and it is the poster child use case of these technologies.

But what other options do you have out there to build efficient and effective high available backup targets creatively except for S2D? What if you would like to repurpose existing hardware to build those? Let’s take a look together at how continuous available general purpose file shares & ReFSv3v3 provide high available backup targets

CSV, S2D, ReFSv3 & Archival Data

In Windows Server 2016, traditional shared storage (iSCSI, FC, Shared SAS, Shared RAID) with CSV are not recommend to be used with ReFSv3. Why isn’t exactly clear. The biggest impact you’ll see is the performance difference when not writing to the owner node of the CSV in this use case. Even with a well configured RDMA network that difference is significant. But that doesn’t mean that the performance is bad. It’s just that many of the super-fast meta data operations are relatively and significantly slower when compared each other, not that any of these two are slow.

clip_image002

Microsoft does state that an S2D with ReFSv3 and SOFS shares can be used for archival data. Storage spaces and ReFSv3 also have the benefit of offering automatic repair of corrupt data from a redundant copy on the fly even when needed. So yes, the best know supported scenario is this one.

Continuous available general purpose file shares and ReFSv3 provide high available backup targets

But what if we need a high available backup target and would love to leverage ReFSv3 with Veeam Backup & Replication 9.5? Well, you can have 95% of your cookie and eat it to. All this without ignoring the cautions offered.

We could set up SOFS shares on a Windows Server 2016 Cluster with ReFSv3 with traditional shared storage. Some storage vendors do state this is supported actually.

That only means you don’t have the auto repair functionality ReFSv3 combined with storage spaces offers. But perhaps you want to avoid the risk of using ReFSv3 with CSV in a non S2D scenario all together. What you could do is forgo ReFSv3 and use NTFS. How well this will work for archival data or backup is something you’ll need to test and find out how well this holds up. There is not much info is out there, only other cautions and warnings that might keep you up at night.

There is another scenario however and that is using Windows Server 2016 failover clustering to set up continuously available general purpose file shares that leverage SMB3 transparent failover.

The good news is that general purpose file shares (no CSV) do work consistently with ReFSv3 because such a share/LUN is only exposed on one cluster node at the time, the owner. By having multiple shares and setting preferred owners we can load balance the workload across all cluster nodes.

Thank to continuous availability for general file shares and SMB 3 transparent failover we can still get a high available backup target this way. The failover is fast enough to make this happen and all we see with Veeam Backup & Replication is a short pause in throughput before it resumes after failover. To put the icing on the cake, you can leverage SMB multichannel SMB Direct for both backup and restores.

I would take a sizeable whitepaper to walk through the setup so instead I’ll show you a a quick video of a POC we did in the lab here https://vimeo.com/212886392.

clip_image004

If you want to learn more come to the community & other conferences I’m speaking at and will be around for Ask The Experts time opportunities. I’ll be at the German Hyper-V community meet up, The Cloud & Datacenter Conference in Germany 2017, Dell EMC World 2017 and last but not least VeeamON 2017 (see  May 2017 will be a travelling month). 

Conclusion

What do you lose?

Potentially there is one big loss in regards to the capabilities of ReFSv3 with this solution when you are not using storage spaces. This is that you lose the capability to automatic repair of corrupt data. The ability of ReFSv3 to do so is tied into the redundant copies of Storage Spaces (parity/mirror).

What do you get?

That’s fine, the strength of this design is that you get the speed and space efficiencies of ReFSv3and high available backup targets in way more scenarios than “just” S2D. After all, not everyone is in a position to choose their storage fabric for backup targets green field or at will. But they might be able to leverage existing storage and opt to use SMB 3 for their data transport.

So even if you can’t have it all, you can still build very good solutions. It offers ReFSv3 benefits and high availability for your backup target via transparent failover with SMB transparent failover on continuous available general purpose file shares. This also only requires Windows Server 2016 Standard Edition, which is a cost saving. You get to leverage SMB Multichannel and SMB Direct. All this while not ignoring the cautions of using ReFSv3 in certain scenarios.

On top of that, if you use NTFS with this approach it will also work for Windows Server 2012 (R2) as the OS for the backup target cluster hosts.

Disclaimer

I do not work for or at Microsoft, nor am I perfect or infallible just because I’m an MVP. You’ll have to do your own testing and validation. From our testing and without ReFSv3 bugs ruining the show, to me this is a very valid and cost effective approach.

Tips on using Convert-VMGeneration.ps1 with Windows Server 2016

Introduction

Recently I was involved in getting a bunch of “holy cow” virtual machines updated/migrated to be future ready (shielded VMs, see Guarded fabric and shielded VMs overview).

That means they have to be on Windows 2012 R2 as the guest OS minimally .For us anyway, we’re not falling behind the curve OS wise. That’s the current legacy OS in the environment. Preferably they need to be at Windows Server 2016. This is has been taken care of and 40% of the virtual machines is already running Windows Server 2016 for the Guest OS, the remainder is at Windows Server 2012 R2 and those are moving to Windows Server 2016, where useful and possible, at a steady pace.,

When deploying new virtual machines the default is to use generation 2 virtual machines. Any remaining virtual machines that cannot be replaced need to be converted to generation 2. For that we routinely use the great script provided by Microsoft’s John Howard (see Hyper-V generation 2 virtual machines – part 10)  We’ll share some tips on using Convert-VMGeneration.ps1 with Windows Server 2016, which is an OS / Hyper-V version later than what the script was written for and tested against.

Tips on using Convert-VMGeneration.ps1 with Windows Server 2016

During the use of this script we came across a couple of new situations for us. One of those were Window Server 2016 virtual machines that are still generation 1 and reside on either a Windows Server 2012 (R2) or Windows Server 2016 host. Another were virtual machines with Windows Server 2012 R2 or Windows Server 2016 as a guest OS that already live on Windows Server 2016 and are still generation 1 and have either already been converted to or installed on a virtual machine version 8 or not (still at 5). All these can be death with successfully.

Situation 1

Running the script on a Windows Server 2016 Host. This throws an error reporting that the was only tested with PS version 4.

clip_image002

This is easily dealt with by using the -noPSVersionCheck switch, it even tells you to do so in the error message. I have found no issues in doing so.

.\Convert-VMGeneration.ps1 -VMName “MyVM” -path “C:\ClusterStorage\Volume1\ConvertedMyVM” -NoPSVersionCheck

Situation 2

Running the script against a generation 1 virtual machine with a Windows Server 2016 guest OS required a little adaptation of the script as it has an issue with detecting the guest OS version as supported. This is due to the fact that in the script the check is done against string values and they generate a logical “bug” when the doing.

clip_image004

Checking if a string of value 7 -lt 6 will evaluate correctly but doing the same with 10 doesn’t, that’s false. An error message is show that the “Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later”. Well is most certainly is, but the 10 in 10.0.14393.206 is not seen as greater or equal to six.

We fixed by converting the 1st and 2nd part (for good measure) of the OS version string to an integer before the check happens. That fixes it for us.

We’ll demonstrate this in a code snippet to run on a Windows Server 2016 host.

$SourceNTDLL = "C:\windows\system32\ntdll.dll"

$script:ProgressPoint = 651

$SourceOSVersion = ([System.Diagnostics.FileVersionInfo]::GetVersionInfo($SourceNTDLL).FileVersion)

$script:ProgressPoint = 652

$SourceProductName = ([System.Diagnostics.FileVersionInfo]::GetVersionInfo($SourceNTDLL).ProductName)

$SourceOSVersionParts = $SourceOSVersion.split(".")

if ($SourceOSVersionParts[0]-lt 6) { Write-Host "Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later." }

if (($SourceOSVersionParts[0] -eq 6) -and ($SourceOSVersionParts[1] -lt 2)) {Write-Host "Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later." }

This will give you the massage that the “Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later”. So, we cast the $SourceOSVersionParts[X] variables to an integer to overcome this.

$SourceNTDLL = "C:\windows\system32\ntdll.dll"

$script:ProgressPoint = 651

$SourceOSVersion = ([System.Diagnostics.FileVersionInfo]::GetVersionInfo($SourceNTDLL).FileVersion)

$script:ProgressPoint = 652

$SourceProductName = ([System.Diagnostics.FileVersionInfo]::GetVersionInfo($SourceNTDLL).ProductName)

$SourceOSVersionParts = $SourceOSVersion.split(".")

#Cast the OS version parts to an integer

$OSVersionPart1 =[INT]$SourceOSVersionParts[0]

$OSVersionPart2 =[INT]$SourceOSVersionParts[1]

if ($OSVersionPart1 -lt 6) { Write-Host -ForegroundColor Green "Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later." }

if (($OSVersionPart1 -eq 6) -and ($OSVersionPart2 -lt 2)) { CleanUp "Source OS must be version 6.2 (Windows 8/Windows Server 2012) or later." }

Do this and it evaluates correctly now so your script will run. That’s the only adaption we had to make in the script to make it run with a Windows Server 2016 guest OS.

Situation 3

My virtual machine is already a version 8 VM but still a generation 1 virtual machine. That’s not a problem at all. As long as you deal with situation 1 and 2, it will convert correctly.

Conclusion

If you’re prepping legacy virtual machines that need to be moved into a modern private cloud or on premises deployment you might need to convert them to generation 2 in order to take full advantage of the capabilities of the current Hyper-V platform (i.e. Shielded VMs). To do so you’ll be fine as long as they are running Windows Server 2012 (R2) as a guest OS on a Windows 2012 R2 host. If not, some creativity is all you need to get things going. Upgrade the guest OS if needed and fix the script if you encounter the situations as we described above. Sure, we have to herd virtual machines as cattle and avoiding holy cows VMs is important. But they do still exist and if they provide valuable services and we can’t let this hold us back from moving ahead. By proceeding like we did we prevented just that and avoided upsetting too many processes and people in the existing situation, let alone hindering them in the execution of their job. We still arrived at a situation where the virtual machines can be hosted as shielded virtual machines. Good luck!