Back from the Cloud and Datacenter Conference Germany 2017

I just got back from a very successful Cloud & Datacenter Conference Germany, 2017 edition. I took some vacation days to go educate myself with the help of my peers and the excellent speakers. The people have a wealth of expertise and experience in real world solutions that address the challenges we face today. All this without marketing or too ego. Just pure knowledge sharing about the facts and realties. Take a look here at the wealth of industry experts that were attending and presenting.

image

The attendees were amongst the better educated customers & partners you encounter in the field. This is great as this gives everybody good feedback and insight in the challenges we all face. I’d say there are definitely very pragmatic patterns in how businesses deal with the evolving cloud & datacenter ecosystem successfully. Cyber security is also part of that. I’m happy to see the insights shared with us by an expert lite Tudor Damian reminding us to always keep security in mind and showing us and that Microsoft is indeed making serious efforts to protect us in an IT environment they approach with the assumption that Ii t is compromised.

I did my part for the conference with a session on Failover Clustering Evolved (in Windows Server 2016) as well as with a presentation called SMB Direct – The Secret Decoder Ring for the Hyper-V Community day event the day before the conference. That was also awesome and I had a great response to the session and interest in our experiences with RDMA. Oh yes, I also got some hands on training in filming to the delight of my fellow MVPs Winking smile.

If it’s up to me, I’ll be back in 2018!

PowerShell Script to Load Balance DNS Server Search Order

Load Balance DNS Server Search Order

DNS servers need to be configured correctly, operate perfectly and respond as fast as possible to their clients. For some applications this is critical, but many have a more relaxed attitude. Hence a DNS Server has a full second to respond to a query. That means that even when you have 2 DNS servers configured on the clients the second will only be used when the first is not available or doesn’t respond quickly enough. This has a side effect which is that moving traffic away from an overloaded DNS servers isn’t that easy or optimal. We’ll look at when to use a PowerShell script to Load balance DNS server search order.

DHCP now and then

The trick here is to balance the possible DNS servers search order amongst the clients. We used to do this via split scopes and use different DNS servers search orders in each scope. When we got Windows server 2012(R2) we not only gained policies to take care of this but also DHCP failover with replica. That’s awesome as it relieves us of much of the tedious work of keeping track of maintaining split scopes and different options on all DCHP servers involved. For more information in using the MAC addresses and DCHP policies to load balance the use of your DNS servers read this TechNet article Load balancing DNS servers using DHCP Server Policies.

Fixed IP configurations

But what about servers with fixed IP addresses? Indeed, the dream world where we’ll see dynamically assigned IP configuration everywhere is a good one but perfection is not of this world. Fixed IP configurations are still very common and often for good reasons. Some turn to DCHP reservations to achieve this but many go for static IP configuration on the servers.

image

When that’s the case, our sys admins are told the DNS servers to use. Most of the time they’ll enter those in the same order over and over again, whether they do this manual or automated. So that means that the first and second DNS server in the search order are the same everywhere. No load balancing to be found. So potentially one DNS server is doing all the work and getting slower at it while the second or third DNS servers in the search order only help out when the first one is down or doesn’t respond quickly enough anymore. Not good. When you consider many (most?) used AD integrated DNS for their MSFT environments that’s even less good.

PowerShell Script to Load Balance DNS Server Search Order

That’s why when replacing DNS Servers or seeing response time issues on AD/DNS servers I balance the DNS server search order list. I do this based on their IP address its last octet. If that’s even, DNS Server A is the first in the search order and if not it’s DNS Server B that goes in first. That mixes them up pseudo random enough.

I use a PowerShell script for that nowadays instead of my age-old VBScript one. But recently I wanted to update it to no longer use WMI calls to get the job done. That’s the script I’m sharing here, or at least the core cons pet part of it, you’ll need to turn it into a module and parameterize if further to suit your needs. The main idea is here offering an alternative to WMI calls. Do note you’ll need PowerShell remoting enabled and configured and have the more recent Windows OS versions (Windows Server 2012 and up).

cls
#The transcipt provides a log to check what was found and what changed.
Start-Transcript -Path C:\SysAdmin\MyDNSUpdateLog.txt #
$VMsOnHost = (Get-VM -ComputerName MyHyperVHostorClusterName).Name

foreach ($VM in $VMsOnHost)
{
    Invoke-Command -ComputerName $VM -ScriptBlock {

    #This function checks if the last octet of an IP address is even or not
    Function IsLastOctetEven ($IPAddress)
        {
             #$FirstIP
             $Octets = $IPAddress.Split(".")
             #$Octets[3] #0 based array, grab 4th octet

             #See if 4th octect is even
             $Boolean = [bool]!($Octets[3]%2)
             if ($Boolean)
             {
                 Return $Boolean
                 #write-host "even"
             }
             else
             {
                 Return $Boolean
                 #write-host "odd"
             }
        }

        $OldDns1 = "10.15.200.10"
        $OldDns2 = "10.15.200.11"
        $NewDns1 = "10.18.50.110"
        $NewDns2 = "10.18.50.120"

        $NicInterfaces = Get-DnsClientServerAddress

        foreach ($NICinterface in $NicInterfaces)
        {
                #Here we filter out all interfaces that are not used for client/server connectivity.
                #Cluster Interfaces, HeartBeats, Loop back adapters, ...
                #We also filter out IPv6 here as this is for a IVp4 environment.
             if($NicInterface.InterfaceAlias -notmatch "isatap" -and $NicInterface.InterfaceAlias -notmatch "Pseudo" `
                -and $NicInterface.InterfaceAlias.Contains("Local Area Connection*") -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("KEMP-DSR-LOOPBACK") -ne $True `
                -and $NicInterface.InterfaceAlias.ToLower().Contains("Heartbeat".Tolower()) -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("NLB-PRIVATE") -ne $True-and $NicInterface.AddressFamily -ne "23")
             {

                $Output = "Hello from  $env:computername" + $NICinterface.InterfaceAlias
                write-Output $Output            
           
                $Output = $NicInterface.InterfaceAlias +": DNS1=" + $NicInterface.ServerAddresses.GetValue(0) + " & DNS2=" +  $NicInterface.ServerAddresses.GetValue(1)
                write-Output $Output

                If (($NicInterface.ServerAddresses.GetValue(0) -like $OldDns1 -or $NicInterface.ServerAddresses.getvalue(0) -like $OldDns2) -and ($NicInterface.ServerAddresses.getvalue(1) -like $oldDns1 -or $NicInterface.ServerAddresses.getvalue(1) -like $OldDns2))
                {
                    #If the IP address is DHCP assignd, leave it alone,
                    #that's handled via DHCP policies on the MAC address
                    $GetNetIPInfo = Get-NetIpAddress -InterfaceIndex  $NicInterface.InterfaceIndex
                     if ($GetNetIPInfo.PrefixOrigin -like "DHCP")
                     {
                        $VM                   
                        write-output "DHCP address - leave it alone"
                     }
                     Else
                     {
                         $IPAddresses = $GetNetIPInfo.IPv4Address
                         $FirstIP = $IPAddresses[1] #1 based array
                 
                         if (IsLastOctetEven($FirstIP)){
                            $VM
                            write-output "EVEN 4th IP octet => so DNS search order becomes $NewDns1 , $NewDns2"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns1,$NewDns2)
                         }
                         else
                         {   
                            $VM
                            write-Output "ODD 4th IP octet => so DNS search order becomes $NewDns2 , $NewDns1"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns2, $NewDns1)
                         } 
                         $NicInterface |  Select-Object -ExpandProperty ServerAddresses    
                     }
                }
                else
                {
                    $VM
                    write-Output "Existing DNS values not like expected old values. They are propably already changed"
                }        
            }
        }
    }
}
Stop-Transcript

May 2017 will be a travelling month

Introduction

In ICT, you never stop learning. Changes come and go fast. Navigating through these turbulent times of rapid change, short value cycles in order to provide continuity in both services & value without falling behind or being held back is a challenge we all face every day. If you hire or employ technologists, please take a moment to consider what they pull off for you every day. It helps to be realistic on what to expect from and to achieve with them. For that a solid understanding of the technology ecosystem and good doctrine to achieve your goals are necessary. For that to really happen and for their efforts to pay off we need to make sure politics and bureaucracy are kept under control. Let your people shine and move ahead. Long term planning does not equate a strategy and you might find yourself out paced & maneuvered by the industry and your competitors. That’s a reason why you see technologist move up the ladder and take on the leadership role inside many companies. They tend to be better placed to see the opportunities and what these require. In that respect, it pays off to walk out of your office every now and then in order to prevent tunnel vision and echo chambers. That’s one of the reasons that for me May 2017 will be a travelling month.

May 2017 will be a travelling month

Cloud & Datacenter Conference Germany

First, I’ll be in Munich, Germany, fort he Hyper-V community day and to both attend / speak at the Cloud & Datacenter Conference Germany 2017. That’s a conference for and by the community and the speakers are all highly experienced people who talk the talk and walk the walk.

clip_image002

I you can grab a ticket asap. From the very 1st edition the Cloud & Datacenter Conference Germany has set the standard for what a conference should be like.  I’ll be talking about SMB Direct / RDMA on the Hyper-V community day and about Windows Server 2016 Failover Clustering & Hyper-V at the conference. Please feel free to come over an chat.

Dell EMC World 2017

After that I’m off to DELL EMC World 2017 where I’ll be diving into the offerings that exit today and in the near future. As you might have guessed I’m very interested in the DELL Storage Spaces solutions, there take on and use of ReFSv3 and Windows Server 2016. Next to that, I would not be nick named RoCE Balboa if I was not interested in networking. Hardware wise I have my eye on the S-Series S6100-ON as that is one versatile piece of equipment. Man, I imaging having a lab with a 6 of those to test and play around with. No to mention the S2D clusters & backup targets to hammer them with a nice workload. Throw in the Mellanox cards for good measure. I can dream right ? As I’m a realist I’m also very interested in their servers and still, the Compellent offerings, which as far as traditional SANs go is one easy to manage & leverage piece of gear. It goes without saying I’ll be taking a look at what the EMC addition to the portfolio can achieve for us as well as the DELL EMC 3rd party offerings

clip_image004

VeeamOn 2017

After that I continue on to VeeamON 2017 which makes a great addition to the two above. The Windows Server 2016 core stack as the basis for Azure Stack, S2D running on that great DELLEMC hardware. Now have that protected and made continuous available by the Veeam Availability Suite 9.5. That’s how you get an amazing stack of technologies on which to build, support amazingly good services.

clip_image005

At VeeamON 2017 I’ll be joining two big names in the industry Luca Dell’Oca and Carsten Rachfahl to talk about ReFSv3. We’ll be attending sessions and “hanging out” at the MSFT boot as well.

So, no rest for a Microsoft MVP, Microsoft Extended Experts team member, Azure Advisor, a DELL Community Rockstar and a Veeam Vanguard. We’re always reading up, learning, investigating, sharing experiences & insights with our peers and learning from them. Conferences done right are very valuable and a great networking / leaning opportunity. Make the most of them when you can.

My value is your value

These conferences together with our focus on some very innovative and promising public and hybrid cloud technologies in Azure will keep me busy contemplating designs, testing the concepts of solution I have in my mind and delivering very efficient and effective solutions both in functionality as well as in TCO and ROI. That (and caffeine) combined with working with great and smart people is what makes me run. So for that reason alone I do not mind that May 2017 will be a travelling month.

Veeam Backup & Replication Preferred Subnet & SMB Multichannel

Introduction

In a previous blog post Veeam Backup & Replication leverages SMB Multichannel post we showed that Veeam backup & Replication leverages SMB multichannel when possible.

But what about Veeam Backup & Replication Preferred Subnet & SMB Multichannel, does that work? We mentioned that we wanted to answer the question what happens if we configure a preferred back-up network in Veeam Backup & Replication. Would this affect the operation of SMB multichannel at all? By that I means, would enabling a preferred network in Veeam prevent multichannel from using more than one NIC?

In this blog post we dive in to that question and some scenarios. We actually need to be able to deal with multiple scenarios. When you have equally capable NICs that are on different subnets you might want to make sure it uses only one. Likewise, you want both to be used whether they are or are not on the same subnet even if you set a preferred subnet in Veeam. The good news is that the nature of SMB Multichannel and how Veaam preferred networks work do allow for flexibility to achieve this. But it might not work like you would expect, unless you understand SMB Multichannel.

Veeam Backup & Replication Preferred Subnet & SMB Multichannel

For this blog post we adapt our lab networking a bit so that our non-management 10Gbps rNICs are on different subnets. We have subnet 10.10.110.0/24 for one set of NICs and 10.10.120.0/24 for the second set of NICs. This is shown in the figure below.

image

These networks can live in a separate VLAN or not, that doesn’t really matter. It does matter if to have a tagged VLAN or VLANS if you want to use RDMA because you need it to have the priority set.

We now need to configure our preferred network in Veeam Backup & Replication. We go to the main menu and select Network Traffic Rules

clip_image004

In the Global Network Traffic Rules window, click Networks.

clip_image006

In the Preferred Networks window, select the Prefer the following networks for backup and replication traffic check box.

clip_image008

Click Add. We use the CIDR notation to fill out our preferred network or you can use the network mask and click OK.

To prove a point in regards to how Multichannel works isn’t affected by what you fill out here we add only one of our two subnets here. SMB will see where it can leverage SMB multichannel and it will kick in. Veeam isn’t blocking any of its logic.

So now we kick of a backup of our Hyper-V host to our SMB hare target backup repository. We can see multichannel work just fine.

clip_image010

Below is a screenshot on the backup target of the backup running over SMB multichannel, leveraging 2 subnets, while having set only one of those as the preferred network in Veeam Backup & Replication

clip_image012

Look at my backup fly … and this is only one host that’s being backup (4 VMs actually). Have I told you how much I love flash storage? And why I’m so interested in getting ReFS hybrid volumes with SSD/SATA disks to work as backup target? I bet you do!

Looking good and it’s easy, right? Well not so fast!

Veeam does not control SMB Multichannel

Before you think you’re golden here and in control via Veeam lets do another demo. In the preferred network, we enter a subnet available to both the source and the target server but that is an LBFO (teamed) NIC with to 1Gbps members (RSS is enabled).

clip_image014

No let’s see what happens when we kick of a backup.

clip_image016

Well SMB multichannel just goes through its rules and decided to take the two best, equally capable NICs. These are still our two 10Gbps rNICs. Whatever you put in the preferred network is ignored.

This is neither good or bad but you need to be aware of this in order to arrange for backups to leverage the network path(s) you had in mind. This is to avoid surprises. The way to do that the same as you plan and design for all SMB multichannel traffic.

As stated in the previous blog post you can control what NICs SMB multichannel will use by designing around the NIC capabilities or if needed disabling or enabling some of these or by disabling SMB multichannel on a NIC. This isn’t always possible or can lead to issues for other workloads so the easiest way to go is using SMB Multichannel Constraints. Do note however that you need to take into consideration what other workloads on your server leverage SMB Multichannel when you go that route to avoid possible issues.

As an example, I disabled multichannel on my hosts. Awful idea but it’s to prove point. And still with our 10.10.0.0/16 subnet set as preferred subnet I ran a backup again.

clip_image018

As you can see the 2*1Gbps LBFO NIC is doing all the lifting on both hosts as it’s switch independent and not LACP load balancing mode we’re limited to 1Gbps.

So how do we control the NICs used with SMB multichannel?

Well SMB Multichannel rules apply. You use your physical design, the capabilities of the NICs and SMB constraints. In reality you’re better off using your design and if needed SMB multichannel constraints to limit SMB to the NICs you want it to use. Do not that disabling SMB Multichannel (client and or server) is a global for the host. Consider this as it affects all NICs on the host, not just the ones you have in mind for your backups. In most cases these NICs will be the same. Messing around with disabling multichannel or NIC capabilities (RSS, RDMA) isn’t a great solution. But it’s good to know the options and behavior.

Some things to note

Realize you don’t even have to set both subnets in the preferred subnets if they are different. SMB kicks of over one, sees it can leverage both and just does so. The only thing you manipulated here SMB multichannel wise is which subnet is used first.

If both of our rNICs would have been on the same subnet you would not even have manipulated this.

Another thing that’s worth pointing out that this doesn’t require your Veeam Backup & Replication VM to have an IP address in any of the SMB multichannel subnets. So as long as the source Hyper-V hosts and the backup target are connected you’re good to go.

Last but not least, and already mentioned in the previous blog post, this also leverages RDMA capabilities when available to help you get the best throughput, lowest latency and leave those CPU cycles for other needs. Scalability baby! No I realize that you might think that the CPU offload benefit is not a huge deal on your Hyper-V host but consider the backup target being hammered by several simultaneous backups. And also consider that some people their virtual machines look like below in regards to CPU usage, in ever more need of more vCPU and CPU time slices.

clip_image020

And that’s what the Hyper-V host looks like during a backup without SMB Direct (with idle VMs mind you).

clip_image022

All I’m saying here is don’t dismiss RDMA too fast, everything you can leverage to help out and that is available for free in the box is worth considering.

Note: I have gotten the feedback that Veeam doesn’t support SMB Direct and that this was confirmed by Veeam Support. Well, Veeam Backup & Replication leverages SMB 3 but that’s an OS feature. Veeam Backup & Replication will work with SMB Multichannel, Direct, Signing, Transparent Failover … It’s out of the Veeam Backup & Replication scope of responsibilities as we have seen here. You feel free to leverage SMB Direct whether that is using iWarp/Roce or Infiniband. This information was confirmed by Veeam and bears the “Anton Gostev seal of approval”. So if SMB Direct cause issues you have a configuration problem with that feature, it’s not Veeam not being able to support it, it doesn’t know or care actually.

Conclusion

The elegance and simplicity of the Veeam Backup & Replication GUI are deceiving. Veeam is extremely powerful and is surprisingly flexible in how you can leverage and configure it. I hope both my previous blog post and this one have given you some food for thought and ideas. There’s more Veeam goodness to come in the coming months when times allows. Many years ago, when SMB 3 was introduced I demonstrated the high availability capabilities this offered for Veeam backups. I’ll be writing about that in another blog post.