Preventing Live Migration Over SMB Starving CSV Traffic in Windows Server 2012 R2 with Set-SmbBandwidthLimit

One of the big changes in Windows Server 2012 R2 is that all types of Live Migration can now leverage SMB 3.0 if the right conditions are met. That means that Multichannel & SMB Direct (RDMA) come in to play more often and simultaneously. Shared Nothing Live Migration & certain forms of Storage Live Migration are often a lot more planned due to their nature. So one can mitigate the risk by planning.  Good old standard Live Migration of virtual machines however is often less planned. It can be done via Cluster Aware Updating, to evacuate a host for hardware maintenance, via Dynamic optimization. This means it’s often automated as well. As we have demonstrated many times Live Migration can (easily) fill 20Gbps of bandwidth. If you are sharing 2*10Gbps NICs for multiple purposes like CSV, LM, etc. Quality of Service (QoS) comes in to play. There are many ways to achieve this but in our example here I’ll be using DCB  for SMB Direct with RoCE.

New-NetQosPolicy “CSV” –NetDirectPortMatchCondition 445 -PriorityValue8021Action 4
Enable-NetQosFlowControl –Priority 4
New-NetQoSTrafficClass "CSV" -Priority 4 -Algorithm ETS -Bandwidth 40
Enable-NetAdapterQos –InterfaceAlias SLOT41-CSV1+LM2
Enable-NetAdapterQos –InterfaceAlias SLOT42-LM1+CSV2
Set-NetQosDcbxSetting –willing $False

Now as you can see I leverage 2*10Gbps NIC, non teamed as I want RDMA. I have Failover/redundancy/bandwidth aggregation thanks to SMB 3.0. This works like a charm. But when leveraging Live Migration over SMB in Windows Server 2012 R2 we note that the LM traffic also goes over port 445 and as such is dealt with by the same QoS policy on the server & in the switches (DCB/PFC/ETS). So when both CSV & LM are going one how does one prevent LM form starving CSV traffic for example? Especially in Scale Out File Server Scenario’s this could be a real issue.

The Solution

To prevent LM traffic & CSV traffic from hogging all the SMB bandwidth ruining the SOFS party in R2 Microsoft introduced some new capabilities in Windows Server 2012 R2. In the SMBShare module you’ll find:

  • Set-SmbBandwidthLimit
  • Get-SmbBandwidthLimit
  • Remove-SmbBandwidthLimit

image

To use this you’ll need to install the Feature called SMB Bandwidth Limit via Server Manager or using PowerShell:  Add-WindowsFeature FS-SMBBW

You can limit SMB bandwidth for Virtual machine (Storage IO to a SOFS), Live Migration & Default (all the rest).  In the below example we set it to 8Gbps maximum.

Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 1000MB

So there you go, we can prevent Live Migration from hogging all the bandwidth. Jose Baretto mentions this capability on his recent blog post on Windows Server 2012 R2 Storage: Step-by-step with Storage Spaces, SMB Scale-Out and Shared VHDX (Virtual). But what about Fibre Channel or iSCSI environments?  It might not be the total killer there as in SOFS scenario but still. As it turns out the Set-SmbBandwidthLimit also works in those scenarios. I was put on the wrong track by thinking it was only for SOFS scenarios but my fellow MVP Carsten Rachfahl kindly reminded me of my own mantra “Trust but verify” and as a result, I can confirm it even works to cap off Live Migration traffic over SMB that leverages RDMA (RoCE). So don’t let the PowerShell module name (SMBShare) fool you, it’s about all SMB traffic within the categories.

So without limit LM can use all bandwidth (2*10Gbps)

image

With Set-SmbBandwidthLimit -Category LiveMigration -BytesPerSecond 1250MB you can see we max out at 10Gbps (2*5Gbps).
image

Some Remarks

I’d love to see a minimum bandwidth implementation of this (that could include safety buffer for spikes in CSV traffic with SOFS). The hard cap limit might lead to some wasted bandwidth. In other scenarios you could still get into trouble. What if you have 2*10Gbps available but one of those dies on you and you capped Live Migration Traffic at 16Gbps. With one NIC gone you’re potentially in trouble until the NIC has been replaced. OK, this is not a daily occurrence & depending on you environment & setup this is less or more of a potential issue.

PowerShell: Monitoring DrainStatus of a Hyper-V Host & The Time Limited Value of Information In Beta & RC Era Blogs

I was writing some small PowerShell scripts to kick pause and resume Hyper-V cluster hosts and I wanted to monitor the progress of draining the virtual machines of the node when pausing it. I found this nice blog about Draining Nodes for Planned Maintenance with Windows Server 2012 discussing this subject and providing us with the properties to do just that.

It seems we have two common properties at our disposal: NodeDrainStatus and NodeDrainTarget.

image

So I set to work but I just didn’t manage to get those properties to be read. It was like they didn’t exist. So I pinged Jeff Wouters who happens to use PowerShell for just about anything and asked him if it was me being stupid and missing the obvious. Well it turned out to be missing the obvious for sure as those properties do no exist. Jeff told me to double check using:

Get-ClusterNode MyNode -cluster MyCluster | Select-Object -Property *

Guess what, it’s not NodeDrainStatus and NodeDrainTarget but DrainStatus and DrainTarget.

image

What put me off here was the following example in the same blog post:

Get-ClusterResourceType "Virtual Machine" | Get-ClusterParameter NodeDrainMoveTypeThreshold

That should have been a dead give away. As we’ve been using MoveTypeTresHold a lot the recent months and there is no NodeDrain in that value either. But it just didn’t register. By the way you don’t need to create the property either is exists. I guess this code was valid with some version (Beta?) but not anymore. You can just get en set the property like this

Get-ClusterResourceType “Virtual Machine” -Cluster MyCluster | Get-ClusterParameter MoveTypeThreshold

Get-ClusterResourceType “Virtual Machine” -Cluster MyCluster | Set-ClusterParameter MoveTypeThreshold 2000

So lessons learned. Trust but verify Smile.  Don’t forget that a lot of things in IT have a time limited value. Make sure that to look at the date of what you’re reading and about what pre RTM version of the product the information is relevant to.

To conclude here’s the PowerShell snippet I used to monitor the draining process.


Suspend-clusternode –Name crusader -Cluster warrior -Drain

Do
{
    Write-Host (get-clusternode –Name “crusader” -Cluster warrior).DrainStatus -ForegroundColor Magenta    
    Sleep 1
}
until ((get-clusternode –Name “crusader” -Cluster warrior).DrainStatus -ne "InProgress")

If ((get-clusternode –Name “crusader” -Cluster warrior).DrainStatus -eq "Completed")
{
    Write-Host (get-clusternode –Name “crusader” -Cluster warrior).DrainStatus -ForegroundColor Green
}

Which outputs

image

Monitoring Startup,Shutdown and restart of a Virtual machine With PowerShell 3.0

During scripting some maintenance PowerShell scripts for Hyper-V guests I felt the need for a more accurate way to monitor the startup of a virtual machine. Pings, telnet to a known open port it all doesn’t do the job accurately enough as I want to know when CTRL+AL+DEL appears on the screen. So I pinged Jeff Wouters who told me I could monitor Get-VM -Name DC01 | Get-VMIntegrationService  to detect when PrimaryStatusDescription goes to “OK”.

Now when you look at the Integration services there are 5 of them.

image

Which one is the best to use for our purpose? Well,I tested them out and after some experimenting with the various services I concluded that the PrimaryStatusDescription of the  Key-Value Pair Exchange works best for this purpose. All others become available a bit to soon in the process of starting a VM, which seems logical.

Monitor a starting virtual machine

So how to use this in a script? We’ll here’s a snippet to monitor the boot process of a guest.

$Vm = Get-VM "MyVM"
start-VM "$Vm"
#This means the VM is now shutting down ...    
$Counter = 0
$ProgressCount = 0
Do
{    
    $Operational = Get-VM -Name $VM | Get-VMIntegrationService -Name "Key-Value Pair Exchange"
    $Counter = $Counter + 1 
    $ProgressCount =  $ProgressCount +1
    $PercentComplete = ($ProgressCount * 20)
    Write-Progress -Activity "$VM" -status "VM starting up: $Status - Progressbar indicates activity, not a percent of completion: ($Counter Seconds)"  -percentComplete ($PercentComplete / 100 *100)
    if ($PercentComplete -gt 90) {$ProgressCount = 0}
    sleep 1
}
While ($Operational.PrimaryStatusDescription -ne "OK")
$Status = (Get-VM -Name $VM | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
Write-Progress -Activity "VM $VM is up and running" -status "VM status: $Status - We're done here. Completed in a total of $Counter seconds."  -percentComplete (100)

 

Monitor a stopping virtual machine

Likewise, sometime we want to monitor a VM shutting down, which is the same code as above but with reverse logic.

$Vm = Get-VM "MyVM"
stop-VM "$Vm"
$Counter = 0
$ProgressCount = 0
#This means the VM is now shutting down  in the retart cycle ...    
   Do
   {            
    $Operational = Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange"
    $Counter = $Counter + 1 
    $Status = (Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
    $ProgressCount =  $ProgressCount + 1
    $PercentComplete = ($ProgressCount * 20)
    Write-Progress -Activity "$VM" -status "VM shutting down : $Status - Progressbar indicates activity, not a percent of completion: ($Counter Seconds)"  -percentComplete ($PercentComplete / 100 *100)
    if ($PercentComplete -gt 90) {$ProgressCount = 0}
    sleep 1
   }
   While ($Operational.PrimaryStatusDescription -eq "OK")
   $Status = (Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
   Write-Progress -Activity "VM $Vm has now been shutdown" -status "VM status: $Status - We're done here. Completed in a total of $Counter seconds."  -percentComplete (100)

Monitor a restarting a virtual machine.

When in a PowerShell script you want to monitor progress of a virtual machine restarting you can combine both. You monitor shutdown and you monitor startup.

$VmThatRestarts = Get-VM "MyVM"
#Restart the VM
#This means the VM is now shutting down  in the retart cycle ...
 $Counter = 0
 $ProgressCount = 0
Do
{            
    $Operational = Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange"
    $Counter = $Counter + 1 
    $Status = (Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
    $ProgressCount =  $ProgressCount + 1
    $PercentComplete = ($ProgressCount * 20)
    Write-Progress -Activity "$VM" -status "VM restarting - Shutdown phase : $Status - Progressbar indicates activity, not a percent of completion: ($Counter Seconds)"  -percentComplete ($PercentComplete / 100 *100)
    if ($PercentComplete -gt 90) {$ProgressCount = 0}
    sleep 1
}
While ($Operational.PrimaryStatusDescription -eq "OK")
$Status = (Get-VM -Name $Vm | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
Write-Progress -Activity "VM $Vm has now been shutdown in restart cycle" -status "VM status: $Status - VM has shut down in $Counter Seconds"  -percentComplete (100)
   
#Any thing worthwhile is worth adding 1 second of waiting for good measure 🙂
Sleep 1

#This means the VM is now starting  ...    
$ProgressCount = 0
Do
{    
    $Operational = Get-VM -Name $VM | Get-VMIntegrationService -Name "Key-Value Pair Exchange"
    $Counter = $Counter + 1 
    $ProgressCount =  $ProgressCount +1
    $PercentComplete = ($ProgressCount * 20)
    Write-Progress -Activity "$VM" -status "VM restarting - Startup phase: $Status - Progressbar indicates activity, not a percent of completion: ($Counter Seconds)"  -percentComplete ($PercentComplete / 100 *100)
    if ($PercentComplete -gt 90) {$ProgressCount = 0}
    sleep 1
}
While ($Operational.PrimaryStatusDescription -ne "OK")
$Status = (Get-VM -Name $VM | Get-VMIntegrationService -Name "Key-Value Pair Exchange").PrimaryStatusDescription
Write-Progress -Activity "VM $VM is up and running again" -status "VM status: $Status - We're done here. Completed in a total of $Counter seconds."  -percentComplete (100)

Note that in all the above snippets  I’ve thrown some logic in to us the progress bar as an activity bar as I know of no way to calculate real % done in a startup, shutdown, restart process. It looks something like this in ISE

image

or like this in a PowerShell prompt

image

Checking Host Integration Services Version on all Nodes of A Windows Server 2012 Hyper-V Cluster With PowerShell

It’s important to keep our Hyper-V cluster hosts and the virtual machines running on them up to date. Whilst we have great and free solutions to achieve this there are some things missing like centralized reporting on the Integration Services component version running on all of the nodes in a cluster and way to upgrade all the virtual machines to version running on the host. This post deals with the first issue.

Before we upgrade the Integration Services components on the virtual machines we always check if all nodes in the cluster are on the same version themselves. Sure this should not happen if you mange them right but my world isn’t perfect. So trust but verify.With cluster sizes now up to 64 nodes it’s ever more important to keep an eye on them. But even for smaller cluster the task of determining the Integration Services components manually via the GUI, event viewer and/or registry is rather tedious. Out of sync Integration Services components can be troublesome and cause many issues and if you have out of sync virtual machines, imagine the extra mess you’ll be in when even the cluster nodes are running different versions.

To make live easier I threw a little PowerShell script together to check the host Integration Services component version on all nodes of a Window Server 2012 Hyper-V Cluster With PowerShell. I’m far from a PowerShell guru, but you’ll see that you can do a lot of things  done even if you’re not. I’m sharing it here for you to use, adapt for your own needs and get some inspiration. It basically allows you to optionally pass an expected version of the IS components and a cluster name like this

CheckHyperVClusterHostsICVersion -Version 6.2.9200.16433 -cluster "MyClusterName"

It does the following:

  • It will list per Integration Services component version found on cluster nodes what version was found on what nodes. This gives you a nice overview. I hope this never becomes to much of a list in your clusters.
  • If you don’t specify a cluster it will try to connect to the cluster to which the host you’re running on belongs, if any.
  • If the host does not belong to a cluster it will just provide feedback on the IS version of that Hyper-V host you’re running the script on.

Here’s a screen shot of when you run this on a none clustered host, without Hyper-V installed:

image

This is the result of running it against a well maintained cluster without any parameters that has been updated with KB2770917:

image

The same but now with the expected version and cluster name passed as parameters

image

So, there you go, I hope you find it useful.

#===========================================================
# # Microsoft PowerShell Source File 
# 
# NAME:    CheckISCOnNodesOfHyperVCluster.ps1
# VERSION:    1.0.0.0

# AUTHOR:    Didier Van Hoye
# DATE :    17/11/2012
# 
# COMMENT:     This script is intended to be run 
#              against Windows Server 2012 and assumes
#            the use of PowerShell 3.0
#            The parameters are optional but if you
#            leave out some the remainder should be named.
# # =======================================================
 
cls
$ErrorActionPreference = "Stop"

 
function CheckHyperVClusterHostsICVersion
{
    Param
    (
        #Param help description
        [Version]
        $ExpectedISCVersion,
        #Param help description
        [String]
        $Cluster
    )

    Write-Host "This script will check the IS components on all nodes of a cluster." -ForegroundColor Green
 
    If ($ExpectedISCVersion) {Write-Host "You specified the expected IS component version to be $ExpectedISCVersion" -ForegroundColor Green}
    Else {Write-host "You did not specify an expected IS component version." -ForegroundColor Green}
    
    If ($Cluster)
    {
        Try
        {
            $ClusterObject= Get-Cluster -Name $Cluster
        }
        Catch
        {     
            Write-Host "We cannot contact the cluster you specified"
        }
    }
    Else
    {    
        write-Host "`n`n"
        Write-host "You did not specify a cluster to connect to. We'll use the cluster to which the node this script is running on belongs if any." -ForegroundColor Yellow
        write-Host "`n`n"
  
        Try
        {
            $ClusterObject = Get-Cluster
        }
  
        Catch
        {
            $LocalHost = $env:computername
            Write-Host
            Write-Host "The current node ($LocalHost) is not a member of a cluster. As a courtsey to you we'll check the IS components for current host" -foregroundcolor Magenta
            Write-Host
        }
 
    }
  
    If ($ClusterObject) {$ToCheck= "the nodes of cluster $ClusterObject"} Else { $ToCheck = "server $env:computername"}
 
    write-Host "Attempting to running Integration Components version check on" $ToCheck -ForegroundColor Green
    Write-Host


    If ($ClusterObject)
    {

        $ClusterNodes = Get-Clusternode -cluster $ClusterObject.Name
        
        #Declare an hashtable to hold all host/IS version values. The hosts are the key here.
        $HostISVersions = @{}
 
        foreach ($ClusterNode in $ClusterNodes)
        {
            Try
            {
                 $HostISVersions[$ClusterNode.Name]=Get-ItemProperty "HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionVirtualizationGuestInstallerVersion" | select -ExpandProperty Microsoft-Hyper-V-Guest-Installer
            }
            Catch
            {
            Write-Host "We could not determine the version of the Integration Services on this host, probably due to this not being a Hyper-V host" -ForegroundColor Orange
            Write-Host "We'll check this for you right now" -ForegroundColor Orange
            $HyperVFeature = Get-WindowsFeature Hyper-V
            If ($HyperVFeature.Installstate -eq "Installed")
            {
              Write-Host "Hyper-V seems to be installed on this node. Something else is wrong." -ForegroundColor Red
            }
            Else
            {
                Write-Host "Hyper-V is indeed not installed on this node." -ForegroundColor Orange
            }
            }
        }
         #Use GetEnumerator or thise sorting thing doesn't work out well on an hash tabel 🙂
        $UniqueIcVersions = $HostISVersions.GetEnumerator() | Sort-Object -Property Value -Unique
 
        Write-Host "We've found " $UniqueIcVersions.count "versions on the" $HostISVersions.count "nodes of your cluster" $ClusterObject.Name
 
        ForEach ($IcVersion in $UniqueIcVersions )
        {
            $Counter = 1
            $IcVersionValue = $IcVersion.value
            "IC version " + $IcVersion.value + " is found in:"
            foreach ($Key in ($HostISVersions.GetEnumerator()| Where-Object { $_.value -eq $IcVersionValue}))
            {
                "`t" + "$Counter : " + $Key.Name
                $Counter= $Counter + 1
            }
 
            If ($ExpectedISCVersion)
            {
               
                $CompareVersions = ([Version]$IcVersion.Value).CompareTo([Version]$ExpectedISCVersion)
                        
                switch ($CompareVersions)
                {
                    0 {Write-Host "This version ($IcVersionValue) is equal to the expected version ($ExpectedISCVersion)." -ForegroundColor Green}
                    1 {Write-Host "This version ($IcVersionValue) is higher than the expected version ($ExpectedISCVersion). Please ensure all hosts run the same IC version level." -ForegroundColor Yellow}
                    -1 {Write-Host "This version ($IcVersionValue) is lower than the expected version ($ExpectedISCVersion). Please ensure all hosts run the same IC version level." -ForegroundColor Red}
                }
            }

        }
    }

    Else
    {
        Try
        {
            $HostIcVersion = Get-ItemProperty "HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionVirtualizationGuestInstallerVersion" | select -ExpandProperty Microsoft-Hyper-V-Guest-Installer
            Write-Host "The IS component version on server $localhost is $HostIcVersion"
            If ($ExpectedISCVersion)
            {
               
                   $CompareVersions = ([Version]$HostIcVersion).CompareTo([Version]$ExpectedISCVersion)
                        
                switch ($CompareVersions)
                {
                0 {Write-Host "This version ($HostIcVersion) is equal to the expected version ($ExpectedISCVersion)." -ForegroundColor Green}
                1 {Write-Host "This version ($HostIcVersion) is higher than the expected version ($ExpectedISCVersion). Please check if you need to downgrade your host or if the expected version is correct." -ForegroundColor Yellow}
                -1 {Write-Host "This version ($HostIcVersion) is lower than the expected version ($ExpectedISCVersion). Please check if you need to upgrade your host or if the expected version is correct." -ForegroundColor Red}
                }
            }
        }
        Catch
        {
            Write-Host "We could not determine the version of the Integration Services on this host, probably due to this not being a Hyper-V host" -ForegroundColor yellow
            Write-Host "We'll check this for you right now" -ForegroundColor yellow
            $HyperVFeature = Get-WindowsFeature Hyper-V
            If ($HyperVFeature.Installstate -eq "Installed")
            {
                Write-Host "Hyper-V seems to be installed on this node. Something else is wrong." -ForegroundColor Red
            }
            Else
            {
                Write-Host "Hyper-V is indeed not installed on this node." -ForegroundColor yellow
            }
        }
    }
}
 
CheckHyperVClusterHostsICVersion -Version 6.2.9200.16433 -cluster "MyClusterName"