Collect cluster nodes with HBA WWN info

Introduction

Below is a script that I use to collect cluster nodes with HBA WWN info. It grabs the cluster nodes and their HBA (virtual ports) WWN information form an existing cluster. In this example the nodes have Fibre Channel (FC) HBAs. It works equally well for iSCSI HBA or other cards. You can use the collected info in real time. As an example I also demonstrate writing and reading the info to and from a CSV.

This script comes in handy when you are replacing the storage arrays. You’ll need that info to do the FC zoning for example.  And to create the cluster en server object with the correct HBA on the new storage arrays if it allows for automation. As a Hyper-V cluster admin you can grab all that info from your cluster nodes without the need to have access to the SAN or FC fabrics. You can use it yourself and hand it over to those handling them, who can use if to cross check the info they see on the switch or the old storage arrays.

image

Script to collect cluster nodes with HBA WWN info

The script demos a single cluster but you could use it for many. It collects the cluster name, the cluster nodes and their Emulex HBAs. It writes that information to a CSV files you can read easily in an editor or Excel.

image

The scripts demonstrates reading that CSV file and parsing the info. That info can be used in PowerShell to script the creation of the cluster and server objects on your SAN and add the HBAs to the server objects. I recently used it to move a bunch of Hyper-V and File clusters to a new DELLEMC SC Series storage arrays. That has the DELL Storage PowerShell SDK. You might find it useful as an example and to to adapt for your own needs (iSCSI, brand, model of HBA etc.).

#region Supporting Functions
Function Convert-OutputForCSV {
    <#
        .SYNOPSIS
            Provides a way to expand collections in an object property prior
            to being sent to Export-Csv.

        .DESCRIPTION
            Provides a way to expand collections in an object property prior
            to being sent to Export-Csv. This helps to avoid the object type
            from being shown such as system.object[] in a spreadsheet.

        .PARAMETER InputObject
            The object that will be sent to Export-Csv

        .PARAMETER OutPropertyType
            This determines whether the property that has the collection will be
            shown in the CSV as a comma delimmited string or as a stacked string.

            Possible values:
            Stack
            Comma

            Default value is: Stack

        .NOTES
            Name: Convert-OutputForCSV
            Author: Boe Prox
            Created: 24 Jan 2014
            Version History:
                1.1 - 02 Feb 2014
                    -Removed OutputOrder parameter as it is no longer needed; inputobject order is now respected 
                    in the output object
                1.0 - 24 Jan 2014
                    -Initial Creation

        .EXAMPLE
            $Output = 'PSComputername','IPAddress','DNSServerSearchOrder'

            Get-WMIObject -Class Win32_NetworkAdapterConfiguration -Filter "IPEnabled='True'" |
            Select-Object $Output | Convert-OutputForCSV | 
            Export-Csv -NoTypeInformation -Path NIC.csv    
            
            Description
            -----------
            Using a predefined set of properties to display ($Output), data is collected from the 
            Win32_NetworkAdapterConfiguration class and then passed to the Convert-OutputForCSV
            funtion which expands any property with a collection so it can be read properly prior
            to being sent to Export-Csv. Properties that had a collection will be viewed as a stack
            in the spreadsheet.        
            
    #>
    #Requires -Version 3.0
    [cmdletbinding()]
    Param (
        [parameter(ValueFromPipeline)]
        [psobject]$InputObject,
        [parameter()]
        [ValidateSet('Stack', 'Comma')]
        [string]$OutputPropertyType = 'Stack'
    )
    Begin {
        $PSBoundParameters.GetEnumerator() | ForEach {
            Write-Verbose "$($_)"
        }
        $FirstRun = $True
    }
    Process {
        If ($FirstRun) {
            $OutputOrder = $InputObject.psobject.properties.name
            Write-Verbose "Output Order:`n $($OutputOrder -join ', ' )"
            $FirstRun = $False
            #Get properties to process
            $Properties = Get-Member -InputObject $InputObject -MemberType *Property
            #Get properties that hold a collection
            $Properties_Collection = @(($Properties | Where-Object {
                        $_.Definition -match "Collection|\[\]"
                    }).Name)
            #Get properties that do not hold a collection
            $Properties_NoCollection = @(($Properties | Where-Object {
                        $_.Definition -notmatch "Collection|\[\]"
                    }).Name)
            Write-Verbose "Properties Found that have collections:`n $(($Properties_Collection) -join ', ')"
            Write-Verbose "Properties Found that have no collections:`n $(($Properties_NoCollection) -join ', ')"
        }
 
        $InputObject | ForEach {
            $Line = $_
            $stringBuilder = New-Object Text.StringBuilder
            $Null = $stringBuilder.AppendLine("[pscustomobject] @{")

            $OutputOrder | ForEach {
                If ($OutputPropertyType -eq 'Stack') {
                    $Null = $stringBuilder.AppendLine("`"$($_)`" = `"$(($line.$($_) | Out-String).Trim())`"")
                }
                ElseIf ($OutputPropertyType -eq "Comma") {
                    $Null = $stringBuilder.AppendLine("`"$($_)`" = `"$($line.$($_) -join ', ')`"")                   
                }
            }
            $Null = $stringBuilder.AppendLine("}")
 
            Invoke-Expression $stringBuilder.ToString()
        }
    }
    End {}
}
function Get-WinOSHBAInfo {
<#
Basically add 3 nicely formated properties to the HBA info we get via WMI
These are the NodeWWW, the PortWWN and the FabricName. The raw attributes
from WMI are not readily consumable. WWNs are given with a ":" delimiter.
This can easiliy be replaced or removed depending on the need.
#>

param ($ComputerName = "localhost")
 
# Get HBA Information
$Port = Get-WmiObject -ComputerName $ComputerName -Class MSFC_FibrePortHBAAttributes -Namespace "root\WMI"
$HBAs = Get-WmiObject -ComputerName $ComputerName -Class MSFC_FCAdapterHBAAttributes  -Namespace "root\WMI"
 
$HBAProperties = $HBAs | Get-Member -MemberType Property, AliasProperty | Select -ExpandProperty name | ? {$_ -notlike "__*"}
$HBAs = $HBAs | Select-Object $HBAProperties
$HBAs | % { $_.NodeWWN = ((($_.NodeWWN) | % {"{0:x2}" -f $_}) -join ":").ToUpper() }
 
ForEach ($HBA in $HBAs) {
 
    # Get Port WWN
    $PortWWN = (($Port |? { $_.instancename -eq $HBA.instancename }).attributes).PortWWN
    $PortWWN = (($PortWWN | % {"{0:x2}" -f $_}) -join ":").ToUpper()
    Add-Member -MemberType NoteProperty -InputObject $HBA -Name PortWWN -Value $PortWWN
    # Get Fabric WWN
    $FabricWWN = (($Port |? { $_.instancename -eq $HBA.instancename }).attributes).FabricName
    $FabricWWN = (($FabricWWN | % {"{0:x2}" -f $_}) -join ":").ToUpper()
    Add-Member -MemberType NoteProperty -InputObject $HBA -Name FabricWWN -Value $FabricWWN
 
    # Output
    $HBA
}
}
#endregion 

#Grab the cluster nane in a variable. Adapt thiscode to loop through all your clusters.
$ClusterName = "DEMOLABCLUSTER"
#Grab all cluster node 
$ClusterNodes = Get-Cluster -name $ClusterName | Get-ClusterNode
#Create array of custom object to store ClusterName, the cluster nodes and the HBAs
$ServerWWNArray = @()

ForEach ($ClusterNode in $ClusterNodes) {
    #We loop through the cluster nodes the cluster and for each one we grab the HBAs that are relevant.
    #My lab nodes have different types installed up and off, so I specify the manufacturer to get the relevant ones.
    #Adapt to your needs. You ca also use modeldescription to filter out FCoE vers FC HBAs etc.
    $AllHBAPorts = Get-WinOSHBAInfo -ComputerName $ClusterNode.Name | Where-Object {$_.Manufacturer -eq "Emulex Corporation"} 

    #The SC Series SAN PowerShell takes the WWNs without any delimiters, so we dump the ":" for this use case.
    $WWNs = $AllHBAPorts.PortWWN -replace ":", ""
    $NodeName = $ClusterNode.Name

    #Build a nice node object with the info and add it to the $ServerWWNArray 
    $ServerWWNObject = New-Object psobject -Property @{
        WWN         = $WWNs
        ServerName  = $NodeName 
        ClusterName = $ClusterName         
    }
    $ServerWWNArray += $ServerWWNObject
}

#Show our array
$ServerWWNArray

#just a demo to list what's in the array
ForEach ($ServerNode in $ServerWWNArray) {    
    $Servernode.ServerName
    
    ForEach ($WWN in $Servernode.WWN)
    {$WWN}

}

#Show the results
$Export = $ServerWWNArray | Convert-OutputForCSV
#region write to CSV and read from CSV

#You can dump this in a file
$Export | export-csv -Path "c:\SysAdmin\$ClusterName.csv" -Delimiter ";"

#and get it back from a file
Get-Content -Path "c:\SysAdmin\$ClusterName.csv"
$ClusterInfoFile = Import-CSV -Path "c:\SysAdmin\$ClusterName.csv" -Delimiter ";"
$ClusterInfoFile | Format-List

#just a demo to list what's in the array
$MyClusterName = $ClusterInfoFile.clustername | get-unique
$MyClusterName
ForEach ($ClusterNode in $ClusterInfoFile) {  

    $ClusterNode.ServerName
    
    ForEach ($WWN in $ClusterNode.WWN) {
        $WWN
    }

}

PowerShell Script to Load Balance DNS Server Search Order

Load Balance DNS Server Search Order

DNS servers need to be configured correctly, operate perfectly and respond as fast as possible to their clients. For some applications this is critical, but many have a more relaxed attitude. Hence a DNS Server has a full second to respond to a query. That means that even when you have 2 DNS servers configured on the clients the second will only be used when the first is not available or doesn’t respond quickly enough. This has a side effect which is that moving traffic away from an overloaded DNS servers isn’t that easy or optimal. We’ll look at when to use a PowerShell script to Load balance DNS server search order.

DHCP now and then

The trick here is to balance the possible DNS servers search order amongst the clients. We used to do this via split scopes and use different DNS servers search orders in each scope. When we got Windows server 2012(R2) we not only gained policies to take care of this but also DHCP failover with replica. That’s awesome as it relieves us of much of the tedious work of keeping track of maintaining split scopes and different options on all DCHP servers involved. For more information in using the MAC addresses and DCHP policies to load balance the use of your DNS servers read this TechNet article Load balancing DNS servers using DHCP Server Policies.

Fixed IP configurations

But what about servers with fixed IP addresses? Indeed, the dream world where we’ll see dynamically assigned IP configuration everywhere is a good one but perfection is not of this world. Fixed IP configurations are still very common and often for good reasons. Some turn to DCHP reservations to achieve this but many go for static IP configuration on the servers.

image

When that’s the case, our sys admins are told the DNS servers to use. Most of the time they’ll enter those in the same order over and over again, whether they do this manual or automated. So that means that the first and second DNS server in the search order are the same everywhere. No load balancing to be found. So potentially one DNS server is doing all the work and getting slower at it while the second or third DNS servers in the search order only help out when the first one is down or doesn’t respond quickly enough anymore. Not good. When you consider many (most?) used AD integrated DNS for their MSFT environments that’s even less good.

PowerShell Script to Load Balance DNS Server Search Order

That’s why when replacing DNS Servers or seeing response time issues on AD/DNS servers I balance the DNS server search order list. I do this based on their IP address its last octet. If that’s even, DNS Server A is the first in the search order and if not it’s DNS Server B that goes in first. That mixes them up pseudo random enough.

I use a PowerShell script for that nowadays instead of my age-old VBScript one. But recently I wanted to update it to no longer use WMI calls to get the job done. That’s the script I’m sharing here, or at least the core cons pet part of it, you’ll need to turn it into a module and parameterize if further to suit your needs. The main idea is here offering an alternative to WMI calls. Do note you’ll need PowerShell remoting enabled and configured and have the more recent Windows OS versions (Windows Server 2012 and up).

cls
#The transcipt provides a log to check what was found and what changed.
Start-Transcript -Path C:\SysAdmin\MyDNSUpdateLog.txt #
$VMsOnHost = (Get-VM -ComputerName MyHyperVHostorClusterName).Name

foreach ($VM in $VMsOnHost)
{
    Invoke-Command -ComputerName $VM -ScriptBlock {

    #This function checks if the last octet of an IP address is even or not
    Function IsLastOctetEven ($IPAddress)
        {
             #$FirstIP
             $Octets = $IPAddress.Split(".")
             #$Octets[3] #0 based array, grab 4th octet

             #See if 4th octect is even
             $Boolean = [bool]!($Octets[3]%2)
             if ($Boolean)
             {
                 Return $Boolean
                 #write-host "even"
             }
             else
             {
                 Return $Boolean
                 #write-host "odd"
             }
        }

        $OldDns1 = "10.15.200.10"
        $OldDns2 = "10.15.200.11"
        $NewDns1 = "10.18.50.110"
        $NewDns2 = "10.18.50.120"

        $NicInterfaces = Get-DnsClientServerAddress

        foreach ($NICinterface in $NicInterfaces)
        {
                #Here we filter out all interfaces that are not used for client/server connectivity.
                #Cluster Interfaces, HeartBeats, Loop back adapters, ...
                #We also filter out IPv6 here as this is for a IVp4 environment.
             if($NicInterface.InterfaceAlias -notmatch "isatap" -and $NicInterface.InterfaceAlias -notmatch "Pseudo" `
                -and $NicInterface.InterfaceAlias.Contains("Local Area Connection*") -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("KEMP-DSR-LOOPBACK") -ne $True `
                -and $NicInterface.InterfaceAlias.ToLower().Contains("Heartbeat".Tolower()) -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("NLB-PRIVATE") -ne $True-and $NicInterface.AddressFamily -ne "23")
             {

                $Output = "Hello from  $env:computername" + $NICinterface.InterfaceAlias
                write-Output $Output            
           
                $Output = $NicInterface.InterfaceAlias +": DNS1=" + $NicInterface.ServerAddresses.GetValue(0) + " & DNS2=" +  $NicInterface.ServerAddresses.GetValue(1)
                write-Output $Output

                If (($NicInterface.ServerAddresses.GetValue(0) -like $OldDns1 -or $NicInterface.ServerAddresses.getvalue(0) -like $OldDns2) -and ($NicInterface.ServerAddresses.getvalue(1) -like $oldDns1 -or $NicInterface.ServerAddresses.getvalue(1) -like $OldDns2))
                {
                    #If the IP address is DHCP assignd, leave it alone,
                    #that's handled via DHCP policies on the MAC address
                    $GetNetIPInfo = Get-NetIpAddress -InterfaceIndex  $NicInterface.InterfaceIndex
                     if ($GetNetIPInfo.PrefixOrigin -like "DHCP")
                     {
                        $VM                   
                        write-output "DHCP address - leave it alone"
                     }
                     Else
                     {
                         $IPAddresses = $GetNetIPInfo.IPv4Address
                         $FirstIP = $IPAddresses[1] #1 based array
                 
                         if (IsLastOctetEven($FirstIP)){
                            $VM
                            write-output "EVEN 4th IP octet => so DNS search order becomes $NewDns1 , $NewDns2"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns1,$NewDns2)
                         }
                         else
                         {   
                            $VM
                            write-Output "ODD 4th IP octet => so DNS search order becomes $NewDns2 , $NewDns1"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns2, $NewDns1)
                         } 
                         $NicInterface |  Select-Object -ExpandProperty ServerAddresses    
                     }
                }
                else
                {
                    $VM
                    write-Output "Existing DNS values not like expected old values. They are propably already changed"
                }        
            }
        }
    }
}
Stop-Transcript

 

Optimizing Backups: PowerShell Script To Move All Virtual Machines On A Cluster Shared Volume To The Node Owing That CSV

When you are optimizing the number of snapshots to be taken for backups or are dealing with storage vendors software that leveraged their hardware VSS provider you some times encounter some requirements that are at odds with virtual machine mobility and dynamic optimization.

For example when backing up multiple virtual machines leveraging a single CSV snapshot you’ll find that:

  • Some SAN vendor software requires that the virtual machines in that job are owned by the same host or the backup will fail.
  • Backup software can also require that all virtual machines are running on the same node when you want them to be be protected using a single CSV snap shot. The better ones don’t let the backup job fail, they just create multiple snapshots when needed but that’s less efficient and potentially makes you run into issues with your hardware VSS provider.

image

VEEAM B&R v8 in action … 8 SQL Server VMs with multiple disks on the same CSV being backed up by a single hardware VSS writer snapshot (DELL Compellent 6.5.20 / Replay Manager 7.5) and an off host proxy Organizing & orchestrating backups requires some effort, but can lead to great results.

Normally when designing your cluster you balance things out a well as you can. That helps out to reduce the needs for constant dynamic optimizations. You also make sure that if at all possible you keep all files related to a single VM together on the same CSV.

Naturally you’ll have drift. If not you have a very stable environment of are not leveraging the capabilities of your Hyper-V cluster. Mobility, dynamic optimization, high to continuous availability are what we want and we don’t block that to serve the backups. We try to help out to backups as much a possible however. A good design does this.

If you’re not running a backup every 15 minutes in a very dynamic environment you can deal with this by live migrating resources to where they need to be in order to optimize backups.

Here’s a little PowerShell snippet that will live migrate all virtual machines on the same CSV to the owner node of that CSV. You can run this as a script prior to the backups starting or you can run it as a weekly scheduled task to prevent the drift from the ideal situation for your backups becoming to huge requiring more VSS snapshots or even failing backups. The exact approach depends on the storage vendors and/or backup software you use in combination with the needs and capabilities of your environment.

cls

$Cluster = Get-Cluster
$AllCSV = Get-ClusterSharedVolume -Cluster $Cluster

ForEach ($CSV in $AllCSV)
{
    write-output "$($CSV.Name) is owned by $($CSV.OWnernode.Name)"
    
    #We grab the friendly name of the CSV
    $CSVVolumeInfo = $CSV | Select -Expand SharedVolumeInfo
    $CSVPath = ($CSVVolumeInfo).FriendlyVolumeName

    #We deal with the \ being and escape character for string parsing.
    $FixedCSVPath = $CSVPath -replace '\\', '\\'

    #We grab all VMs that who's owner node is different from the CSV we're working with
    #From those we grab the ones that are located on the CSV we're working with
      $VMsToMove = Get-ClusterGroup | ? {($_.GroupType –eq 'VirtualMachine') -and ( $_.OwnerNode -ne $CSV.OWnernode.Name)} | Get-VM | Where-object {($_.path -match $FixedCSVPath)} 
     
    ForEach ($VM in $VMsToMove)

    {
        write-output "`tThe VM $($VM.Name) located on $CSVPath is not running on host $($CSV.OwnerNode.Name) who owns that CSV"
        write-output "`tbut on $($VM.Computername). It will be live migrated."
        #Live migrate that VM off to the Node that owns the CSV it resides on
        Move-ClusterVirtualMachineRole -Name $VM.Name -MigrationType Live -Node $CSV.OWnernode.Name
    }

Now there is a lot more to discuss, i.e. what and how to optimize for virtual machines that are clustered. For optimal redundancy you’ll have those running on different nodes and CSVs. But even beyond that, you might have the clustered VMs running on different cluster, which is the failure domain.  But I get the remark my blogs are wordy and verbose so … that’s for another time Winking smile