Recently in a talk with a pen tester I was demoing an end-user security risk that is relatively new on the scene. Apps that automatically confirm MFA push notifications. This effectively bypasses conscious user interaction and approval of any login attempt secured with a push notification. Hence the Darwin award with MFA push notifications phrase was born.
The Darwin award with MFA push notifications
Just when some security people worried more about the people with push notification suffering from security fatigue as being the biggest risk we go step further. Never mind people accepting any notification like Pavlov’s dog in a semi-unconscious, conditioned action. They have even grown tired of this an turn to MFA bypass apps to handle this for them.
More then ever it seems that disabling any kind of self-service for device registration with MFA is key. On top of that, it is a sobering reminder that a strong password and conscious user actions are still very much key to providing security via MFA. I am not bashing DUO here. This was just the one I tested and it worked shockingly well.
Conclusion
I think for some people and organizations one or more FIDO2 keys will be the better option. Unless mobile device management can prevent people from installing auto-responder apps for push notifications you might have an issue. Or, they need to find a way to block such tools. Whatever I can come up with breaks the ease of use of push notifications but there are smarter and more knowledgeable people out there than me, so who know what they come up with. Microsoft Authenticator seems to have some capabilities to prevent this. I don’t know if you can enforce it 100% and if this cannot be bypassed in code as well.
This, however, does nothing against conditioned responses of pushing a button or scanning a fingerprint on a FIDO2 key. So, remain vigilant. The sobering fact is that the adoption of MFA is disappointingly low. And no matter how many scary MFA bypass stories your read MFA is a key aspect of securing access today. It puts you far ahead of the curve. If done well and with well thought out procedures it is a formidable barrier for but the most determined attackers. Actually MFA bypass attacks are very rare still. Most of us are not that interesting targets but it can help keep out the majority of casual or professional thieves looking for quick wins on easy targets.
In Shared nothing live migration with a virtual switch change and VLAN ID configuration I published a sample script. The script works well. But there are two areas of improvement. The first one is here in Checkpoint references a non-existent virtual switch. This post is about the second one. Here I show that I also need to check the virtual switch QoS mode during migrations. A couple of the virtual machines on the source nodes had absolute minimum and/or maximum bandwidth set. On the target nodes, all the virtual switches are created by PowerShell. This defaults to weight mode for QoS, which is the more sensible option, albeit not always the easiest or practical one for people to use.
Virtual switch QoS mode during migrations
First a quick recap of what we are doing. The challenge was to shared nothing live migrate virtual machines to a host with different virtual switch names and VLAN IDs. We did so by adding dummy virtual switches to the target host. This made share nothing live migration possible. On arrival of the virtual machine on the target host, we immediately connect the virtual network adapters to the final virtual switch and set the correct VLAN IDs. That works very well. You drop 1 or at most 2 pings, this is as good as it gets.
This goes wrong under the following conditions:
The source virtual switch has QoS mode absolute.
Virtual network adapter connected to the source virtual switch has MinimumBandwidthAbsolute and/or MaximumBandwidth set.
The target virtual switch with QoS mode weighted
This will cause connectivity loss as you cannot set absolute values to a virtual network attached to a weighted virtual switch. So connecting the virtual to the new virtual switch just fails and you lose connectivity. Remember that the virtual machine is connected to a dummy virtual switch just to make the live migration work and we need to swap it over immediately. The VLAN ID does get set correctly actually. Let’s deal with this.
Steps to fix this issue
First of all, we adapt the script to check the QoS mode on the source virtual switches. If it is set to absolute we know we need to check for any settings of MinimumBandwidthAbsolute and MaximumBandwidth on the virtual adapters connected to those virtual switches. These changes are highlighted in the demo code below.
Secondly, we adapt the script to check every virtual network adapter for its bandwidth management settings. If we find configured MinimumBandwidthAbsolute and MaximumBandwidth values we set these to 0 and as such disable the bandwidth settings. This makes sure that connecting the virtual network adapters to the new virtual switch with QoS mode weighted will succeed. These changes are highlighted in the demo code below.
Finally, the complete script
#The source Hyper-V host
$SourceNode = 'NODE-A'
#The LUN where you want to storage migrate your VMs away from
$SourceRootPath = "C:\ClusterStorage\Volume1*"
#The source Hyper-V host
#The target Hypr-V host
$TargetNode = 'ZULU'
#The storage pathe where you want to storage migrate your VMs to
$TargetRootPath = "C:\ClusterStorage\Volume1"
$OldVirtualSwitch01 = 'vSwitch-VLAN500'
$OldVirtualSwitch02 = 'vSwitch-VLAN600'
$NewVirtualSwitch = 'ConvergedVirtualSwitch'
$VlanId01 = 500
$VlanId02 = 600
if ((Get-VMSwitch -name $OldVirtualSwitch01 ).BandwidthReservationMode -eq 'Absolute') {
$OldVirtualSwitch01QoSMode = 'Absolute'
}
if ((Get-VMSwitch -name $OldVirtualSwitch01 ).BandwidthReservationMode -eq 'Absolute') {
$OldVirtualSwitch02QoSMode = 'Absolute'
}
#Grab all the VM we find that have virtual disks on the source CSV - WARNING for W2K12 you'll need to loop through all cluster nodes.
$AllVMsOnRootPath = Get-VM -ComputerName $SourceNode | where-object { $_.HardDrives.Path -like $SourceRootPath }
#We loop through all VMs we find on our SourceRoootPath
ForEach ($VM in $AllVMsOnRootPath) {
#We generate the final VM destination path
$TargetVMPath = $TargetRootPath + "\" + ($VM.Name).ToUpper()
#Grab the VM name
$VMName = $VM.Name
$VM.VMid
$VMName
#If the VM is still clusterd, get it removed form the cluster as live shared nothing migration will otherwise fail.
if ($VM.isclustered -eq $True) {
write-Host -ForegroundColor Magenta $VM.Name "is clustered and is being removed from cluster"
Remove-ClusterGroup -VMId $VM.VMid -Force -RemoveResources
Do { Start-Sleep -seconds 1 } While ($VM.isclustered -eq $True)
write-Host -ForegroundColor Yellow $VM.Name "has been removed from cluster"
}
#If the VM checkpoint, notify the user of the script as this will cause issues after swicthing to the new virtual
#switch on the target node. Live migration will fail between cluster nodes if the checkpoints references 1 or more
#non existing virtual switches. These must be removed prior to of after completing the shared nothing migration.
#The script does this after the migration automatically, not before as I want it to be untouched if the shared nothing
#migration fails.
$checkpoints = get-vmcheckpoint -VMName $VM.Name
if ($Null -ne $checkpoints) {
write-host -foregroundcolor yellow "This VM has checkpoints"
write-host -foregroundcolor yellow "This VM will be migrated to the new host"
write-host -foregroundcolor yellow "Only after a succesfull migration will ALL the checpoints be removed"
}
#Do the actual storage migration of the VM, $DestinationVMPath creates the default subfolder structure
#for the virtual machine config, snapshots, smartpaging & virtual hard disk files.
Move-VM -Name $VMName -ComputerName $VM.ComputerName -IncludeStorage -DestinationStoragePath $TargetVMPath -DestinationHost $TargetNode
$MovedVM = Get-VM -ComputerName $TargetNode -Name $VMName
$vNICOnOldvSwitch01 = Get-VMNetworkAdapter -ComputerName $TargetNode -VMName $MovedVM.VMName | where-object SwitchName -eq $OldVirtualSwitch01
if ($Null -ne $vNICOnOldvSwitch01) {
foreach ($VMNetworkadapater in $vNICOnOldvSwitch01) {
if ($OldVirtualSwitch01QoSMode -eq 'Absolute') {
if (0 -ne $VMNetworkAdapter.bandwidthsetting.Maximumbandwidth) {
write-host -foregroundcolor cyan "Network adapter $VMNetworkAdapter.Name of VM $VMName MaximumBandwidth will be reset to 0."
Set-VMNetworkAdapter -Name $VMNetworkAdapter.Name -VMName $MovedVM.Name -ComputerName $TargetNode -MaximumBandwidth 0
}
if (0 -ne $VMNetworkAdapter.bandwidthsetting.MinimumBandwidthAbsolute) {
write-host -foregroundcolor cyan "Network adapter $VMNetworkAdapter.Name of VM $VMName MaximuBandwidthAbsolute will be reset to 0."
Set-VMNetworkAdapter -Name $VMNetworkAdapter.Name -VMName $MovedVM.Name -ComputerName $TargetNode -MinimumBandwidthAbsolute 0
}
}
write-host 'Moving to correct vSwitch'
Connect-VMNetworkAdapter -VMNetworkAdapter $vNICOnOldvSwitch01 -SwitchName $NewVirtualSwitch
write-host "Setting VLAN $VlanId01"
Set-VMNetworkAdapterVlan -VMNetworkAdapter $vNICOnOldvSwitch01 -Access -VLANid $VlanId01
}
}
$vNICsOnOldvSwitch02 = Get-VMNetworkAdapter -ComputerName $TargetNode -VMName $MovedVM.VMName | where-object SwitchName -eq $OldVirtualSwitch02
if ($NULL -ne $vNICsOnOldvSwitch02) {
foreach ($VMNetworkadapater in $vNICsOnOldvSwitch02) {
if ($OldVirtualSwitch02QoSMode -eq 'Absolute') {
if ($Null -ne $VMNetworkAdapter.bandwidthsetting.Maximumbandwidth) {
write-host -foregroundcolor cyan "Network adapter $VMNetworkAdapter.Name of VM $VMName MaximumBandwidth will be reset to 0."
Set-VMNetworkAdapter -Name $VMNetworkAdapter.Name -VMName $MovedVM.Name -ComputerName $TargetNode -MaximumBandwidth 0
}
if ($Null -ne $VMNetworkAdapter.bandwidthsetting.MinimumBandwidthAbsolute) {
write-host -foregroundcolor cyan "Network adapter $VMNetworkAdapter.Name of VM $VMName MaximumBandwidth will be reset to 0."
Set-VMNetworkAdapter -Name $VMNetworkAdapter.Name -VMName $MovedVM.Name -ComputerName $TargetNode -MinimumBandwidthAbsolute 0
}
}
write-host 'Moving to correct vSwitch'
Connect-VMNetworkAdapter -VMNetworkAdapter $vNICsOnOldvSwitch02 -SwitchName $NewVirtualSwitch
write-host "Setting VLAN $VlanId02"
Set-VMNetworkAdapterVlan -VMNetworkAdapter $vNICsOnOldvSwitch02 -Access -VLANid $VlanId02
}
}
#If the VM has checkpoints, this is when we remove them.
$checkpoints = get-vmcheckpoint -ComputerName $TargetNode -VMName $MovedVM.VMName
if ($Null -ne $checkpoints) {
write-host -foregroundcolor yellow "This VM has checkpoints and they will ALL be removed"
$CheckPoints | Remove-VMCheckpoint
}
}
Below is the output of a VM where we had to change the switch name, enable a VLAN ID, deal with absolute QoS settings and remove checkpoints. All this without causing downtime. Nor did we change the original virtual machine in case the shared nothing migration fails.
Some observations
The fact that we are using PowerShell is great. You can only set weighted bandwidth limits via PowerShell. The GUI is only for absolute values and it will throw an error trying to use it when the virtual switch is configured as weighted.
This means you can embed setting the weights in your script if you so desire. If you do, read up on how to handle this best. trying to juggle the weight settings to be 100 in a dynamic environment is a bit of a challenge. So use the default flow pool and keep the number of virtual network adapters with unique settings to a minimum.
Conclusion
To avoid downtime we removed all the set minimum and maximum bandwidth settings on any virtual network adapter. By doing so we ensured that the swap to the new virtual switch right after the successful shared nothing live migration will succeed. If you want you can set weights on the virtual network adapters afterward. But as the bandwidth on these new hosts is now a redundant 25 Gbps, the need was no longer there. As a result, we just left them without. this can always be configured later if it turns out to be needed.
Warning: this is a demo script. It lacks error handling and logging. It can also contain mistakes. But hey you get it for free to adapt and use. Test and adapt this in a lab. You are responsible for what you do in your environments. Running scripts downloaded from the internet without any validation make you a certified nut case. That is not my wrongdoing.
I hope this helps some of you. thanks for reading.
VeeamON Virtual 2019 takes place on November 20th, 2019. This is a global event. You can join this virtual conference from the comfort of your home office, couch or whatever spot you’d like to enjoy it from.
Veeam will bring the virtual conference to you where ever you are. First of all, you will learn about the latest and greatest in data protection, security and management. In order to fully optimize your experience you can engage with Veeam’s® leading experts and partners. But you can also network with the more than 5,000 attendees. Joining this conference gives you exclusive access to all the sessions. As a result, the content will keep you on top of the new innovations, tools and techniques needed today.
Experts from around the globe are volunteering their time
Meanwhile, along with the presentations and discussions, you will have the opportunity to ask questions. Professionals from around the globe are volunteering their time. I will be there to help out with Hyper-V related questions. You can discuss anything ranging from network, storage, compute to architecture and design.
Likewise, if you want to discuss Off-Host proxies or Scale Out Backup repositories, just ask. Need to talk about general backup architectures, designs and migrations? Feel free to engage us for a chat. That is real world experience and expertise to augment the value from VeeamON Virtual 2019 for you!
This blog post is actually one of the two additions or improvements I need to make the sample code in a previous blog post. In Shared nothing live migration with a virtual switch change and VLAN ID configuration I published a sample script. The script works well. But there are to areas of improvement. This is the first one. I should check for existing checkpoints before migrating a virtual machine. When a checkpoint references a non-existent virtual switch it prevents live migration.
Why check for existing checkpoints before migrating a virtual machine.
If you read the previous blog, you know the challenge was to shared nothing live migrate virtual machines to a host with different virtual switch names and VLAN IDs. We did so by adding dummy virtual switches to the target host. This made share nothing live migration possible. On arrival of the virtual machine on the target host, we immediately connect the virtual machine to the final virtual switch and set the correct VLAN IDs. That works very well. You drop 1 or at most 2 pings, this is as good as it gets.
However, this can lead to a problem with live migration between cluster nodes. The temporary dummy virtual switches are removed. They were only needed to perform the shared nothing live migration. But any checkpoint of a virtual machine that existed before that still refers to them. When we now try to live migrate that virtual machine it fails as the virtual switches do not exist on the target host.
Preventing live migration issues
So the first addition to the script is to add a check for existing checkpoints. The script can offer to remove the checkpoint, skip the virtual machine or continue. When you continue you should realize you will need to remove the checkpoint at the target if the virtual switches no longer exist on all cluster nodes.
Asking for a decision is an interactive process, so that is not optimal. Noted, most of us will sort of babysit such a script. But still, so adding a warning and logging it is maybe the better choice. The most important this is to be aware of this. In the improved demo script above I check for checkpoints, notify of their presence and then after a successful shared nothing migration I delete them (high lighted code lines).
#The source Hyper-V host
$SourceNode = 'NODE-A'
#The LUN where you want to storage migrate your VMs away from
$SourceRootPath = "C:\ClusterStorage\Volume1*"
#The source Hyper-V host
#The target Hypr-V host
$TargetNode = 'ZULU'
#The storage pathe where you want to storage migrate your VMs to
$TargetRootPath = "C:\ClusterStorage\Volume1"
$OldVirtualSwitch01 = 'vSwitch-VLAN500'
$OldVirtualSwitch02 = 'vSwitch-VLAN600'
$NewVirtualSwitch = 'ConvergedVirtualSwitch'
$VlanId01 = 500
$VlanId02 = 600
#Grab all the VM we find that have virtual disks on the source CSV - WARNING for W2K12 you'll need to loop through all cluster nodes.
$AllVMsOnRootPath = Get-VM -ComputerName $SourceNode | where-object { $_.HardDrives.Path -like $SourceRootPath }
#We loop through all VMs we find on our SourceRoootPath
ForEach ($VM in $AllVMsOnRootPath){
#We generate the final VM destination path
$TargetVMPath = $TargetRootPath + "\" + ($VM.Name).ToUpper()
#Grab the VM name
$VMName = $VM.Name
$VM.VMid
$VMName
#If the VM is still clusterd, get it removed form the cluster as live shared nothing migration will otherwise fail.
if ($VM.isclustered -eq $True) {
write-Host -ForegroundColor Magenta $VM.Name "is clustered and is being removed from cluster"
Remove-ClusterGroup -VMId $VM.VMid -Force -RemoveResources
Do { Start-Sleep -seconds 1 } While ($VM.isclustered -eq $True)
write-Host -ForegroundColor Yellow $VM.Name "has been removed from cluster"
}
#If the VM checkpoint, notify the user of the script as this will cause issues after swicthing to the new virtual
#switch on the target node. Live migration will fail between cluster nodes if the checkpoints references 1 or more
#non existing virtual switches. These must be removed prior to of after completing the shared nothing migration.
#The script does this after the migration automatically, not before as I want it to be untouched if the shared nothing
#migration fails.
$checkpoints = get-vmcheckpoint -VMName $VM.Name
if ($Null -ne $checkpoints)
{
write-host -foregroundcolor yellow "This VM has checkpoints"
write-host -foregroundcolor yellow "This VM will ne migrated to the new host"
write-host -foregroundcolor yellow "Only after a succesfull migration will ALL the checpoints be removed"
}
#Do the actual storage migration of the VM, $DestinationVMPath creates the default subfolder structure
#for the virtual machine config, snapshots, smartpaging & virtual hard disk files.
Move-VM -Name $VMName -ComputerName $VM.ComputerName -IncludeStorage -DestinationStoragePath $TargetVMPath -DestinationHost $TargetNode
$MovedVM = Get-VM -ComputerName $TargetNode -Name $VMName
$OldvSwitch01 = Get-VMNetworkAdapter -ComputerName $TargetNode -VMName $MovedVM.VMName | where-object SwitchName -eq $OldVirtualSwitch01
if ($Null -ne $OldvSwitch01) {
foreach ($VMNetworkadapater in $OldvSwitch01)
{ write-host 'Moving to correct vSwitch'
Connect-VMNetworkAdapter -VMNetworkAdapter $OldvSwitch01 -SwitchName $NewVirtualSwitch
write-host "Setting VLAN $VlanId01"
Set-VMNetworkAdapterVlan -VMNetworkAdapter $OldvSwitch01 -Access -VLANid $VlanId01
}
}
$OldvSwitch02 = Get-VMNetworkAdapter -ComputerName $TargetNode -VMName $MovedVM.VMName | where-object SwitchName -eq $OldVirtualSwitch02
if ($NULL -ne $OldvSwitch02) {
foreach ($VMNetworkadapater in $OldvSwitch02) {
write-host 'Moving to correct vSwitch'
Connect-VMNetworkAdapter -VMNetworkAdapter $OldvSwitch02 -SwitchName $NewVirtualSwitch
write-host "Setting VLAN $VlanId02"
Set-VMNetworkAdapterVlan -VMNetworkAdapter $OldvSwitch02 -Access -VLANid $VlanId02
}
}
#If the VM has checkpoints, this is when we remove them.
$checkpoints = get-vmcheckpoint -ComputerName $TargetNode -VMName $MovedVM.VMName
if ($Null -ne $checkpoints)
{
write-host -foregroundcolor yellow "This VM has checkpoints and they will ALL be removed"
$CheckPoints | Remove-VMCheckpoint
}
}
Conclusion
The issue, in this case, was fall out from our creative shared nothing live migration. But there are other ways to get into this situation. The takeaway is that we should be aware of the live migration issue when a checkpoint references a non-existent virtual switch. I hope this helps someone out there. The next blog will be about another hiccup you might come across when performing a shared nothing live migration. Thanks for reading.