Introduction
It’s not a secret that while guest clustering with VHDSets works very well. We’ve had some struggles in regards to host level backups however. Right now I leverage Veeam Agent for Windows (VAW) to do in guest backups. The most recent versions of VAW support Windows Failover Clustering. I’d love to leverage host level backups but I was struggling to make this reliable for quite a while. As it turned out recently there are some virtual machine permission issues involved we need to fix. Both Microsoft and Veeam have published guidance on this in a KB article. We automated correcting the permissions on the folder with VHDS files & checkpoints for host level Hyper-V guest cluster backup
The KB articles
Early August Microsoft published KB article with all the tips when thins fail Errors when backing up VMs that belong to a guest cluster in Windows. Veeam also recapitulated on the needed conditions and setting to leverage guest clustering and performing host level backups. The Veeam article is Backing up Hyper-V guest cluster based on VHD set. Read these articles carefully and make sure all you need to do has been done.
For some reason another prerequisite is not mentioned in these articles. It is however discussed in ConfigStoreRootPath cluster parameter is not defined and here https://docs.microsoft.com/en-us/powershell/module/hyper-v/set-vmhostcluster?view=win10-ps You will need to set this to make proper Hyper-V collections needed for recovery checkpoints on VHD Sets. It is a very unknown setting with very little documentation.
But the big news here is fixing a permissions related issue!
The latest addition in the list of attention points is a permission issue. These permissions are not correct by default for the guest cluster VMs shared files. This leads to the hard to pin point error.
Error Event 19100 Hyper-V-VMMS 19100 ‘BackupVM’ background disk merge failed to complete: General access denied error (0x80070005). To fix this issue, the folder that holds the VHDS files and their snapshot files must be modified to give the VMMS process additional permissions. To do this, follow these steps for correcting the permissions on the folder with VHDS files & checkpoints for host level Hyper-V guest cluster backup.
Determine the GUIDS of all VMs that use the folder. To do this, start PowerShell as administrator, and then run the following command:
get-vm | fl name, id
Output example:
Name : BackupVM
Id : d3599536-222a-4d6e-bb10-a6019c3f2b9b
Name : BackupVM2
Id : a0af7903-94b4-4a2c-b3b3-16050d5f80f
For each VM GUID, assign the VMMS process full control by running the following command:
icacls <Folder with VHDS> /grant “NT VIRTUAL MACHINE\<VM GUID>”:(OI)F
Example:
icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant “NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2”:(OI)F
icacls “c:\ClusterStorage\Volume1\SharedClusterDisk” /grant “NT VIRTUAL MACHINE\d3599536-222a-4d6e-bb10-a6019c3f2b9b”:(OI)F
My little PowerShell script
As the above is tedious manual labor with a lot of copy pasting. This is time consuming and tedious at best. With larger guest clusters the probability of mistakes increases. To fix this we write a PowerShell script to handle this for us.
#Didier Van Hoye
#Twitter: @WorkingHardInIT
#Blog: https://blog.Workinghardinit.work
#Correct shared VHD Set disk permissions for all nodes in guests cluster
$GuestCluster = "DemoGuestCluster"
$HostCluster = "LAB-CLUSTER"
$PathToGuestClusterSharedDisks = "C:\ClusterStorage\NTFS-03\GuestClustersSharedDisks"
$GuestClusterNodes = Get-ClusterNode -Cluster $GuestCluster
ForEach ($GuestClusterNode in $GuestClusterNodes)
{
#Passing the cluster name to -computername only works in W2K16 and up.
#As this is about VHDS you need to be running 2016, so no worries here.
$GuestClusterNodeGuid = (Get-VM -Name $GuestClusterNode.Name -ComputerName $HostCluster).id
Write-Host $GuestClusterNodeGuid "belongs to" $GuestClusterNode.Name
$IcalsExecute = """$PathToGuestClusterSharedDisks""" + " /grant " + """NT VIRTUAL MACHINE\"+ $GuestClusterNodeGuid + """:(OI)F"
write-Host "Executing " $IcalsExecute
CMD.EXE /C "icacls $IcalsExecute"
}
Below is an example of the output of this script. It provides some feedback on what is happening.
Correcting the permissions on the folder with VHDS files & checkpoints for host level Hyper-V guest cluster backup
PowerShell for the win. This saves you some searching and typing and potentially making some mistakes along the way. Have fun. More testing is underway to make sure things are now predictable and stable. We’ll share our findings with you.
On line 23 you should replace
“C:\ClusterStorage\NTFS-03\GuestClustersSharedDisks”
with
$PathToGuestClusterSharedDisks
Fixed.
I’m assuming that for your script to work, the guest cluster and the underlying host cluster need to be on the same domain?
No, that all happens on the domain the host cluster is part of. But for Veeam B&R to work you’ll need to have your FW rules set up to allow for this.
Have you managed to get this working consistently now? I am at this cross road right now. 2 Server 2016 Hosts fully patched. 2 Server 2016 Guest VM’s running a SQL Cluster inside of it fully patched. VHD Sets used for shared disks. None of the disks are CSV’s, instead they are just shared disks.
Have worked with Veeam support and Microsoft and it’s incredible the lack of knowledge and acknowledgement to this issue.
We want to do Host Level backups. So how is that working these days and have you found it requires more than what is stated in your blogs?
Thanks for posting this and all the great feedback.
Yes, it works if you take care of all the notes I have left here overtime on this subject. One thing I still need to blog is about the Veeam registry fix for orphaned reference id of Hyper-V Backups. These can trip you up when you ever have to clean out collection checkpoints in HYper-V as they are then left in the VMCX. Alsoe see https://forums.veeam.com/microsoft-hyper-v-f25/increasing-vmcx-files-in-vms-that-are-veeam-replicated-t52948.html. this might have been ficed/added in U4a.
I am struggling to find a use case for a guest cluster when you cant use a CSV to share storage between the guests and you also want to be able to back it up (and what can you have that you want to be HA but never backed up?!).
We can’t have HA file servers, nor SQL FCI. Enterprise editions of SQL can use AG so don’t need shared storage anyway.
Veeam Agent reports it ignores CSV so that doesn’t even seem to be a saving grace.
General Purpose File Servers, with or without continuous availability are a good reason. Those do not require (actually a bad idea / not supported and not needed). Then there are other roles (services, scripts, apps) which I see less off. Guest clustering is not a finished subject and as apps become ever more inherently HA by design it might never be. Bu the fact it isn’t done and was problematic at the start hurts adoption as well.
I am trying to set permissions with this command
icacls “D:\Hyper-V\Main Server\VHDX Files” /grant “NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2”:(OI)F
but facing below error
OI : The term ‘OI’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:110
+ … grant “NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2”:(OI)F
+ ~~
+ CategoryInfo : ObjectNotFound: (OI:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
please guide
In our case (still working on this) is that when CBT is enabled in the Veeam Job, the system issues a vendor specific Checkpoint call [$set.GetText(“CimDtd20”) $resultCKPcreation = $c.CreateSnapshot($vmobject, $SetXML, 32768)]. But when we do it like how the GUI of HyperV would do it [$set.GetText(“CimDtd20”) $resultCKPcreation = $c.CreateSnapshot($vmobject, $SetXML, 2)] all is fine and the base disk is not in READ security state for the VM UUID when it wants to merge.
Very interesting. Is this a joint MSFT/Veeam case?
Not yet, Veeam pushed us to create a ticket with MSFT to find out. Since we have a active support contract with both, we are going to find out exactly who’s fault this is (tickets ref: MSFT #2109230040001977, Veeam #04987891). Will give a update when available.
Hi Sander, was there any final conclusion to your case?
Make sure you are running the icacls command in an elevated CMD prompt and not powershell!