How Windows Server 2012 R2 backups differ from Windows Server 2012 and earlier
You’ll remember our previous blog about an error when backing up a virtual machine on Windows Server 2012 R2, throwing this error:
Dealing With Event ID 10103 “The virtual machine ‘VM001’ cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID DCFE14D3-7E08-845F-9CEE-21E0605817DC)” In Windows Server 2012 R2
The fix was easy enough, adding a virtual SCSI controller to the virtual machine. But why does it need that now?
Well, this all has to do with the changed way Windows Server 2012 R2 backups work. Before Windows Server 20012 R2 the VSS provider created a VSS snapshot inside the guest virtual machine. That snapshot was exposed to the host, to create a volume snapshot for backup purposes. Right after the volume snapshot has been taken this VSS snapshot inside the guest virtual machine needed to be reverted. The backups then run against that volume snapshot and is consistent thanks to both host & guest VSS capabilities.
For an overview of VSS based backup process in general take a peak at Overview of Processing a Backup Under VSS
Now it is the “Hyper-V Integration Services Shadow Copy Provider” that is being used. When the the host initiates a volume snapshot (Microsoft or hardware VSS provider) the host VSS writer goes in to freeze. This process leverages the Hyper-V Integration Services Shadow Copy Provider to create the virtual machine checkpoint. After that the volume/LUN/CSV snapshot is taken. When that is done the host VSS writes goes into thaw and the virtual machine checkpoint is deleted. After that the backup runs against the Volume snapshot and at the end that is also deleted. You can follow this process quite nicely in the GUI of your Hyper-V host, you SAN (if you use a Hardware VSS provider).
Dear storage vendors: a great, reliable, fast VSS Hardware Provider is paramount to success in a Microsoft environment. You need to get this absolutely right and out of the door before spending any more time and money on achieving yet more IOPS. Keep scalability in mind when doing this.
Dear backup software vendors: think about the scalability when designing your products. If we have 200 or 500 or a thousand VMs … can we leverage CSV based backups to protect every VM on the LUN or do we need to snap the LUN for every VM backed up? Choice there is good for both data protection schemes and scalability.
At this stage the hardware VSS snapshot is being taken …
Contrary to common belief this means that the backup will indeed application consistent to the time of the checkpoint as the CSV snapshot being taken is of a consistent checkpoint. It’s the delta in the active avhdx that is only crash consistent, like any running VM by the way. Now pay attention to the screenshot below. The two red arrows are indicating to ntfs source events, two volumes seem to be exposed to the next free drive letters. E: and F: here as C: is the virtual machine OS and D: the DVD.
Look at the detail. Indeed two. Well it the previous screenshot we only saw one in the CSV path but there are two avhdx files indeed.
![image_thumb[1]](https://blog.workinghardinit.work/wp-content/uploads/2014/02/image_thumb110.png "image_thumb[1]")
Exposing a snapshot on the SAN to a server actually shows us this much better … look here at the avhdx with the GUID and one with “AutoRecovery” in the name. So that makes for two nfts events … and as the backup needs to do this life it requires a vSCSI controller to be present in the virtual machine … and vIDE controller can’t do this.
![image_thumb[3]](https://blog.workinghardinit.work/wp-content/uploads/2014/02/image_thumb311.png "image_thumb[3]")
Anyway, enough under the hood detective work for now, In VEEAM that stage looks like this:
And on the Compellent it looks like this. The screenshots are from different backups at different times so don’t get confused about the time stamps here. It’s just as illustration of what you can expect to see.
Now when the CSV snapshot has been taken the virtual machine checkpoint is removed. At that time the backup runs against the CSV snapshot. In our case (hardware VSS provider) this is a snapshot on the SAN that gets exposed in a view and mapped to the off host backup proxy VEEAM server. On the DELL Compellent it looks like this.
This takes a while to o…but after a while the backup will kick off. Do not that the checkpoint has merged and is no longer visible at this time.
Once the backup is complete, the mapping is removed, the view deleted and the snapshot expired. So your SAN is left as the backup found it.
There you go. I hope this helped clarify certain things on how Hyper-V guest backups work in Windows 2012 R2. So your backups are still application consistent, just not when you’re running Linux or DOS or NT4.0 as there is no support / VSS for that. However they are based on a consistent virtual machine snapshot which explains why Hyper-V backups can protect Linux guests very adequately!
Great, adds nicely to @joe_elway http://www.aidanfinn.com/?p=15759. Why are you using HW providers, I thought those aren’t needed from WS2012 onward?
Speed, reduce resource load on hosts, transportable snapshots for off host backups …
Does the snapshotting of the guest pose any risk for Virtual DC’s that happen to be pre-2012?
They are only used to create a LUN snapshot needed to bakup the VM and are not applied or even kept around to be applied long term so no. Restoring DCs, whatever version, requires you to follow any rules that apply
Pingback: Markus Bäker » DPM 2012 R2: Abbrüche bei Sicherung auf Hyper-V Hosts
Awesome Info….We have a similar setup, but using DPM 2012. I have question regarding how you place your VM’s in the cluster, For example if i am starting a backup job for 10VM’s that are in CSV1, do all the 10 VM’s need to be on one cluster node during the backup ? at the moment unless they are on the same node, we are finding it hard to get successful backups…..when you have hundreds of VM’s it gets bit harder to do that
Thank you 🙂
Great article and the inclusion of the VM snapshot in the VSS process is a great enhancement. I do however have a question for you:
– If you wanted to mount the array snapshot and bring the VM online in an application consistent state, do you have to roll the autorecovery.avhdx into the parent vhdx to achieve this?
– If so, what data is inside the autorecovery.avhdx? Is it system state data from within the VM etc?
I have a blog post in the queue with some more background on the whole autorecovery which in VSS backups isn’t that new. The way that Hyper-V uses checkpoints & leverages autorecovery is. The backup app should handle everything for you but if you go directly to the snaps on a SAN you might notice things are not application consistent depending on the version of the guest OS. If you create a new VM form a VHD(X) You should be ok. With the avhdx not so much, that’s crash consistent and booting from it should show you that. If you want to merge it will be manually and then you have to create a VM from the resulting VHD(X). Note that you should not have to mess with the autorecovery.avhx. It’s a huge topic in itself and I haven’t tested all possible OS versions & permutations either, there’s other work to be done 🙂
Thanks, I suspected that was the case. I am testing with DiskShadow and hence the question. 🙂
Hi we have an Equalogic and now with the Hardware Provider the Backup Process locks like the first Picture. With 2012 Hyper-V it makes Hardware Snapshot on the Equalogic equal the Dell Compelled Picture. What is the right way now? So the Process is completely different from 2012 Hype-V Backup? Is this true?
Thanks for your answer in advantage
René
The way Windows Hyper-V does backups has indeed changed in Windows 2012 R2. This is seen reflected in the files in a snapshot. There is no wrong or right way, it just works differently.
Thanks for this clarification
Good Day,
We are a small business that is just getting into the virtual environment due to most of our servers running 2003. We are running into all kinds of headaches with going virtual. We (the boss) have chose to go with Hyper-V as our first server migrations and we have a good server with two VM’s running but now I am getting in the management console the Best Practices saying I need to backup the VM’s weekly.
Have been trying to find a solution that is built in or at lease from Microsoft since it is there product and they are suggesting to do the backups. But mostly what I am finding is 3rd party applications.
Looked at the Powershell code but I am not that familiar with Powershell and the almost 600 likes of coding, that would have to be modified, is a little intimidating.
So I have come to the conclusion that we will have to get a third party application but by boss if very unhappy about it. So I am asking is there any easy way to do backups that Microsoft recommends with Server 2012 or other utility?
Thank you for your time.
Mark M.
You can use the in box Windows Backup in 2012 (R2): see http://blogs.msdn.com/b/virtual_pc_guy/archive/2013/02/18/windows-backup-and-hyper-v-in-server-2012.aspx and http://virtualizationreview.com/blogs/virtual-insider/2013/02/back-up-hyper-v-vms.aspx
VEEAM has a free product for small environments and works very well … try it out: http://hyperv.veeam.com/free-hyper-v-backup/
cheers
haha awsome article! we have exactly the same combo (2012 R2 + VEEAM + Compellent) :>
Hi there workinghardinit,
Having massive problems with my Windows 2012 Server Guest HyperV VM at the minute and slightly clutching at straws but getting nowhere with it at the minute so wondered if you had any ideas / experience on this / could help.
I created a brand new VM on Hyper-V installed all software OS Office VS etc. and it worked like a dream for 2/3 days. I think then I got prompted for a windows update and to restart and ever since I have not been able to sucessfully boot it. Mostly it just sits with the windows logo on bootup but never gets anywhere. Sometimes it goes as far as the windows options – Troubleshoot your pc, Continue to Windows Server 2012 but whichever you choose you still go back to the logo or it asks for windows admin options.
I’ve spent ages googling on this, and just to let you know, I cannot start any of the Hyper-V synchronisation, timeout, heartbeat etc. services other than management. I get a service started but stopped error when I try to. In windows log also on the host I get a “The Data Exchange integration service is either not enabled” which is what prompted me to first look at the services. I’ve tried disabling all integration services in Hyper-V setttings as well, powering off, and even stopping the management services but all to no avail – still all fails to boot as above. Short of trying to create another new vm, installing do you have any ideas where to go next ? Any ideas would be greatly appreciated as I am at a loss.
Recover from backup or rebuild and recover data from backup. That will cost less time and money than any other option.
I should also add the host is windows 8.1 if that helps in anyway.
Pingback: Hyper-V backup challenges Windows Server 2016 needs to address | StarWind Software
Pingback: Remove Lingering Backup Checkpoints from a Hyper-V Virtual Machine - Working Hard In IT