A Last-ditch Effort
First of all you need to realize this might not work. It’s a last-ditch effort. There is a reason why you have backups (with tested restores) and why you should monitor your environment for things that are not as they should be. Early intervention can pay off.
Also, see blog post on a couple of more preferred actions.
If you have lost checkpoints, you have basically lost data and corruption/data inconsistencies are very much a possibility and reality. If the files have been copied and information about what file is the parent the dates/timestamps are what you have to go by. You might not know for sure if you have them all.
Setting up the demo
For demo purposes, we take a test VM and ad files to indicate what checkpoint we’re at.
We start with ORGINAL.TXT on the desktop and we create a checkpoint, which we rename accordingly.
We add a file called CHECK01.TXT and we create a checkpoint, which we rename accordingly.
We add a file called CHECK02.TXT and we create a checkpoint, which we rename accordingly.
We add a file called NOW.TXT no more checkpoints are taken.
The file names represent the content you’d see disappear if you applied the checkpoint and we have reflected this in the name for the checkpoints.
As we want to merge all the snapshots and end up with a usable VHDX we’ll work back from the most recent differencing disk until all is merged. As you can see this is a straightforward situation and I hope you’ll never be running having to deal with a vast collection of subtrees.
Finding out what are the parents of avhdx files
In this demo, it’s pretty obvious what snapshots exist and what avhdx files they represent. We’ve even shown you the single tree visualized in Hyper-V Manager. In reality, bad things have happened and you don’t see this information anymore. So you might have to find out yourself. This is done via inspect disk in Hyper-V manager. If you’re confused about what the parent is of (a)vhdx files this tool will help you find out or show you what the most recent one was.
Sometimes the original files have been renamed or moved and that it will show you the last known valid parent.
Manually Merging the checkpoints
Remember to make a copy of all files as a backup! Also, make sure you have enough free disk space … you need working space! You might need another shot at this. As we want to merge all the snapshots and end up with a usable vhdx we’ll work back from the most recent differencing disk until all is merged in the oldest one which is the vhdx. You can look at the last modified timestamps to find out the correct order in which to work. The most recent avhdx is the one used in the virtual machine configuration file and locate the information for the virtual hard disk.
The configuration file’s avhdx is the one containing the “NOW” running state of the VM.
Note: You might find some information that you need to rename the extension avhdx to vhdx (or avhd to vhd). The reason for this was that in Windows 2008 Hyper-V Manager did not show avhd files in the Edit virtual disk wizard. You can still do this and it will still works, but you do not need to. Ever since Windows Server 2008 R2 avhd (and with since Windows Server 2012 avhdx) files do show up in Hyper-V Managers Disk edit.
For some insights as to why the order is important read this blog by Ben Armstrong What happens when a snapshot is being merged? [Hyper-V]
WARNING: If you did not start with the most recent one and work your way down, which is the easiest and least confusing way all is not lost. But you will have to reconnect the first more recent (a)vhdx to one to its new parent. This is needed as by merging a snapshot out of order more recent one will have lost it’s will have lost its original parent.
Here’s how to do this: Select reconnect. This is the page you’ll get if you’d start edit disk wizard as all other options are unavailable due to the missing parent.
The wizard will tell you what used to be the parent and allow you to select a new one. Make sure to tick the check box for Ignore ID mismatch or the reconnect will fail as you’re previous out of order merge has created a new a(vhdx). If your in this pickle by renaming (a)vhdx files or after a copy this isn’t needed by the way.
Follow the wizard after that and when your done you can launch the edit disk wizard again and perform a merge. It’s paramount that you do not mix up orders when doing so that you reconnect to the parent this or you’ll end up in a right mess. There are many permutations, keep it simple!. Do it in order. If you start having multiple checkpoint trees/subtrees things can get confusing very fast.
You might also have to reconnect if the checkpoints have lost their connection the what they know to be their last parent for other reasons. In that case, you do this and when that’s done, you merge. Rinse and repeat. The below walk-through assumes you have no reconnects to be done. If so it will tell you like in the example above.
Walk through:
Open the Edit Disk Wizard
Select the most recent avhdx & click “Next”
We choose to merge the avhdx
In our case into its parent disk
Verify the options are correct and click “Finish”
Let the wizard complete
That’s it. You’ve merged the most recent snapshot into it’s parent. That means that you have not lost the most recent state of the virtual machine as when it was running before you shut it down. This can be verified by mounting the now most recent avhdx and looking at the desktop for my user profile. You can see the NOW.txt text file is there!
OK, dismount the avhdx and now it’s rinse and repeat.
You do this over and over again until your merge the last avhdx into the vhdx.
Than you have the vhdx you will use to create a new virtual machine.
Make sure you get the generation right.
Assign memory
Connect to the appropriate virtual switch or not if you’re not ready to do this yet
Use your vhdx disk that’s the remaining result of your merging efforts
When you boot that virtual machine you’ll see that all the text files are there. It’s as if you’ve deleted the checkpoints in the GUI and retained “NOW” in the vhdx.
Last but not least, you can use PowerShell or even DiskPart for this but I found that most people in this pickle value a GUI. Use what you feel most comfortable with.
Thanks for reading and hope this helps someone. Do remember “big boy” rules apply. This is not safe, easy or obvious in each and every situation so you are responsible for everything you do in your environment. If your in to deep, way over your head, etc. call in some expert help.
Thank you, it worked and saved us a lot of trouble 🙂
Glad to hear that. Thx for reading!
Hi, I’m using native virtualisation and differencing disks.
Each diff-disk (child) is created with parental reference.
If I have a base, and 3 diff disks below that, and I merge with a depth of 2 from the first child, is there any way of linking the last layer?
base
diff1 – drivers
diff2 – software (tested, keeping)
diff3 – other stuff (retain, but don’t merge)
merge diff1 with depth 2 = new diff1 (diff1 + diff2, merged)
discard old diff2… diff3 won’t work, right?
Hi, this is a great blog post to understand whet happens when deleting snapshots/checkpoints: https://blogs.msdn.microsoft.com/virtual_pc_guy/2009/04/15/what-happens-when-i-delete-a-snapshot-hyper-v/
In essence to merge down strean, not upstream. In your case, if I understand you correctly you can delete diff1 & 2, get a new now() and the 3rd diff will be around to apply if you want to. A great tip is to recreate the situation on a test VM with a text file on the desktop for each checkpoint/snapshot and test your asumption toward the merge behavior.
Good luck
Pingback: Background disk merge failed to complete: The requested operation could not be completed due to a file system limitation (0x80070299) #HyperV – Charbel.N MVP – Cloud and Datacenter Management
Had the same issue with 50 checkpoints for each disk…
so to make it a bit more #Power(shell)Full 😉
$files = (Get-ChildItem “X:\HyperV\server*.avhdx” | Sort-Object -Property lastwritetime -Descending)
foreach ($file in $files)
{
$newName = $file.Name + “.vhdx”
Rename-Item $file $newName
Merge-VHD -Path “D:\HyperV\VHD\$newName” -force
}
wops, you need to edit the path on both first and last line
Yup automation is the way to go once the mechanism is understood and when the logic of the child/ is not messed up by tampering. Well done 🙂
Hi, Im in the same state. Am I correct in thinking as long as all the checkpoints exist in the same folder and its one consistent tree this will merge all the VMDK’s based on the same file name in the given date so newest checkpoint to oldest?
Thanks
I cannot speak for VMDK files however as that’s not Hyper-V but VMware and this article deals with Hyper-V checkpoints. If you meant VHD/VHDX/AVHDX files and if all is well otherwise and you do it correctly, than yes, than you should be able to merge one by one manually.
My apologies I did mean (a)vhd(x) rather than vmdk’s… 🙂
Just to confirm – do I start by merging the NOW disks to their parents or the last checkpoint files to their parents? Thanks
If you want to end up with the situation in “now” with no checkpoints at all yes, you keep moving most recent to oldest. You can do it asynchronously but do your self a favor and keep things in the flow as a simple and logical to follow as possible.
Any thoughts as to why after I have manually merged the old checkpoints they still show in the checkpoint tree within Hyper-V. If I use powershell to remove them it fails .
IF they’re really all merged, maybe it’s just a GUI artifact. Close hyper-V manager, restart the Hyper-V VMMS service and see if they’re still there.
Restarted the service and the checkpoints are still there even though there are no avhd(x) files with the checkpoint timestamp on it. Am I correct in thinking the VM looks at the virtual drive locations and cross references the checkpoints with the dates on the avhd(x) files? Or is the information stored somewhere else?
Without seeing the situation it’s hard to be definitive about what’s happening. If I understand you correctly you still have some checkpoints (avhdx) you want to keep but those are not the one you see in the GUI or when you query for them via PowerShell? Do you need to keep those or can you just merge them all. If you still cannot get rid of them in any other way , Export the VM, dump the orginal one and import the exported one (keep the ID).
That’s correct I have a whole list of -backup checkpoints which I want to get rid of. But because there are so many just for now I wanted to delete say 5 days worth to see if it would resolve the problem. The ones which I see in the GUI include the ones I have manually merged so in theory shouldn’t be there. As you said – my next plan is to merge all the -backup checkpoints so I am left with just the vhd(x) files and then re-create the VM without the checkpoints.
MAN you saved my life. Thanks alot it worked for me.
Thanks, I’m very happy it helped you out!
Hi,
Just to check – if I point my newly created VM and the drives which have been merged back does this effect the old VM? I appreciate you don’t run 2 VM’s from one set of disks but I just wanted to make sure if the new VM doesn’t boot for what ever reason the old one will.
That should work. I the disks are in use already you’ll get an error throw anyway. But if all merges have succeeded your old one should be good to go. Not sure why you need to create a new one in this case.
So I encountered my error when i tried to expand the E:\ drive on my SQL_DB and for one reason or the other I had to delete the entire VM. I recovered the VM’s .vhdx and .avhdx files from a snapshot taken at storage level. I am copying to C:\Cluster storage now (this is my 2nd copy on this issue). Please note that my VM has two vhds (i.e. drive C:\ (OS) and drive E:\ (DB files).) I have created a new VM and I used the existing .vhds that was recovered however, the VM booted up but it is not up to date as at when i shut down the server to expand drive E:\. I would like to merge my .avhdx to my .vhdx on the two drives and use that to create a new VM. I have taken note of the most recent .avhdx file and i have listed them in descending order of date and time stamp for the two drives. On drive C:\ I have 22 .avhdx snapshots while on drive E:\ I have 17 .avhdx snapshots. This is my plan. The only way I can merge the .avhdx to the .vhdx file is to create a test VM and use the .vhdx file for drive C:\ as the exisiting vhd for the VM. Then I start merging the .avhdx file to the .vhd file with my listed order.
Please note that i also have a BIN file and VSV file for the two drive. Is this required for the manual merge.
Thanks to your swift response.
Probably the least of your troubles. You should not need them after dealing with the checkpoint. As your working on copies, you won’t risk much here by not taking them along.
Is that an ok plan for the manual merge. Kindly confirm
Should be and a s long as you work on copies, you’re not risking anything.
can you please explain why we are creating a new VM and attaching the same vhdx there.?
Just one scenario … just like we rok on copies, not on original files etc.
Hyper-V Manager says edit is not available because checkpoints exist for this virtual machine, how do I therefore open the edit disk wizard?
Always work on a copy to be safe – copy the VM and all its files to another location/folder and work on those. Secondly the UI is trying to protect you. Use PowerShell and see where you get.