Sometimes things go wrong with a backup job. There are many reasons for this, but that’s not the focus of this blog post. We’re going to show you how to remove lingering Backup checkpoints from a Hyper-V Virtual Machine that where not properly removed after a backup on Windows Server 2012 R2 Hyper-V.
You see checkpoint with a “odd” icon that looks like this
You can right click on the checkpoint but you will not find an option to apply or delete it.This is a recovery snapshot and you’re not supposed to manipulate this manually. It’s there as part of the backup process. You can read more about that in my blog posts Some Insights Into How Windows 2012 R2 Hyper-V Backups Work and What Is AutoRecovery.avhdx all about?.
When we look ad the files we see the traces of a Windows Server 2012R2 Hyper-V backup
Some people turn to manually merging these checkpoints. I have discussed this process in blog posts 3 Ways To Deal With Lingering Hyper-V Checkpoints Formerly Known as Snapshots and Manually Merging Hyper-V Checkpoints. But you don’t have to do this! As a matter of fact you should avoid this if possible at all. The good news is that this can be done via PowerShell.
I’m not convinced the fact that you can’t do it in the GUI is to be considered a bug as some would suggest. The fact is you’re not supposed to have to do this, so the functionality is not there. I guess this is to protect people from trying to delete one by mistake when they see one during backups.
Anyway, PowerShell to the rescue!
So we run:
Get-VMSnapshot -ComputerName "MyHyperVHost" -VMName "VMWithLingeringBackupCheckpoint"
This show us the checkpoint information (note it does not show you the XXX-AutoRecovery.avhdx” as a checkpoint.
And then we simply run Remove-VMSnapshot
#Just remove the lingerging backup checkpoint like you would a normal one via PoSh Get-VMSnapshot -ComputerName "MyHyperVHost" -VMName "VMWithLingeringBackupCheckpoint" | Remove-VMSnapshot
I would love this to work, but for me it doesn’t.
I have some of this checkpoints left over from Acronis Backup jobs. I only get this error when trying to remove the checkpoints
Remove-VMSnapshot : ‘NAME’ failed to remove checkpoint. (Virtual machine ID 4EDF5034-3260-46C7-B44F-885A7AAEF4DB)
Cannot delete checkpoint: Catastrophic failure (0x8000FFFF). Checkpoint ID 1964A143-6EB9-40F6-B649-5C9ACD2C27FF.
Virtual machine failed to generate VHD tree: ‘Catastrophic failure'(‘0x8000FFFF’).
You might have a more complex issue with the avhd/x tree/cahin not being correct. Take a look here: https://blog.workinghardinit.work/2014/08/29/manually-merging-hyper-v-checkpoints/
WARNING WORK ON A COPY OF THE FILES TO TEST THIS. USE AT YORU OWN RIKS.
Flawless.
My situation came from pulling a backup off of system center DPM into filesystem after a failed recovery over VM.
Ended up deleteing corrupted vhds and copying over the recovered vhds through filesystem. In failover cluster, adjusted the vm config so disks point to the -AutoRecovery.avhds, OKed through a bunch of intimidating “you can corrupt the snapshot chains” messages, and then started the VM. Ran the powershell command to remove the snapshot and the merge happened so fast I had to double check the files to make sure they were gone, but it worked!
Cheers m8
I’ve got a very similar situation, which has been going on for around a week. I actually have a chain of recovery checkpoints, four of them!
So is it still safe to run – Get-VMSnapshot -ComputerName “MyHyperVHost” -VMName “VMWithLingeringBackupCheckpoint” | Remove-VMSnapshot
As I think it may attempt to remove the checkpoints in the wrong order, basically the order that powershell reports them with the “Get-VMSnapshot” command, which is earliest first.
Any ideas? This is a production web server so I’m not happy about diving in and giving it a go. I have a backup, but I’ve no idea they’re OK given these recovery points – the backups could have been disrupted totally by the server reboot. 🙁
Just for info, the reason I have four chained recovery checkpoints is because I accidentally set up a server reboot scheduled task (twice a week) at the exact same time in the evening as the Windows Server Backup (ran every evening). Every day the server reboots, it throws up another recovery checkpoint!
Thanks!
If they are truely lingering checkpoints it should not matter. But I cannot determine this from here. What you can always do is create yet another checkpoint manually of the VM when it’s shut down (that’s “app’ consistent). You can export that checkpoint (even while the VM is running) your clean up action goes south all you need to do is import the VM again which is free of checkpoints. Good luck.
Many thanks. Good idea about taking another snap…
I have tried the command out on another VM which was doing the same thing – this one a lot less critical. It had two such checkpoints.
Seemed to work okay – I suspect S2012R2 Hyper-V is sufficiently developed enough for it not to be a problem.
I’ve had other things on the go so haven’t yet done it on the production VM.
Thanks!
I thought I’d do it.
Confirmed working. You CAN remove a chain of recovery checkpoints using the powershell method above – at least using a bang up to date S2012 R2 server.
It has been a couple of weeks since the checkpoints were made, so it took quite a time to merge them back in. The machine went a bit unresponsive in patches, but came back responsive before the merge finished up.
Thanks!
Any idea what causing this? Doesn’t matter if the server is online or shut down.
“Remove-VMSnapshot : The operation cannot be performed while the virtual machine is in its current state. The name of the snapshot is xxxxx – (31/01/2017 – 17:59:33) and its ID is xxxxx. At line:1 char:66.”
Also, how much free space does the merging require? The same as the VM’s actual disks or more? I have a VM with a chain of almost 30 checkpoints that’s taking almost all available space.
1) Can be a number of things. Hard to say just like that. Try the Hyper-V Virtual Machine Management service a restart.
2) You need space to merge the files together before it will delete the orginal files. See my other blog s for a possible way out https://blog.workinghardinit.work/2014/08/25/3-ways-to-deal-with-lingering-hyper-v-checkpoints-formerly-known-as-snapshots/ is the original LUN is to full to do anything anymoe. Shut down VM, create snapshot export it, delete the original VM, copy the exported VM import the exported VM … make sure you have a backup juts in case.
Snapshot export -> import did the trick. All good now. I also tried creating and deleting the latest snapshot, gave a Catastrophic Failure message. Same for trying to merge any or all of the old snapshots.
Thanks for the articles.
Wrote a previous reply but that went.. somewhere. Anyway, the snapshot exporting-importing did the trick. Creating and deleting a new snapshot and remove-vmsnapshot for a single/all snapshots all gave Catastrophic Failures, probably due to low disk space.
Thanks for the article.
Happy to hear you got it sorted!
It could be because you have hyper-v replication enabled?
Thank you sir, worked perfectly for me 😉
Happy to hear it.
Can you please re-post the powershell in plain text? As my browser cant read the commands!
Thanks,
Richard.
Another incredibly helpful article!
Pingback: Drive Snapshot: Virtuelle Computer unter Hyper-V wiederherstellen | Andys Blog – Linux & Windows
Need your Help. am getting below error while trying to delete checkpoint. I have manually merged the checkpoints but when i go to hyper-v manager all the checkpoints are still attached to the main disk. and no delete option. please help
PS C:\Windows\system32> Get-VMSnapshot -VMName “vm1” -Name “BK-CRM-DB02 – (2/10/2019 – 10:03:52 PM)” | Remove-VMSnapshot
Remove-VMSnapshot : ‘vm1’ failed to remove checkpoint.
Cannot delete checkpoint: The system cannot find the file specified. (0x80070002).
Virtual machine failed to generate VHD tree.
Failed to get VHD parent.
An error occurred because the checkpoint is missing.
‘vm1’ failed to remove checkpoint. (Virtual machine ID C58DBB12-C6BF-4764-995C-CF759622729F)
Cannot delete checkpoint: The system cannot find the file specified. (0x80070002). Checkpoint ID D4D260B3-993B-44F2-B4AD-9EC219308C92.
Virtual machine failed to generate VHD tree: ‘The system cannot find the file specified.'(‘0x80070002’).
Failed to get VHD (‘C:\vm1_2C2B2D12-0CB8-4D14-A19F-0BA5A157FCC0.avhdx’) parent: ‘The system cannot find the
file specified.'(‘0x80070002’).
‘vm1’ is missing a checkpoint. The disk paths missing used to be located at: C:\vm1_2C2B2D12-0CB8-4D14-A19F-0BA5A157FCC0.avhdx. The system cannot find the file specified. (0x80070002) (Virtual Machine ID
C58DBB12-C6BF-4764-995C-CF759622729F.)
At line:1 char:88
+ … ” -Name “vm1 – (2/10/2019 – 10:03:52 PM)” | Remove-VMSnapshot
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (:) [Remove-VMSnapshot], VirtualizationException
+ FullyQualifiedErrorId : ObjectNotFound,Microsoft.HyperV.PowerShell.Commands.RemoveVMSnapshot
If all else fails and If you have no avhdx left so they are all merged create a new checkpoint of the healthy VM and export it. You will have a VM that you can import and replace the one with the checkpoints you cannot delete. When all is well you can dump that old VM.
Thanks a lot team but am getting below error while trying to create new checkpoint.
PS C:\Windows\system32> Checkpoint-VM -Name “BK-CRM-DB02”
Checkpoint-VM : ‘BK-CRM-DB02’ failed to take a checkpoint.
Cannot take checkpoint.
‘BK-CRM-DB02’ failed to take a checkpoint. (Virtual machine ID C58DBB12-C6BF-4764-995C-CF759622729F)
Cannot take checkpoint for ‘BK-CRM-DB02’ because the maximum limit of 50 checkpoints has been reached. Delete unwanted checkpoints and try again. (Virtual machine ID
C58DBB12-C6BF-4764-995C-CF759622729F)
At line:1 char:1
+ Checkpoint-VM -Name BK-CRM-DB02
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Checkpoint-VM], VirtualizationException
+ FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.CheckpointVM
OK. Shutdown original vm. copy all the VM files to a new folder and import form there as new VM. You will have nev VM without the checkpoints if there are indeed non needed.you cab then remove the old one. Move will also work if you do not have space for copy.
Thanks,
I have space and i can copy but the VM is on production and being used right now. about moving it, can that solve the issue when i move the VM without shutting it down? will i have to import it after moving it or issue will be sorted?
No the files are in use.
Thank you so much for your support!!! going to plan a maintenance and will let you know how it goes.
If the VM is running on a cluster see if live migration helps work around the issue of deleting the checkpoints.
Thanks guys for your help. The issue is sorted now.
Can you please let me know why the checkpoints are created automatically without someone intervention.
Hyper-V Replica, Backups … read up on what types of checkpoints there are and why/when they are used.
I have found that you can remove through the GUI if you select the checkpoint and then press the delete key on the keyboard.
Hi! I’m using Veeam, and i have like 10 or 12 .avhdx files, when i listed the checkpoints only listed “DPM-**” which is the actual checkpoint i wanted to deleted.. the query worked, but the .avhx didn’t merge at all, it just dissapeared the “checkpoint” from hyper-v (btw. im using scvmm too).
Please tell me there’s another way to merge manually 10 or 12 .avhx files.. i saw you other post for the merge information, and all the pictures has broken links.
Thanks!
Manually merging can sometimes be the only option left, best left if all else fails. But first, assure what checkpoint the VMs are running against to make sure those avhdx are still in used (check child/parent chain). There are other option to fix the situation but they all require some time and hands on work (export and import or a backup restore, both replace the original). I noticed the issue with the images, I will fix that when I have time.
Should we shutdown the VM while deleting the snapshot through powershell ? I have a lingering snapshot which is tripple the size of .vhdx file.VM is connected and runnning from .Avhdx file
A merge should work online
The problem we ran into =
one of the VSH in our cluster had the VHDX file ‘attached’ (this VM was powered off when the VSH mounted it). We could not perform any actions on the VM (like create a checkpoint/inspect HD files).
After we did find the VSH that had the file attached, we could ‘not’ detach the disk.
We think that the reason we could not detach it = The server that had the VHDX attached was were the VM was hosted… However, load balancing moved the VM to a different VSH.
Since the VHDX was managed by the 2nd VSH the 1st VSH was not able to make changes to the file (like detaching).
We solved our issue by draining the VSH (which had the attached VHDX) and then shutting it down.
As soon as that server was powered off, we were able to perform actions on the VM.
Worked perfectly to get around a Veeam checkpoint and expand the disk on the VM
You can highlight the snapshot in question and hit the “delete” key on the keyboard. Not sure if this was available when this post was written, but leaving this here for those that just have one or two to remove.
I think that also doesn’t always work, hence the need to try it with PoSh and even that is not 100% guaranteed. There are other tricks to get rid of them but then we often get in the realm of downtime.
Great article, solved my problem with VEEAM – thank you very much!
BR,
Michael
Or… you could just simply highlight the checkpoint you don’t want anymore and hit the delete key on your keyboard. Problem solved!
It does not always work, but yes!
Brilliant worked perfectly for me, I didn’t need to delete the .AVHDX files after! Once the merge completed they deleted themselves.
Many Thanks
Ian
Just worked for us on al old W2012R2 HyperV server after a failed Veem Backup copy tha ran out of space.
Pingback: HYPER-V: Backup Checkpoint não está sendo excluído – A Vida de Suporte
Thank you, it worked well