Recover From Expanding VHD or VDHX Files On VMs With Checkpoints

So you’ve expanded the virtual disk (VHD/VHDX) of a virtual machine that has checkpoints (or snapshots as they used to be called) on it. Did you forget about them?  Did you really leave them lingering around for that long?  Bad practice and not supported (we don’t have production snapshots yet, that’s for Windows Server 2016). Anyway your virtual machine won’t boot. Depending on the importance of that VM you might be chewed out big time or ridiculed. But what if you don’t have a restore that works? Suddenly it’s might have become a resume generating event.

All does not have to be lost. Their might be hope if you didn’t panic and made even more bad decisions. Please, if you’re unsure what to do, call an expert, a real one, or at least some one who knows real experts. It also helps if you have spare disk space, the fast sort if possible and a Hyper-V node where you can work without risk. We’ll walk you through the scenarios for both a VHDX and a VHD.

How did you get into this pickle?

If you go to the Edit Virtual Hard Disk Wizard via the VM settings it won’t allow for that if the VM has checkpoints, whether the VM is online or not.

image

VHDs cannot be expanded on line. If the VM had checkpoints it must have been shut down when you expanded the VHD. If you went to the Edit Disk tool in Hyper-V Manager directly to open up the disk you don’t get a warning. It’s treated as a virtual disk that’s not in use. Same deal if you do it in PowerShell

Resize-VHD -Path “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” -SizeBytes 15GB

That just works.

VHDXs can be expanded on online if they’re attached to a vSCSI controller. But if the VM has checkpoints it will not allow for expanding.

image

So yes, you deliberately shut it down to be able to do it with the the Edit Disk tool in Hyper-V Manager. I know, the warning message was not specific enough but consider this. The Edit disk tool when launched directly has no idea of what the disk you’re opening is used for, only if it’s online / locked.

Anyway the result is the same for the VM whether it was a VHD or a VHDX. An error when you start it up.

[Window Title]
Hyper-V Manager

[Main Instruction]
An error occurred while attempting to start the selected virtual machine(s).

[Content]
‘DidierTest06’ failed to start.

Synthetic SCSI Controller (Instance ID 92ABA591-75A7-47B3-A078-050E757B769A): Failed to Power on with Error ‘The chain of virtual hard disks is corrupted. There is a mismatch in the virtual sizes of the parent virtual hard disk and differencing disk.’.

Virtual disk ‘C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD_8DFF476F-7A41-4E4D-B41F-C639478E3537.avhd’ failed to open because a problem occurred when attempting to open a virtual disk in the differencing chain, ‘C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd’: ‘The size of the virtual hard disk is not valid.’.

image

You might want to delete the checkpoint but the merge will only succeed for the virtual disk that have not been expanded.  You actually don’t need to do this now, it’s better if you don’t, it saves you some stress and extra work. You could remove the expanded virtual disks from the VM. It will boot but in many cased the missing data on those disks are very bad news. But al least you’ve proven the root cause of your problems.

If you inspect the AVVHD/AVHDX file you’ll get an error that states

The differencing virtual disk chain is broken. Please reconnect the child to the correct parent virtual hard disk.

image

However attempting to do so will fail in this case.

Failed to set new parent for the virtual disk.

The Hyper-V Virtual Machine Management service encountered an unexpected error: The chain of virtual hard disks is corrupted. There is a mismatch in the virtual sizes of the parent virtual hard disk and differencing disk. (0xC03A0017).

image

Is there a fix?

Let’s say you don’t have a backup (shame on you). So now what? Make copies of the VHDX/AVHDX or VHD/AVHD and save guard those. You can also work on copies or on the original files.I’ll just the originals as this blog post is already way too long. If you. Note that some extra disk space and speed come in very handy now. You might even copy them of to a lab server. Takes more time but at least you’re not working on a production host than.

Working on the original virtual disk files (VHD/AVHD and / or VHDX/AVHDX)

If you know the original size of the VHDX before you expanded it you can shrink it to exactly that. If you don’t there’s PowerShell to the rescue if you want to find out the minimum size.

image

But even better you can shrink it to it’s minimum size, it’s a parameter!

Resize-VHD -Path “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” -ToMinimumSize

Now you not home yet. If you restart the VM right now it will fail … with the following error:

‘DidierTest06’ failed to start. (Virtual machine ID 7A54E4DB-7CCB-42A6-8917-50A05354634F)

‘DidierTest06’ Synthetic SCSI Controller (Instance ID 92ABA591-75A7-47B3-A078-050E757B769A): Failed to Power on with Error ‘The chain of virtual hard disks is corrupted. There is a mismatch in the identifiers of the parent virtual hard disk and differencing disk.’ (0xC03A000E). (Virtual machine ID 7A54E4DB-7CCB-42A6-8917-50A05354634F)

image

What you need to do is reconnect the AVHDX to it’s parent and choose to ignore the ID mismatch. You can do this via Edit Disk in Hyper-V Manager of in PowerShell. For more information on manually merging & repairing checkpoints see my blogs on this subject here. In this post I’ll just show the screenshots as walk through.

image

image

image

image

image

Once that’s done you’re VHDX is good to go.

For a VHD you can’t shrink that with the inbox tools. There is however a free command line tool that can do that names VHDTool.exe. The original is hard to find on the web so here is the installer if you need it. You only need the executable, which is portable actually, don’t install this on a production server. It has a repair switch to deal with just this occurrence!

Here’s an example of my lab …

D:\SysAdmin>VhdTool.exe /repair “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD.vhd” “C:\ClusterStorage\Volume2\DidierTest06\Virtual Hard Disks\RuinFixedVHD_8DFF476F-7A41-4E4D-B41F-C639478E3537.avhd”

image

That’s it for the VHD …

You’re back in business!  All that’s left to do is get rid of the checkpoints. So you delete them. If you wanted to apply them an get rid of the delta, you could have just removed the disks, re-added the VHD/VHDX and be done with it actually. But in most of these scenarios you want to keep the delta as you most probably didn’t even realize you still had checkpoints around. Zero data loss Winking smile.

Conclusion

Save your self the stress, hassle and possibly expense of hiring an expert.  How? Please do not expand a VHD or VHDX of a virtual machine that has checkpoints. It will cause boot issues with the expanded virtual disk or disks! You will be in a stressful, painful pickle where you might not get out of if you make the wrong decisions and choices!

As a closing note, you must have have backups and restores that you have tested. Do not rely on your smarts and creativity or that others, let alone luck. Luck runs out. Otions run out. Even for the best and luckiest of us. VEEAM has save my proverbial behind a few times already.

28 thoughts on “Recover From Expanding VHD or VDHX Files On VMs With Checkpoints

  1. Hi Didier. Luckily I haven’t encountered this, but I imagine this walk-through will be a godsend to anyone unfortunate to end up in this situation.

    Thanks for another great post – been following your blog for a while, since E2EVC in Rome. Hope to see yo at the next one :).

    • Just saved my butt bigtime. I use Veeam and had a replica from 12 minutes prior to screwing things up, but…, I followed these instructions and shazaam, my VM came back up. Merged the old snapshots, and then expanded the disk. Life saver. Really. Thank you so much for posting this.

  2. Pingback: Launch your weekend with the Friday Fives! - The Microsoft MVP Award Program Blog - Site Home - MSDN Blogs

  3. Thanks for much! I thought I killed the client’s server but this saved me big time at 1am in the morning! The VDHTool with the repair option that I found was here: http://www.mediafire.com/download/3ll2zfpy2v3e47p/VhdTool.zip (the one at http://www.techieshelp.com/downloads/VhdTool.exe did not have the repair option). Thankfully for me, just repairing the .vdh files was all I needed to do. I did not have to reconnect them to the parent.

  4. Hello,

    Is there a way to repair a VHD with no snapshots? I was using my VM in Virtual Box and I made no snapshots before the VHD file get corrupted.
    Thank you in advance.

    • If it’s really corrupt restoring form backup or storage snapshots (or a Windows Shadow copy) is the fasted way. There are tools / companies out there that might be able to recover it, but it’s a matter of value as that attempt/effort has a small or large cost associated with it.

    • Hi, as long as you have space to work on a copy of the entire VM you cannot ruin any other possibilities of recovery0 What you need to do is described in the blog post but, no guarantees it will work. Your most failsafe options are to restore from backup or from a storage snapshot if you have those available.

  5. Thank you for your reply. I am actually following your blog: I’m at the step where I am about to do the repair with vhd took. But I don’t feel 100% confident that I have fully understood your steps in the blog.. I have copied both the vhd and the checkpoints..

    • Copy the entire VM folder/file structure via an “export”. That way you have everything and you cannot forget anything. It is still a vhd and not a vhdx (hence you need the tool?) Well it is always going to be a leap of faith. But if you managed to reduce the size back to the original one (which you can check as the error has changed and now complains about the parent/differencing ID issue. So if you have a copy try it. In the end that’s what would do if I had no backup.

  6. Well i took a copy of the original VHD and the AVHD.. then i merged them.. i guess i can do all the testing i want on copies? 🙂
    i managed to reconnect the AVHD to the VHD(the copies)
    looks like it will start up now. i also created another virtuel machine to avoid any issues in messing with the “original vm”
    but if i browse to the original vm or the “copy vm” i see the VHD/AVHD it boots on in the child.. is there any way to get it back to booting on the regular VHD after this “Merging”?

  7. That is what i an about confused about. I Would you be available for hire for this?
    i just need to double check i have missed something, i tried booting the machine and it would boot and go to the logon screen but now it says ” The Security Database on the Server does not have a Computer Account for this Workstation Trust Relationship” Error on Windows?”
    Before i did the “repair” on the copies i was able to boot it and log in without those issues. but of course it would be from the copi that was “older”

    • When you delete the checkpoints the parent and difference files merge and you end up with the most recent version. IF you end up with an older version you applied the checkpoint and went back in time 9and the checkpoint is still there), which might explain why the trust has gone, but this is not a broken trust but a missing one. Is the VM as DC itself? What version of Hyper-V and what OS version in the VM? First verify you have deleted the checkpoints (if you no longer need them), not applied them. It could also be that you have AD replication issues between domain controllers, perhaps even in a mixed AD versions scenario (domain controllers with different versions of Windows).

      As long as you are working on copies you can retry. I will be afk for a while. Will check comments when back.

  8. yeah the machine is a DC.
    its running SBS 2011..
    The weird thing is that i dont get WHY there is a stuck checkpoint on the original vm/vhd.
    but i cant delete it in the Hyper-V

    i just tried running the vhdtool with the repair function. now at least i am able to turn it on.. sitting on the login screen now. now ill just have to cross my fingers that i will actually be able to login.. but the checkpoint still lingers and i cant see anywhere how to merge it.. i dont want checkpoint sticking around.. but for some reason my backup system has left something from the 26th of december 2020 for some reason

    • As long as you are not booting from the checkpoint avhd and the vhd you are booting from has the situation (date/time) you want you can delete the avhd if you don’t need it anymore. With PowerShell you can try and delete it from the Hyper-V Manager GUI. Sometimes when a backup process fails in some way, artefacts can remain.

Leave a Reply, get the discussion going, share and learn with your peers.

This site uses Akismet to reduce spam. Learn how your comment data is processed.