You cannot shrink a VHDX file because you cannot shrink the volume on the virtual disk


I have discussed the capability of resizing a VHDX on line in this blog post Online Resizing Of Hyper-V Virtual Disks Is Possible in Windows 2012 R2. It’s a good resource to learn how to successfully do so.

Despite this you still might run into issue. As mentioned in the above blog post you need unallocated disk space at the end of the disk inside the virtual machine or you cannot shrink the VHDX at all. This situation is shown in the screenshot below.


In most cased this will call for you to shrink the volume size inside your virtual machine first as all space might be allocated to the volume. For this article we’ve set up a lab virtual machine to recreate the issue. The virtual machine had the page file disabled initially. We copied lots of data in it and then created shadow copies. Only then did we created a 10GB fixed sized page file to make sure it was somewhere in the beginning of the volume space. All of this was done to simulate a real world situation with lot of data churn over time. We then shift deleted the data. We now take a look at the disk where we need to shrink volume C in order to be able to shrink the virtual disk itself.


For the shrinking of a volume to succeed you need free space in that volume. But sometimes this doesn’t shrink a virtual machine as much as you’d like or not at all based on the amount of free space you see in the volume as in the figure below.


We should be able to free up to 26GB it seems. But when you try to shrink that volume you see this:


Only 11GB as available shrink space. Not quite what you’d expect based on the free space on the volume! We’ve seen this a couple of times before with virtual servers in real life. The reasons are actually well known, although more often associated with your PC at home than with virtualized servers. So how do we deal with this?

Dealing with a volume with free space that cannot shrink

The issue at hand is most probably that you have files at the end of that volume on your virtual hard disk file that prevent the disk being shrunk. There are a couple tips and tricks associated with getting this fixed.

Defragment the volume

As long as files are movable fragmentation by itself should not prevent resizing a volume. But it never hurts to run it before and it will create continuous free space at the end of the volume that can be shrunk. What’s more important here is that defragmentation cannot move all files, some are unmovable. These files have their fragments scattered all over the place and might prevent you from shrinking the volume.

On modern Windows operating systems defragmentation is part of the storage optimization maintenance job. It also runs UNMAP which informs the virtual hard disk of free space due to data having been deleted.


That’s all good and it means that you don’t even need to run defragmentation manually. But how can we deal with these unmovable files?

There are free and commercial tools that can defragment unmovable files during a boot time defragmentation run. They can even defragment and move system files that are otherwise impossible to move. A commercial tool can do off line defragmentation of your page file and other system files. By doing the defragmentation during boot time they can handle NTFS metadata files on the %systemdrive% directory (usually C:\) such as $MFTMirr, $LogFile, $Volume, $Bitmap, $Boot, and $BadClus:$Bad.

Not all unmovable files can be dealt with this way however. You must realize that since Windows vista the contents of the System Volume Information directory where Windows stores System Restore Points (shadow copies) are completely off-limits to defragmentation software.

As with many things there are manual workarounds.

Remove any “previous versions” or restore points created by shadow copies

Space efficient as these shadow copies for data protection are they can and do consume space on the disk you’re trying to shrink. As mention above, we cannot deal with them via defragmentation. Getting rid of them temporarily can help in this case. Just enable them again if needed when you’re done resizing the volume.


Tip: You can locate the shadow copies to be on a different disk. That’s worth considering when they grow large for both space considerations and performance.

Could the hibernation file cause issues?

We are discussing resizing and virtual hard disk and on virtual machine you won’t find a hyberfil.sys file. This only comes in to play when shrinking a volume on physical hardware. Hibernation is not supported or even available inside a guest OS. You can see this if you try to enable it:


Disable the page file

The page file itself can be come fragmented and it can reside completely or partially on a location of the disk that prevents the volume from being shrunk. While a page file is important to the operating system you can disable it during a maintenance window to make sure it doesn’t block resizing of the virtual hard disk. Be aware that both disabling and re-enabling the page file requires a reboot. So this does mean the online VHDX resize will cause downtime but that’s not because it’s not supported, but because of the action you need to take here to be able to shrink the volume.




The little extra unallocated space left is taken care or by extending the disk a little. Done!


Don’t forget to turn the page file back on in the best possible configuration for your workload afterwards.

Some situations require even more drastic interventions

Another issue might be that there are multiple volumes on the virtual hard disk and the free space is not at the end of the disk as in the below screen shot.


Unless you can delete volume volume H: and create it again to restore the data to the new volume which is then at the end of volume F: you’ll need to turn to 3rd party tools. Free open source tools like GParted will do the job nicely and I have used it extensively. I have a blog post on using it Using Gparted to fix virtual disk resizing issues. You still want a backup or a copy of your vhdx before doing anything like that, just in case.

The results

In the example above which is a lab setup, deleting the shadow copies and getting rid of the page file which was unfortunately located and prevent shrinking the volume more this allowed to shrink with 23GB instead of 11GB. Not bad.


Which gives us 23GB of unallocated space on the virtual disk.


Which we can now shrink the virtual hard disk with that amount!



The little extra unallocated space left is taken care or by extending the disk a little. Done!


Don’t forget to turn the page file back on in the best possible configuration for your workload afterwards and re-enable shadow copies if needed.

A real Word Example

A real world example of this is when we needed to move a 120 GB of indexing files to a dedicated virtual disk because it was causing the OS volume, the C:\ drive to run out of space. We could and did not want to grow virtual hard disk on which the guest OS drive was located. After we had moved the index we wanted to shrink the volume with about 120 GB, leaving ample frees space for the OS volume to function optimally but we could not. We could gain a pitiful 2GB of space!

First we made sure the index data was shift deleted and ran the optimizer to defrag the disk but that did not help. We check for shadow copies but there were none present. As this was a virtual server we did not have a hyberfil.sys file to worry about. In the end what did the trick for us was disabling the page file, rebooting the virtual machines, shrinking the volume and rebooting the virtual machine again.


You have seen how to address an issue where, despite having free space in a volume you cannot shrink it, and as a result, cannot shrink a VHDX file in size. That was blocking our real goal here, which was to shrink the virtual hard disk. While the latter is possible on line we cannot always mitigate the issues we encounter with shrinking a volume (by itself an online event) without down time. Disabling or enabling the page file require a reboot. Defragmentation can be done on line most of the time, but not when it comes to NTFS metadata. Disabling and enabling shadow copies is an online process however.

This is of cause a prime example of what DevOps and cloud computing at scale is discouraging. That brave new world promotes threating your servers as cattle. When one is giving you an issue you don’t nurse it back to health but fire up the barbeque as Jeffrey Snover would put it. That’s a great model if it applies to your environment. But before you do so I’d make sure that your server is not a holy cow instead of cattle. For many applications, even modern ones, in the enterprise you cannot not just kill them off. If you do you’d better have great backups but even those will not solve issues like we one, we’ve addressed here. The backups are there to protect you when things go wrong with your interventions.

A First look at Cloud Witness


In Windows Server 2012 R2 Failover Clustering we have 2 types of witness:

  1. Disk witness: a shared disk that can be seen by all cluster nodes
  2. File Share Witness (FSW): An SMB 3 file share that is accessible by all cluster nodes

Since Windows Server 2012 R2 the recommendation is to always configure a witness. The reason for this is that thanks to dynamic quorum and dynamic witness. These two capabilities offer the best possible resiliency without administrator intervention and are enabled by default. The cluster dynamically assigns a quorum vote to node when it’s up and removes it when it’s down. Likewise, the witness is given a vote when it’s better to have a witness, if you’re better off without the witness it won’t get a vote. That’s why Microsoft now advises to always set a witness, it will be managed automatically. The result of this is that you’ll get the best possible uptime for a cluster under any given circumstance.

This is still the case in Windows Server 2016 but Failover clustering does introduce a new option witness option: cloud witness.

Why do we need a cloud witness?

For certain scenarios such a cluster without shared storage and especially when a stretched cluster is involved you’ll have to use a FSW. It’s a great solution that works as well as a disk witness in most cases. Why do I say most? Well there is a scenario where a disk witness will provide better resiliency, but let’s not go there now.

Now the caveat here is that you’ll need to place the FSW in a 3rd independent site. That’s a hard order for many to fulfill. You can put in on the desktop of the receptionist at a branch office or on a virtual machine on the cluster itself but it’s “suboptimal”. Ideally the FSW is independent and high available not dependent on what it’s supposed to support in achieving quorum.

One of the other workarounds was to extend AD to Azure, deploy a SOFS Cluster with an non CA file share on a cluster of VMs in Azure and have both other sites have access to it over VPN or express route. That works but in a time of easy, fast, cheap and good solutions it’s still serious effort, just for a file share.

As Microsoft has more and more use cases that require a FSW (site aware stretched clusters, Storage Spaces Direct, Exchange DAG, SQL Availability Groups, workgroup or multi domain clusters) they had to find a solution for the growing number of customers that do not have a 3rd site but do need a FSW. The cloud idea above is great but the implementation isn’t the best as it’s rather complex and expensive. Bar using virtual machines you can’t use Azure file services in the cloud as those are primarily for consumption by applications and the security is done via not via ACLing but access keys. That means the security for the Cluster Name Object (CNO) can’t’ be set. So even when you can expose a cloud file to on premises to Windows 2016 (any OS that supports SMB 3 actually) by mapping it via NET USE the cluster GUI can’t set the required security for the cluster nodes so it will fail. And no you can’t set it manually either. Just to prove this I tried it for you to save you the trouble. Do NOT even go there!


So what is possible? Well come Windows Server 2016 Failover Clustering now has a 3rd type of witness. The cloud witness. Functionally wise it’s like a FSW. The big difference it’s a dedicated, cloud based solution that mitigates the need and costs for a 3rd data center and avoids the cost of the workarounds people came up with.

Implementing the cloud witness

In your Azure subscription you create a storage account, for this purpose I’ve create one named democloudwitness in my resource group RG-Demo. I’m using a separate storage account to keep thing tidy and separated from my other demo storage accounts.

A storage account gets two Access keys and two connection strings. The reason for this is that we you need to regenerate the keys you can have your workloads use the other one this can be done without down time.


In Azure the work is actually already done. The rest will happen on premises on the cluster. We’ll configure the cluster with a witness. In PowerShell this is a one liner.


If you get an error, make sure the information is a correct and you can reach Azure of HTTPS over the internet, VPN or Express Route. You normally do not to use the endpoint parameter, just in the rare case you need to specify a different Azure service endpoint.

The above access key is a fake one by the way, just so you know. Once you’re done Get-ClusterQuorum returns Cloud Witness as QuorumResource.


In the GUI you’ll see


When you open up the Blobs services in your storage account you’ll see that a blob service has been created with a name of msft-cloud-witness. When you select it you’ll see a file with a GUID as the name.


That guid is actually the same as your cluster instance ID that you can find in the registry of your cluster nodes under the HKLM\Cluster key in the string value ClusterInstanceID.

Your storage account can be used for multiple clusters. You’ll just see extra entries each with their own guid.


All this consumes so few resources it’s quite possibly the cheapest ever way of getting a cluster witness. Time will tell.

Things to consider

• Cloud Witness uses the HTTPS REST (NOT SMB 3) interface of the Azure Storage Account service. This means it requires the HTTPS port to be open on all cluster nodes to allow access over the internet. Alternatively an Azure Site-2-Site VPN or Express Route can be used. You’ll need one of those.

• REST means no ACLing for the CNO like on a SMB 3 FSW to be done. Security is handled automatically by the Failover Cluster which doesn’t store the actual access key, but generates a shared access security (SAS) token using the access key and stores it securely.

• The generated SAS Token is valid as long as the access key remains valid. When rotating the primary access key, it is important to first update the cloud witness (on all your clusters that are using that storage account) with the secondary access key before regenerating the primary access key.

• Plan your governance between cluster & Azure admins if these are not the same. I see Azure resources governance being neglected and as a cluster admin it’s nice to have some degree of control or say in the Azure part of the equation.

For completeness I’ll mention that the entire setup of a cloud witness is also very nicely integrated in to the Failover Cluster GUI.

Right click on the desired cluster and select “Configure Cluster Quorum Settings” from menu under “More Actions”


Click through the startup form (unless you’ve never ever done this, then you might want to read it).


Select either “Select the quorum witness” or “Advanced quorum configuration”


We keep the default selection of all nodes.


We select to “Configure a cloud witness”


Type in your Azure storage account name, your primary access key for the “Azure storage account key” and leave the endpoint at its default. You’ll normally won’t need this unless you need to use a different Azure Service Endpoint.


Click “Next”to review what you’re about to do


Click Next again and let the wizard run.


You’ll get a report when it’s done. If you get an error, make sure the information is a correct and you can reach Azure of HTTPS over the internet, VPN or Express Route.


I was pleasantly surprised by how it easy it was to set up a cloud witness. The biggest hurdle for some might be access to Azure in secured environments. The file itself contains no sensitive information at all and while a VPN or Express Route are secured connectivity options this might not be allowed or viable in certain environments. Other than this I have found it to be very reliable, effective cheap and easy. I really encourage you to test it and see what it can do for you.

July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2

Microsoft recently released another update rollup (aka cumulative update). The

July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2.

This rollup includes improvements and fixes but more importantly it also contains ‘improvements’ from June 2016 update rollup KB3161606 and May 2016 update rollup KB3156418. When it comes to the June rollup KB3161606 it’s fixes the bugs that cause concerns with Hyper-V Integration Components (IC) to even serious down time to Scale Out File Server (SOFS) users. My fellow MVP Aidan Finn discuses this in this blog post. Let’s say it caused a wrinkle in the community.

In short with KB3161606 the Integration Components needed an upgrade (to 6.3.9600.18339) but due to a mix up with the manifest files this failed. You could leave them in pace but It’s messy. To make matters worse this cumulative update also messed up SOFS deployments which could only be dealt with by removing it.

Bring in update rollup 3172614. This will install on hosts and guest whether they have  already installed or not and it fixes these issues. I have now deployed it on our infrastructure and the IC’s updated successfully to 6.3.9600.18398. The issues with SOFS are also resolved with this update. We have not seen any issues so far.


In short, CU should be gone from Windows Update and WSUS. It it was already installed you don’t need to remove it. CU will install on those servers (hosts and guests) and this time is does things right.

I hope this leads to better QA in Redmond as it really is causing a lot of people grief at the moment. It also feed conspiracy nuts theories that MSFT is sabotaging on-premises to promote Azure usage even more. Let’s not feed the trolls shall we?

Windows Server 2016 Active Memory Dump


In Windows Server 2016 we have a new option when it comes to creating memory dumps when a system failure occurs. The new option – “Active Memory Dump” – to configure a memory dump is not strictly related to failover clustering or Hyper-V. But this is the poster child environment where this setting will make a significant impact when collecting MEMORY.DMP files trouble shooting.

Hyper-V clusters tend to exist out of multiple hosts with high amounts of RAM. 256GB to 1TB of RAM is not an exception anymore. This has two reasons. In general, virtual machine density increases as the servers become ever more capable and affordable. The second reason is that ever more high performance workloads that are resource intensive are being virtualized.

The N+X nature of clusters means that even more RAM is provisioned as we need to allow for the hosts to serve extra virtual machines during scheduled or unscheduled maintenance.

To trouble shoot issues with a Hyper-V host support engineers often request a complete memory dump. This contains the processor state and the content of what’s in memory at the time of the crash. The size of these memory dumps becomes problematically large on hosts with large amounts of memory. You run out of space to create them (who has 512GB or more free space to write that dump to?) and it is problematic and time consuming to copy such files and upload them for analysis.

Active Memory Dump

So how does active memory dump address these concerns? For trouble shooting issues with the Hyper-V hosts we usually do not need the part of the RAM that is assigned to the virtual machines. On large memory Hyper-V host the majority of the RAM goes to virtual machines. An active memory dump filters out that part of the RAM content. By doing so that memory dump contains the processors state and the memory content related to the parent partition, including the user mode space, which are truly relevant to troubleshooting Blue Screen of Death events. While it’s not the smallest of possible memory dump options it is significantly smaller than a complete memory dump.

How do I configure it?

There’s two ways to do this. Via the GUI or PowerShell. Both result in exactly the same changes and configuration but the PowerShell method gives us a better insight in how an active memory dump is created.


On the Advanced tab of system properties, you select the setting for “Startup and Recovery”. That’s where you can set the memory dump option under Write debugging information.


This is reflected in two registry setting under the HKLM:\System\CurrentControlSet\Control\CrashControl key

The REG_DWORD value CrashDumpEnabled is set to 1 (default is 7) which translates into a complete memory dump.

The REG_DWORD value FilterPages is created and is set to 1

This translates in what we explained above.


An active memory dump is a complete memory dump (CrashDumpEnabed value =1) that is filtered ( FilterPages value = 1). Note that when you choose another option in the GUI the FilterPages value is not set to 0 but is actually removed.



Using PowerShell this is achieved as follows.


Keep in mind that the FilterPages value doesn’t exist if you haven’t configured Active memory dump s trying to read it will throw an error.


If you want to mimic the GUI exactly via PowerShell you’ll need to remove the value instead of setting it to 0.


PoSh code

#Take a look at the settings

Get-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name CrashDumpEnabled

Get-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name FilterPages

#Configure Active memory dump

Set-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name CrashDumpEnabled –value 1

Set-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name FilterPages –value 1

#Set it back to Automatic memory dump (default)

Set-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name CrashDumpEnabled –value 7

Remove-ItemProperty –Path HKLM:\System\CurrentControlSet\Control\CrashControl –Name FilterPages –value 1

NOTE: When you edit the registry to change this setting a reboot is required to active them. So the GUI might be your preferred way of doing things here. For the people using Windows Server Core there is a command “systempropertiesadvanced” you can run from your command prompt to get to the advanced tab of System properties. From there you get to the Startup and Recovery settings. Also note that some changes between these setting will always require a restart.




To get an idea what this means on a large memory Hyper-V host I did some testing on an enterprise grade server in a simulated setup so in real life the Active dump might very well be a bit larger than here but still the ratios tell the story.

Active memory dump: 7,60 GB

Kernel memory dump: 6,62 GB

Complete Memory dump: 319 GB

No what you need to realize that to create that Active memory dump you don’t need to create a page file the size of your physical memory. That a big deal! In many situations you’ll be hitting the problem of insufficient disk space for the page file and the memory dump to achieve this.


The active memory dump option gives us (or the support engineers) all the most relevant information they need without the overhead and practical problems associated with a complete memory dump. It kind of has my vote to become the new default option.