Kick start ADFS when your self- signed certificates have expired already

Posted on December 2, 2016 by workinghardinit

I recently had to do some lab work on a Windows Server 2012 R2 ADFS farm to prep for a migration to Windows Server 2016. Due to some storage shortage and some upgrades and migrations (all hardware in the lab runs Windows Server 2016) I had parked my Windows Server 2012 R2 ADFS farm offline.

So when I copied them back to my cluster and imported them I knew I had to make sure the domain was OK. This is easy enough, just run:

Reset-ComputerMachinePassword [-Credential Mydomain\bigadmin -Server MyDC01

That worked like a charm and soon enough my 2 VMs where up an running happily in the domain. I did have some issues however. My AFDS servers had been of line long enough before the expiration of the token-decrypting and the token-signing certificates to not yet have generate the new certificates for auto renewal and long enough to have them expire already. Darn!

The result was a bunch of errors in the event log as you might expect and appreciate.

An error occurred during an attempt to build the certificate chain for configuration certificate identified by thumbprint ’26AFDC4A226D2605955BF6F844F0866C14B1E82B’. Possible causes are that the certificate has been revoked or certificate is not within its validity period.
The following errors occurred while building the certificate chain:
MSIS2013: A required certificate is not within its validity period when verifying against the current system clock.

But this also raised the question on how to get the ADFS servers back in a working condition. Normally these are generated automatically close to the expiration date of your existing certs (or at the critical threshold you configured). So I disabled / re-enabled auto certificate rollover but does actually does it even kick in if you have already expired? That I don’t know and I really had no time to wait hours or days to see what happens.

Luckily there is a command you can issue to renew the certificates immediately. This is the same command you can use when you have disabled auto rollover and need it re-enabled. That works normally after some patience.

Update-AdfsCertificate -Urgent

The result was immediate, the self signing certs were renewed.

And we can see this in the various entries in the event log

Do note that this command will cause a disruption of the service with your partners until they have refreshed the information from your federation metadata – or in the case this isn’t or can’t be leveraged, manually updated. In my case I had a “service down” situation anyway, but in normal conditions you’d plan this and follow the normal procedure you have in place with any partner that need your ADFS Services.

Disk2VHD on a Generation 2 VM results in an unbootable VHDX

Posted on September 8, 2016 by workinghardinit

106

Most people who have been in IT for a while will know the Windows Sysinternals tools and most certainly the small but brilliant Disk2VHD tool we can use for Physical To Virtual (P2V) and Virtual to Virtual (V2V) conversions. It’s free, it’s good and it’s trustworthy as it’s made available by Microsoft.

For legacy systems, whether they are physical with IDE/SATA/SAS controllers or virtual with an IDE generation 1 VMS thing normally go smooth.

But sometimes you have hiccups. One of those is when you do a V2V of a generation 2 virtual machine using Disk2VHD. It’s a small issue, when you create a new generation 2 VM and point it to the OS vhdx it just won’t boot. That’s pretty annoying.

Why do a V2V in such a case you might ask. Well, sometimes is the only or fasted way to get out of pickle with a ton of phantom, non-removable checkpoints you’ve gotten yourself into.

But back to the real subject, how to fix this. What we need to do is repair the boot partition. Well recreate it actually as when you look at it after the conversion you’ll notice is RAW. That’s no good. So let’s walk through how to fix a vdhx that your created from a source generation 2 Hyper-V vm via Disk2VHD.

First of all create a new generation 2 VM that we’ll use with our new VHDX we created using Disk2VHD. Don’t create a new vdhx but select to use an existing one and point it to the one we just created with Disk2VHD. Rename it if needed to something more suitable.

Don’t boot the VM but add a DVD and attach the Windows Server ISO of the version your vhdx contains to the DVD.

Move the DVD to the top of the boot order I firmware.

The VM will boot to the DVD when you hit a key.

Select your language and keyboard layoout when asked and the don’t install or upgrade the OS but boot

Type diskpart and list the disks. Select the disk we need (the OS disk, the only one here) and list the volumes. You can see that volume 3 off 99MB is RAW. That’s not supposed to be that way. So let’s fix this by creating boot loader directory structure, repair the boot record by creating the boot sector & copy the needed boot files into it.

Type:

select volume 3

assign drive letter L:

FORMAT FS=FAT32 LABEL=”BOOT”

That’s it we can now us that 99MB volume to make our disk bootable to windows again. Type Exit to leave diskpart.

So now we have a formatted boot partition we can create the need folder structure and fix the boot record and configure our UEFI bootloader

Switch to the L: volume

create efi\microsoft\boot folder structure for the bootloader as show below with the md command(make directory)

Type: bootrec /fixboot to create the bootrecord

Type: bcdboot C:\Windows /l en-us /s l: /f ALL

This creates the BCD store & copies the boot files from the windows system directory

Just click Continue to exit and continue to Windows Server 2012R2

.. and voila, your new VM has now booted.

Now it’s a matter of cleaning up the remnants of the original VMs hardware such as the NIC and maybe some other devices. The NIC is very important as it will have any static TCP/IP configuration you might want to assign tied to it which mean you can’t reuse it for your new VM. So, the 1st thing to do is uninstall the old network adapters from device managers, you’ll see them when you select “show hidden devices” in the view menu.

Good luck!

You cannot shrink a VHDX file because you cannot shrink the volume on the virtual disk

Posted on August 9, 2016 by workinghardinit

Introduction

I have discussed the capability of resizing a VHDX on line in this blog post Online Resizing Of Hyper-V Virtual Disks Is Possible in Windows 2012 R2. It’s a good resource to learn how to successfully do so.

Despite this you still might run into issue. As mentioned in the above blog post you need unallocated disk space at the end of the disk inside the virtual machine or you cannot shrink the VHDX at all. This situation is shown in the screenshot below.

In most cased this will call for you to shrink the volume size inside your virtual machine first as all space might be allocated to the volume. For this article we’ve set up a lab virtual machine to recreate the issue. The virtual machine had the page file disabled initially. We copied lots of data in it and then created shadow copies. Only then did we created a 10GB fixed sized page file to make sure it was somewhere in the beginning of the volume space. All of this was done to simulate a real world situation with lot of data churn over time. We then shift deleted the data. We now take a look at the disk where we need to shrink volume C in order to be able to shrink the virtual disk itself.

For the shrinking of a volume to succeed you need free space in that volume. But sometimes this doesn’t shrink a virtual machine as much as you’d like or not at all based on the amount of free space you see in the volume as in the figure below.

We should be able to free up to 26GB it seems. But when you try to shrink that volume you see this:

Only 11GB as available shrink space. Not quite what you’d expect based on the free space on the volume! We’ve seen this a couple of times before with virtual servers in real life. The reasons are actually well known, although more often associated with your PC at home than with virtualized servers. So how do we deal with this?

Dealing with a volume with free space that cannot shrink

The issue at hand is most probably that you have files at the end of that volume on your virtual hard disk file that prevent the disk being shrunk. There are a couple tips and tricks associated with getting this fixed.

Defragment the volume

As long as files are movable fragmentation by itself should not prevent resizing a volume. But it never hurts to run it before and it will create continuous free space at the end of the volume that can be shrunk. What’s more important here is that defragmentation cannot move all files, some are unmovable. These files have their fragments scattered all over the place and might prevent you from shrinking the volume.

On modern Windows operating systems defragmentation is part of the storage optimization maintenance job. It also runs UNMAP which informs the virtual hard disk of free space due to data having been deleted.

That’s all good and it means that you don’t even need to run defragmentation manually. But how can we deal with these unmovable files?

There are free and commercial tools that can defragment unmovable files during a boot time defragmentation run. They can even defragment and move system files that are otherwise impossible to move. A commercial tool can do off line defragmentation of your page file and other system files. By doing the defragmentation during boot time they can handle NTFS metadata files on the %systemdrive% directory (usually C:\) such as $MFTMirr, $LogFile, $Volume, $Bitmap, $Boot, and $BadClus:$Bad.

Not all unmovable files can be dealt with this way however. You must realize that since Windows vista the contents of the System Volume Information directory where Windows stores System Restore Points (shadow copies) are completely off-limits to defragmentation software.

As with many things there are manual workarounds.

Remove any “previous versions” or restore points created by shadow copies

Space efficient as these shadow copies for data protection are they can and do consume space on the disk you’re trying to shrink. As mention above, we cannot deal with them via defragmentation. Getting rid of them temporarily can help in this case. Just enable them again if needed when you’re done resizing the volume.

Tip: You can locate the shadow copies to be on a different disk. That’s worth considering when they grow large for both space considerations and performance.

Could the hibernation file cause issues?

We are discussing resizing and virtual hard disk and on virtual machine you won’t find a hyberfil.sys file. This only comes in to play when shrinking a volume on physical hardware. Hibernation is not supported or even available inside a guest OS. You can see this if you try to enable it:

Disable the page file

The page file itself can be come fragmented and it can reside completely or partially on a location of the disk that prevents the volume from being shrunk. While a page file is important to the operating system you can disable it during a maintenance window to make sure it doesn’t block resizing of the virtual hard disk. Be aware that both disabling and re-enabling the page file requires a reboot. So this does mean the online VHDX resize will cause downtime but that’s not because it’s not supported, but because of the action you need to take here to be able to shrink the volume.

The little extra unallocated space left is taken care or by extending the disk a little. Done!

Don’t forget to turn the page file back on in the best possible configuration for your workload afterwards.

Some situations require even more drastic interventions

Another issue might be that there are multiple volumes on the virtual hard disk and the free space is not at the end of the disk as in the below screen shot.

Unless you can delete volume volume H: and create it again to restore the data to the new volume which is then at the end of volume F: you’ll need to turn to 3^rd party tools. Free open source tools like GParted will do the job nicely and I have used it extensively. I have a blog post on using it Using Gparted to fix virtual disk resizing issues. You still want a backup or a copy of your vhdx before doing anything like that, just in case.

The results

In the example above which is a lab setup, deleting the shadow copies and getting rid of the page file which was unfortunately located and prevent shrinking the volume more this allowed to shrink with 23GB instead of 11GB. Not bad.

Which gives us 23GB of unallocated space on the virtual disk.

Which we can now shrink the virtual hard disk with that amount!

The little extra unallocated space left is taken care or by extending the disk a little. Done!

Don’t forget to turn the page file back on in the best possible configuration for your workload afterwards and re-enable shadow copies if needed.

A real Word Example

A real world example of this is when we needed to move a 120 GB of indexing files to a dedicated virtual disk because it was causing the OS volume, the C:\ drive to run out of space. We could and did not want to grow virtual hard disk on which the guest OS drive was located. After we had moved the index we wanted to shrink the volume with about 120 GB, leaving ample frees space for the OS volume to function optimally but we could not. We could gain a pitiful 2GB of space!

First we made sure the index data was shift deleted and ran the optimizer to defrag the disk but that did not help. We check for shadow copies but there were none present. As this was a virtual server we did not have a hyberfil.sys file to worry about. In the end what did the trick for us was disabling the page file, rebooting the virtual machines, shrinking the volume and rebooting the virtual machine again.

Conclusion

You have seen how to address an issue where, despite having free space in a volume you cannot shrink it, and as a result, cannot shrink a VHDX file in size. That was blocking our real goal here, which was to shrink the virtual hard disk. While the latter is possible on line we cannot always mitigate the issues we encounter with shrinking a volume (by itself an online event) without down time. Disabling or enabling the page file require a reboot. Defragmentation can be done on line most of the time, but not when it comes to NTFS metadata. Disabling and enabling shadow copies is an online process however.

This is of cause a prime example of what DevOps and cloud computing at scale is discouraging. That brave new world promotes threating your servers as cattle. When one is giving you an issue you don’t nurse it back to health but fire up the barbeque as Jeffrey Snover would put it. That’s a great model if it applies to your environment. But before you do so I’d make sure that your server is not a holy cow instead of cattle. For many applications, even modern ones, in the enterprise you cannot not just kill them off. If you do you’d better have great backups but even those will not solve issues like we one, we’ve addressed here. The backups are there to protect you when things go wrong with your interventions.

The Cluster and 0.Cluster Registry Hives

Posted on March 29, 2016 by workinghardinit

The cluster database

In a Windows Server Cluster the cluster database is where the cluster configuration gets stored. It’s a file called CLUSDB with some assisting files (CLUSDB.1.container, CLUSDB.2.container, CLUSDB.blf) and you’ll find those in C:\Windows\Cluster (%systemroot%\Cluster).

But the cluster database also lives in a registry hive that gets loaded when the cluster service gets started. You’ll find under HKEY_LOCAL_MACHINE and it’s called Cluster. You might also find a 0.Cluster hive on one of the nodes of the cluster.

The 0.Cluster hive gets loaded on a node that is the owner of the disk witness. So if you have a cloud share or a file share witness this will not be found on any cluster node. Needless to say if there is no witness at all it won’t be found either.

On a lab cluster you can shut down the cluster service and see that the registry hive or hives go away. When you restart the cluster service the Cluster hive will reappear. 0.Cluster won’t as some other node is now owner of the disk witness and even when restarting the cluster service gets a vote back for the witness the 0.Cluster hive will be on that owner node.

If you don’t close the Cluster or 0.Cluster registry hive and navigate to another key when you test this you’ll get an error message thrown that the key cannot be opened. It won’t prevent the cluster service from being stopped but you’ll see an error as the key has gone. If you navigate away, refresh (F5) you’ll see they have indeed gone.

So far the introduction about the Cluster and 0.Cluster Registry Hives.

How is the cluster database kept in sync and consistent?

Good so now we know the registry lives in multiple places and gets replicated between nodes. That replication is paramount to a healthy cluster and it should not be messed with. You can see an DWORD value under the Cluster Key called PaxosTag (see https://support.microsoft.com/en-us/kb/947713 for more information). That’s here the version number lives that keep track of any changes and which is important in maintaining the cluster DB consistency between the nodes and the disk witness – if present – as it’s responsible for replicating changes.

You might know that certain operations require all the nodes to be on line and some do not. When it’s require you can be pretty sure it’s a change that’s paramount to the health of the cluster.

To demonstrate the PaxosTag edit the Cluster Networks Live Migration settings by enabling or disable some networks.

Hit F5 on the registry Cluster/0.cluster Hive and notice the tag has increased. That will be the case on all nodes!

As said when you have a disk witness the owner node of the witness disk also has 0.Cluster hive which gets loaded from the copy of the cluster DB that resides on the cluster witness disk.

As you can see you find 0.hive for the CLUSTERDB and the equivalent supporting files (.container, .blf) like you see under C:\Windows\Cluster on the cluster disk un the Cluster folder. Note that there is no reason to have a drive letter assigned to the witness disk. You don’t need to go there and I only did so to easily show you the content.

Is there a functional difference between a disk witness and a file share or cloud witness?

Yes, a small one you’ll notice under certain conditions. Remember a file share of cloud witness does not hold a copy of the registry database. That also means there’s so no 0.Cluster hive to be found in the registry of the owner node. In the case of a file share you’ll find a folder with a GUID for its name and some files and with a cloud witness you see a file with the GUID of the ClusterInstanceID for its name in the storage blob. It’s bit differently organized but the functionality of these two is exactly the same. This information is used to determine what node holds the latest change and in combination with the PaxosTag what should be replicated.

The reason I mention this difference is that the disk witness copy of the Cluster DB is important because it gives a disk witness a small edge over the other witness types under certain scenarios.

Before Windows 2008 there was no witness disk but a “quorum drive”. It always had the latest copy of the database. It acted as the master copy and was the source for replicating any changes to all nodes to keep them up to date. When a cluster is shut down and has to come up, the first node would download the copy from the quorum drive and then the cluster is formed. The reliance on that quorum copy was a single point of failure actually. So that’s has changed. The PaxosTag is paramount here. All nodes and the disk witness hold an up to date copy, which would mean the PaxosTag is the same everywhere. Any change as you just tested above updates that PaxosTag on the node you’re working on and is replicated to every other node and to the disk witness.

So now when a cluster is brought up the first node you start compares it’s PaxosTag with the one on the disk witness. The higher one (more recent one) “wins” and that copy is used. So either the local clusterDB is used and updates the version on the disk witness or vice versa. No more single point of failure!

There’s a great article on this subject called Failover Cluster Node Startup Order in Windows Server 2012 R2. When you read this you’ll notice that the disk witness has an advantage in some scenarios when it comes to the capability to keep a cluster running and started. With a file share or cloud witness you might have to use -forcequorum to get the cluster up if the last node to be shut down can’t be started the first. Sure these are perhaps less common or “edge” scenarios but still. There’s a very good reason why the dynamic vote and dynamic witness have been introduced and it makes the cluster a lot more resilient. A disk witness can go just a little further under certain conditions. But as it’s not suited for all scenarios (stretched cluster) we have the other options.

Heed my warnings!

The cluster DB resides in multiple places on each node in both files and in the registry. It is an extremely bad idea to mess round in the Cluster and 0.Cluster registry hives to clean out “cluster objects”. You’re not touching the CLUSDB file that way or the PaxosTag used for replicating changes and things go bad rather quickly. It’s a bad situation to be in and for a VM you tried to remove that way you might see:

You cannot live or quick migrate that VM. You cannot start that VM. You cannot remove that VM from the cluster. It’s a phantom.
Even worse, you cannot add a node to the cluster anymore.
To make it totally scary, a server restart ends up with a node where the cluster service won’t start and you’ve just lost a node that you have to evict from the cluster.

I have luckily only seen a few situations where people had registry corruption or “cleaned out” the registry of cluster objects they wanted to get rid of. This is a nightmare scenario and it’s hard, if even possible at all, to recover from without backups. So whatever pickle you get into, cleaning out objects in the Cluster and/or 0.Cluster registry hive is NOT a good idea and will only get you into more trouble.

Heed the warnings in the aging but still very relevant TechNet blog Deleting a Cluster resource? Do it the supported way!

I have been in very few situations where I managed to get out of such a mess this but it’s a tedious nightmare and it only worked because I had some information that I really needed to fix it. Once I succeeded with almost no down time, which was pure luck. The other time cluster was brought down, the cluster service on multiple nodes didn’t even start anymore and it was a restore of the cluster registry hives that saved the day. Without a system state backup of the cluster node you’re out of luck and you have to destroy that cluster and recreate it. Not exactly a great moment for high availability.

If you decide to do muck around in the registry anyway and you ask me for help I’ll only do so if it pays 2000 € per hour, without any promise or guarantee of results and where I bill a minimum of 24 hours. Just to make sure you never ever do that again.

Working Hard In IT

My view on IT from the trenches

Category Archives: Windows