Windows Server 2012 R2 Clustering brings improved CSV diagnosability

Cluster Shared Volumes (CSV) have can go into different redirected access modes for several reasons. Now a lot of people get (or got) worried about seeing “redirected access” in the GUI. Most of the time however this is due to normal operations such as backups or maintenance (defragmentation) not only losing disk access.

To remediate unneeded troubleshooting, sometimes leading to real issues, calls to MSFT support and so forth it was hidden from the Failover Clustering GUI in Windows 2012 R2. OK, so goal achieved but how do we now troubleshoot and view redirected access that might indicate the presence of real issues? The answer to that is the Get-ClusterSharedVolumeState PowerShell cmdlet. It displays the state of the CSVs on a per node basis for a cluster. You’ll see the type of the IO (Direct, File System Redirected and Block Redirected), if it’s completely unavailable  as well as the reason.

This is what the output looks like on a two node cluster where node A has lost it’s storage path or paths (MPIO) to the CSV. You’ll see that both CSV are in redirected access. Not only that but you can see what type (block redirected) and why (no disk connectivity).

image

Pretty neat and clear. I love this functionality by the way and It’s why I’m leveraging 10Gbps Ethernet extensively to make sure that CSV traffic get’s the bandwidth & latency to handle what it has to. If you realize it leverages SMB3 which provides SMB Multichannel and SMB Direct you know it will get the job done for you in your time of need.

While this is happening in the GUI you’ll see this

image

Nothing is going on … it would seem so a bit of monitoring and alerting would be of use here. The good news is finding out what’s up is very straight forward now.

Now there is still a case where you’ll see that the CSV is in redirected access mode and that when you’ve put it in there yourself via the GUI

image

or via PowerShell for maintenance reasons.

image

As you can see the Icon has change to a networked disk one and it states “Redirected Access”. With Get-ClusterSharedVolumeState the output looks like this.

image

You’ll always see warnings in the event logs.

image

So monitor those with SCOM or another tool that suits your taste and you’ll be in good shape to react when it’s needed and you now know how to find out what’s going on.

Hyper-V did not find virtual machines to import from the location . The operation failed with error code ‘32784’.

I got contacted by some people how ran into some issues importing VMs from W2K12R2 Hyper-V into W2K12 Hyper-V. They got bitten by this “little” issue: Importing a VM that is exported from Windows Server 2012 R2 into Windows Server 2012 is not supported

This means you get greeted by

Hyper-V did not find virtual machines to import from the location <folder location>.
The operation failed with error code ‘32784’.

image

No the trick of not exporting the VM but doing an “in place” registration doesn’t cut it. That’s great for W2K8R2 to W2K12 or W2K12 to W2K12R2 but not from W2K12R2 to a lower version. In that way the title of the KB article could be seen as a bit misleading or incomplete, but the contents is pretty clear.

And that’s it. Woeps! What you have 200 VMs on the LUNs form the old cluster you already blew away to build the new one? You do have a tested exit plan for this right? Uh no?

Facepalm Combo

Oh MAN, NOOOOO!

Now if it’s only one or two VMs you can always work around this by creating new VMs using the old VHDXs. This will leave you to deal with networking cleanup inside of the VMs and configuring TCP/IP. PowerShell can help here but in large volumes this remains as serious effort. This is also the time that documentation pays!

Now what if this happens to you when you’re trying to roll back a migration of a hyper-V cluster (revert W2K12R2 to W2K12 for example). Well for one you should have know as you did test all this right? Right?!

What are your other options to roll back other than  the above? From the top of my head and without details?

  • Move back to your old cluster Smile You didn’t already nuke it, I hope.
  • If you have a SAN take a snapshot of the LUNs before you move them to Windows Server 2012 R2 for faster fall back. But beware, if you’re running applications that require some tender loving care in relation to snapshots like Exchange  or Active Directory in those VMs … shutting all VMs down before you create the can help snapshot mitigates issues but is not a full proof approach! “Know thy apps”!
  • A great backup & RESTORE solution to get you back up and running also comes in handy but don’t forget that it requires you to know your apps as well here. Yes, it’s not always just “CLICKEDYCLICKCLICKDONE”
  • Perhaps it’s now time to activate your paused replicas on the DRC cluster or hosts?  You did test this didn’t you?

Now for anyone involved in a migration to Windows Server 2012 R2 there is no excuse not to know this in advance and to test out the new cluster hardware as much as you can. This minimizes the chance you’ll need to fall back. And please test your exit scenarios, really, I mean it.  Also please, you can migrate one LUN/CSV at the time. Try to run the VMs on the first migrated LUN/CSV before you do all the others. That way you can do some damage control.

Now, this is not great but it is what it is and at least now you know before your migrate Winking smile. We’ve also asked MSFT to make falling back a bit less “"involved” in future versions. Perhaps they’ll do that, I’m pretty sure they’ll consider it. And by what we’ve seen in the recently available Technical Preview they did!

Microsoft Hyper-V S3 Cap warning when upgrading a Hyper-V Virtual Machine

When you do an in place upgrade of a Hyper-V virtual machine you’ll get a warning that Microsoft Hyper-V S3 Cap may not work after the upgrade and that you need to update the driver prior to the upgrade.This warning is logged to the Windows Compatibility Report.htm.

image

Microsoft Hyper-V S3 Cap is an old S3 Trio 765 emulated video device and the driver isn’t included anymore so you’ll get this particular warning. This will never give you an issues, all drivers needed are indeed in the install bits. You can safely ignore this and successfully upgrade.

Some people uninstall the device via device manager but basically that’s pure cosmetics & doesn’t really serve a purpose.

This warning is an artifact of the generation 1 virtual machines who still have this device on a PCI bus.  Below is a screenshot of a VM with W2K12R2, generation 1. As you can see the Microsoft Hyper-V S3 Cap is perfectly fine. No worries.image

As a matter of fact you will not even see this device on a generation 2 virtual machine and we should not see this with an upgrade of those.

image

I will have to wait on a public preview of Windows vNext to test an upgrade of a generation 2 machine to prove my thinking that this cosmetic error won’t be there anymore.

Online Resizing Of Hyper-V Virtual Disks Is Possible in Windows 2012 R2

Windows Server 2012 R2 brought us the ability to resize virtual disks on line. This was a long sought after feature for many of us. It can be done via the GUI or with PowerShell. I do note however that quite often people have some problems when first using this capability. So we’ll go over the rules & prerequisites here.

Listed below are the important factors to keep in mind

  1. It has to be a VHDX
  2. Works for both generation 1 and generation 2 virtual machines
  3. It needs to be attached to a vSCSI controller. Remember this when dealing with with generation 1 virtual machines. In particular note that this means you cannot live resize the system disk as that IDE only (can’t boot from SCSI in generation 1).
  4. The virtual disk cannot be a shared VDHX (it’s on my feature request list for vNext)
  5. You can extend a virtual disk
  6. You can shrink a virtual disk
  7. This feature can leverage ODX for speed when available. The speed of this is quite addictive.

Some notes where people seem to make some other mistakes

You’ll note that you cannot shrink a virtual disk that has no unallocated disk space on the disk inside the virtual machine. When you see this picture inside of the virtual machines you can shrink your VHDX if all the above factors are in order.image

image

If there is no unallocated disk space the option to shrink the VHDX won’t even show up in the GUI.

image

This means you’ll first need to shrink the volume inside the virtual machine if all disk space has already been allocated.

image

(Like wise don’t forget to expand the volume inside the VM to be able to use the added space you see show up as unallocated space on the disk.)

The below image is a great summary of the above

image

Legacy OS in the VM?

It’s also important to note that an OS inside a VM (Windows Server 2003 comes to mind) that does not allow the expansion or shrinking of volumes means you will need a 3rd party tool to do the same. I use GParted, a free partition editor for these scenarios.