High Availability has a price

Posted on March 15, 2016 by workinghardinit

We’ll go back to basics today. Some times the obvious, no matter how evident it is to us technologists, is challenged. Recently we got the remark that we were wasting CPU cycles by assigning to many vCPU to certain virtual machines on our Hyper-V cluster. So we had to explain that high availability has a price. On top of that we had to explain that things are not as wasteful as they seem in a virtual environment.

The case

Here’s one of the “offending” virtual machines. They assumed that we are wasting at least 50% of 12 CPUs.

This is one node in a dual node load balancing (active-active) and highly available solution. This provides for zero down time during scheduled maintenance and very little downtime during system failures.

And here’s the second node (yes the 1st node has been down for scheduled maintenance more recently that node 2).

In a 2 node HA solution you need to make sure that one node can handle the entire workload. This is the absolute border line of an N+1 solution. This means you can lose 1 node. N determines the number of nodes needed to guarantee an agreed upon service level and the number defines how many nodes failures can be tolerated before affecting the service.

In the above example there’s a need to have the CPU resources on each node to run the entire workload on one node without having an effect on the service. Therefore, when both nodes are up this might seem like a waste to the uninitiated. It is however a required to achieve the high availability goal. A constant CPU usage over 75 % will lead to a reduction in service quality in this case and even compromise the usability of the that service.

I did not even dive into the dangers of designing purely based on averages during this “explanation”. That was one step to much for the level of the discussion.

It’s also important to note that Hyper-V CPU scheduling is highly intelligent and is far less susceptible to the waste of CPU cycles via over provisioning of vCPU than some other solutions are or used to be. Knowing the capabilities and inner working of the technology used is also important in all this. More nodes generally also make “over provisioning” less of an issue. When you have 10 nodes and you lose 1, you only have lost 10% of the capabilities, not 33% like in a 3 node cluster.

Ideally you have 3 node so that even during an issue with one node you still maintain redundancy. However if you want acceptable services during a 2 node failure you’ll need to go to N+2, meaning that you need 2 nodes to provide the services and handle losing 2 nodes gracefully. In that case you’ll need 4 node and so on. The larger the node count the wiser it is to go to a N+2 model and ideally you’ll provide separate failure domains over which the nodes are distributed. An example of this is having a redundant geo-load balanced web farm of 32 virtual machine nodes spread over 2 locations and running on separate hardware failover clusters in each location. As you can see the higher the stakes and demands the faster the cost and potential complexity rises. You can offload some of the complexity by leveraging a public cloud like Azure, but the costs will still be there. There is no such thing as a free lunch, some are quite easy and affordable for what you get.

Conclusion

High Availability has a price. I did mention that already, right? To be able to keep your services running at a level that is both workable and acceptable to your customers and stake holders you will need to over provision to a degree. There is no magic here. When your solutions are being scrutinized by people with no real background, experience and context in high availability you might need to explain this.

Get-VMHostSupportedVersion Windows Server 2016 Hyper-V

Posted on March 7, 2016 by workinghardinit

Windows Server 2016 Technical Preview 4 has this great PowerShell command to check what virtual machine versions the Hyper-V host supports: Get-VMHostSupportedVersion

When you run this the output on Windows Server 2016 TPv4 is as below.

As you can see it supports 5.0 (which translates to the version on Windows Server 2012 R2). That’s logical, it’s the version your migrated VMs will have when you move them to Windows Server 2016 as the VM version is no longer automatically updated. You can read more about this here: https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/migrating_vms

You must also realize that updating from 5.0 to higher means moving to the newer Virtual machine configuration file format, which is way better for performance and is a binary format and not longer XML.

6.2 is the version a VM had in the Technical Preview 3 era. Which is cool as when you move your VMS between TPv3 and TPV4 they still worked and were supported. Now in reality you want to be either at 5.0 so you can still move VMs back to a W2K12R2 host or at 7.0 to have all new features and capabilities that come with W2K16 at your disposal.

7.0 is also the current default and are the version any newly created VMs or any VM you update via Update-VMVersion will be at.

Now don’t panic, if you need to stay at version 5.0 even for newly created VMs for backward compatibility in mixed Hyper-V Windows Server 2012 R2 / Windows Server 2016 environment. You can specify this with the –version parameter of New-VM

New-VM -Name “IChooseMyVersion-5.0” -Version 5.0

To know what version you can pass to that parameter you only have to look at the output of Get-VMHostSupportedVersion: 5.0, 6.2, 7.0 at the time of writing (on TPv4).

You can see the result of some fun with this in the below screenshots:

I fully expect Microsoft to keep this up for TPv5 or perhaps even with RTM to allow a smooth transition for VMs running on preview editions to RTM. We’ll see. At RTM time I can see them removing the ability to create a new VM that’s still at 6.2 or so as It’s a “intermediate” work in progress version or so on the way to the final default version number at RTM (see Windows Server 2016 TPv4 Hyper-V brings virtual machine configuration version 7). We’ll see!

Shared VHDX In Windows 2016: VHDS and the backing storage file

Posted on February 9, 2016 by workinghardinit

Introduction into the VHD Set

I have talked about the VHD Set with a VHDS file and a AVHDX backing storage file in Windows Server 2016 in a previous blog post A first look at shared virtual disks in Windows Server 2016. One of the questions I saw pass by a couple of times is whether this is still a “normal VHDX” or a new type of virtual disk. Well the VHDS files is northing but a small file containing some metadata to coordinate disk actions amongst the guest cluster nodes accessing the shared virtual disk. The avhdx file associated with that VHDS file is an automatically managed dynamically expanding or fixed virtual disk. How do I know this? Well I tested it.

There is nothing that preventing you from copying or moving the avhdx file of a VHD Set that not in use. You can rename the extension from avhdx to vhdx. You can attach it to another VM or mount it in the host and get to the data. In essence this is a vhdx file. The “a” in avhdx stands for automatic. The meaning of this is that an vhdx is under control of the hypervisor and you’re not supposed to be manipulating it but let the hypervisor handle this for you. But as you can see for yourself if you try the above you can get to the data if that’s the only option left. Normally you should just leave it alone. It does however serve as proof that the VHD Set uses an standard virtuak disk (VHDX) file.

I’ll demonstrate this with an example below.

Fun with a backing storage file in a VHD Set

Shut down all the nodes of the guest cluster so that the VHD Set files are not in use. We then rename the virtual disk’s extension avhdx to vhdx.

You can then mount it on the host.

And after mounting the VHDX we can see the content of the virtual disk we put there when it was a CSV in that guest cluster.

We add some files while this vhdx is mounted on the host

Rename the virtual disk back to a avhdx extension.

We boot the nodes of the guest cluster and have a look at the data on the CSV. Bingo!

I’m NOT advocating you do this as a standard operation procedure. This is a demo to show you that the backing storage files are normal VHDX files that are managed by the hypervisor and as such get the avhdx extension (automatic vhdx) to indicate that you should not manipulate it under normal circumstances. But in a pinch, it a normal virtual disk so you can get to it with all options and tools at your disposal if needed.

WEBINAR: Troubleshooting Microsoft Hyper-V – 4 Tales from the Trenches

Posted on February 3, 2016 by workinghardinit

I’ve teamed up with Altaro as a guest in one of their Hyper-V webinars to discuss trouble shooting Microsoft Hyper-V. I’ll be joining my fellow Microsoft Cloud and Datacenter Management Andy Syrewicze for this in a virtual cross Atlantic team

We’ll give you some pointers on good practices and where to start when things go south. We’ll also add some real world examples to the mix to spice things up. As you can imagine these issues are often a lot more fun after the facts and after having solved them. If you work in IT long enough you know that one day (or night) trouble will come knocking and that the stress related with that can be gut wrenching.

The good news is that it isn’t the end of the world. In well designed and managed environment you can minimize downtime and you do not have to suffer through unrecoverable losses as long as you’re well prepared. We hope this webinar will help the attendees prevent those problems in their environment. If not, it will provide you with some insight on how to prepare for and handle them.

The webinar is on February 25th, 2016 at 4pm CET / 10am EST!

The time has been chosen to make it feasible for as many many people across all time zones to attend. So, go on, register, it’s free and both Andy and I are looking forward to sharing our trouble shooting tales from the trenches.

Working Hard In IT

My view on IT from the trenches

Category Archives: Hyper-V