BriForum 2011 Europe Here I Come

As you might have read a previous blog and noticed in the sidebar, I’m off to London (UK) to attend  BriForum 2011 Europe. It’s time to get away from the wide screens overlooking my ICT empire toys and broaden my horizons  For those who think the cloud is going to take away your job … think again, I’m getting busier than ever. The reality is that we just can’t push a button and have everything up and running in the cloud. Greenfield projects and startups might beat existing infrastructure & application architecture over the head with cloud and make those businesses run harder for their money but they will run and compete. That race will produce a huge workload.

So I’m of to dive into some sessions on Cloud, Server & Application Virtualization, VDI … should make for some interesting days. I hope to be able to talk to lots of people with a variety of experiences to help find out new or alternate ways to address some issues (or challenges) we need to tackle in the years ahead. Subjects like Disaster Recovery, Business Continuity, application-aware storage in a virtualized environment, Geo Clustering, Site Recovery, … should give us ample to discuss. Give us a shout if you’re there. It’s also a nice opportunity to meet up with some fellow bloggers and twitter. acquaintances.

A colleague of mine is heading to the USA, Atlanta to attend TechEd 2011 USA. So he’s crossing the big pond to get some brand new info on the latest and the greatest in Microsoft technologies on the IT Pro side of the business.

So of to London, I go, onwards & always going forward in IT as there is no turning back I’ll keep you posted when I find the time to do so.

System Center Virtual Machine Manager 2012 Using WSUS To Update Hyper-V Cluster Hosts & Other Fabric Servers

One very neat feature in System Center Virtual Machine Manager 2012 (SCVMM2012), which is currently in Béta, is the integration with WSUS to automate the patching of Hyper-V cluster hosts (+ the Library servers, SCVMM servers and the update servers, i.e the fabric). The fact that SCVMM 2012 will give you the complete toolset to take care of this is yet a great addition to the functionality available in Virtual Machine Manager 2012. More and more I’m looking forward to using it in production as it has so many improvements and new features. Combine that with what’s being delivered in System Center Operations Manager (SCOM2012) and the other member of the System Center family and I’m quite happy with what is coming.

But let’s get back to the main subject of this blog. Using WSUS and SCVMM2012 to auto-update the Hyper-V cluster hosts without interruption to the virtual machines that are running on it. Up until now, we needed to script such a process out with PowerShell even tough having SCVMM2008R2 makes it easier since we have Maintenance Mode in that product which will evacuate all VMs from that particular host, one by one. The workflow of this script looks like this:

  • Place the Host Node in Maintenance Mode in SCOM 2007 R2 (So we don’t get pesky alerts)
  • Place the Host Node in Maintenance Mode in SCVMM2008R2 (this evacuates the VMs from the host via Live Migration to the other nodes in the cluster)
  • Patch the Host and restart it
  • Stop Maintenance Mode on the host node in SCVMM2008R2 (So it can be used to run VMs again)
  • Stop Maintenance Mode on the host node in SCOM 2007 (We want it to be monitored again)
  • Rinse & Repeat until all Host nodes are done. Depending on the size of the cluster you can do this with multiple nodes at the same time. Just remember that there can be only one Live Migration action taking place per node. That means you need at least 4 nodes to do something like Live migrate from Node A to Node B and Live Migrate from Node C to node D. So you need to work out what’s optimal for your cluster depending on load and number of nodes you have to work with.
  • Have the virtual machines redistributed so that the last host also gets its share or virtual machines

Now with SCVMM2012 we can do this out of the box using WSUS and all of this is achieved without ever interrupting any services provided by the guests as all virtual machines are kept running and are live migrated away from the host that will be patched. If you’re a shop that isn’t running System Center Configuration Manager you can still do this thanks to the use of WSUS and that’s great news.  There is an entire sub-section on the subject of Managing Fabric Updates in VMM 2012 already available on TechNet. But it goes beyond the Hyper-V host. It’s also the SCVMM server, the library server, and the Update Server that get patched. But don’t go wild now, that’s the entire scope of this. That means you still need regular WSUS or SCCM for patching the virtual machine guests and other physical servers. The aim of this solution is to patch your virtualization solution’s infrastructure as a separate entity, not your entire environment.

So how do we get this up and running? Well, it isn’t hard. Depending on your needs and environment you can choose to run WSUS and SCVMM on the same server or not. If you choose the latter please make sure you install the SWSUS Administration Console on the SCVMM server. This is achieved by downloading  WSUS 3.0 SP2 and installing it. Otherwise, just use the WSUS role from the roles available on Windows 2008 R2. This handles the prerequisites for you as well. It is also advisable to install the WSUS role on a separate server when your SCVMM 2012 Infrastructure is a highly available clustered one. For more information see http://technet.microsoft.com/en-us/library/gg675099.aspx . Time-saving tip: create a separate domain account for the WSUS server integration, it can not be the SCVMM 2012 service domain account.

Make sure you pay attention to the details in the documentation, don’t forget to install the WSUS 3.0 SP2 Administration Console on the SCVMM 2012 server or servers and to restart the SCVMM service when asked to. That will safe you some trouble. Also, realize that this WSUS Server will only be used for updating the SCVMM 2012 fabric and nothing else. So we do not configure anything except the operating system (W2K8R2) , and the languages needed. All other options & products that are not related to virtualization are unchecked as we don’t need them. Combine this with dynamic optimization to distribute the VM’s for you and you’re golden. A good thing to note here is that you’re completely in control. You as the virtualization infrastructure / SCVMM 2012 Fabric administrator control what happens regarding updates, service packs, …

You do need to get used to the GUI a bit when playing around with SCVMM2012 for the first time to make sure you’re in the right spot, but once you get the hang of it you’ll do fine. I’ll leave you with some screenshots of my lab cluster being scanned to check the compliance status and then being remediated. It works pretty neatly.

Here are the hosts being scanned.

You can right-click and select remediate per baseline or select the host and select remediate form context menu or the ribbon bar.

The crusader host is being remediated. I could see it being restarted in the lab.

Extra Info on Clustering & Hyper-V with Dynamic Memory When You Start With Windows Server 2008 R2 SP1:

Here’s a quick “heads up” if your starting to use or thinking about using Windows Server 2008 R2 SP1 for your Hyper-V clusters. The most common issues I’ve seen in the wild are:

  1. https://blog.workinghardinit.work/2011/04/01/kb2230887-hotfix-for-dynamic-memory-with-windows-2008-standard-web-edition-does-not-apply-to-without-hyper-v-editions/ This one is being worked on and the hotfix will be re-released to support the “Without Hyper-V” SKU of Windows Server 2008 SP2.  It’s a simple oversight but one that can be important when your Hyper-V clusters are filled with that SKU.
  2. We also got bitten by this one Déjà vu Bug: The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1, but the hotfix was already available luckily.
  3. And than one to head and to read the TechNet forum about Cluster Validation Bug In Windows 2008 R2 SP1 – Disk has a Persistent Reservation on it. They are also working on a fix. I’ve written a blog post on this and I suggest you read it and also take note of the discussion in the TechNet forum.

    UPDATE: The hotfix for issue 3 has become available today, April 26th 2011 as announced on the TechNet forum here:

    A hotfix is now available that addresses the Win2008 R2 service pack 1 issue with Validate on a 3+ node cluster. This is KB 2531907. The KB article and download link will be published shortly, in the mean time you can obtain this hotfix immediately free of charge by calling Microsoft support and referencing KB 2531907.   Update 27/05/2011 Here is the link: http://support.microsoft.com/kb/2531907/en-us?sd=rss&spid=14134

An other one that I haven’t seen in the wild is:

Windows Server 2008 R2 installation may hang if more than 64 logical processors are active. There is is a workaround and a hotfix for this one.

Issue: When you try to install Windows Server 2008 R2 on a computer that has more than 64 logical processors, Windows Setup may stop responding in one of the following operations:

  • Initialization of Windows Setup
  • One of the two restarts that are required to complete Setup

Cause: This issue occurs because of an error in the Network Driver Interface Specification.This issue occurs because of an error in the Network Driver Interface Specification (NDIS) driver.
When a computer has more than 64 logical processors, the NDIS driver does not correctly handle some operations. Therefore, the computer encounters stop responding issues and other system failures.

I don’t have any nodes under my care who have more than 64 logical processors so that’s why I guess Smile But with ever more cores available you it’s bound to happen in the near future.

Update 2: To keep me busy this KB article was released within 24 hours of me posting this blog on a BSOD with CSV and redirected access for witch a hot fix is available

Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access

The KB article with instructions on how to get the hot fix is here: http://support.microsoft.com/kb/2494016/en-us?sd=rss&spid=14134

The scenario is detailed as follows:

Consider the following scenario:

  • You enable the cluster shared volume (CSV) feature on a Windows Server 2008 R2-based failover cluster.
  • You create a virtual machine on the CSV on a cluster node.
  • You start the virtual machine on the cluster node.
  • You move the CSV owner to another cluster node, and you change the state of CSV to redirected access.
  • The connection that is used for redirected access is switched to another connection when one of the following scenarios occurs:
    • The cable for local area network (LAN) is disconnected.
    • The related network adapter is disabled.
    • The connection is switched by using Failover Cluster Manager.

In this scenario, you receive a Stop error message that resembles the following in the virtual machine:

STOP 0x0000007a ( parameter1 , parameter2 , parameter3 , parameter4 )
KERNEL_DATA_INPAGE_ERROR

Note

  • The parameters in this Stop error message vary, depending on the configuration of the computer.
  • Not all "0x0000007a" Stop error messages are caused by this issue.
  • You may also receive other Stop error messages when this issue occurs. For example, you may receive a "0x0000004F" Stop error message.

BriForum Europe 2011 & The Experts Conference Europe 2011

Great news from the educational & conference front. First of all, I’m attending BriForum in London, United Kingdom in May (http://briforum.com/Europe/index.html).  That’s good news, normally we’d have to pop over the big pond to go to that one, so this is pretty neat. And timely, due to some prospecting I’m doing for Disaster Recovery,  Business continuity, application aware storage in a virtualized environment It’s a good match and I hope to get in to some educational discussions about the challenges we all face. Some of the storage vendors we’re interested in are there as well so there is certainly some potential to make it a good experience.

And just recently confirmed that The Experts Conference is coming to Europe. TEC2011 Europe will be held in Frankfurt, Germany from October 17th to October 19th 2011. This conference is high quality and created to fill the needs of the most experienced users, which is one of the reasons I would like to attend. The more you learn & grown the more you bump into the next level of challenges and being able to learn form high level content and interact with experienced speakers and attendees who are dealing with the same issues can be very rewarding. Attendees of TechEd have a way to measure the level of the sessions, well, they are all supposed to be Level 400 only. Quest is hosting this, so they certainly should be able to round up the expertise.  I’m going to make it to the new “track” at this conference and that’s “Virtualization & Cloud”. More information can be found here http://www.theexpertsconference.com/europe/2011/virtualization-cloud-training/overview/

The timing of these conferences is pretty good. As I said we’re doing a lot of prospecting right now and hope to get a lot of information from attending these. For anyone interested why I attend conferences and why I think they are valuable see mu blog post on this subject https://blog.workinghardinit.work/2010/06/05/why-i-find-value-in-a-conference/