Déjà vu Bug: The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1

Anyone who’s been doing virtualization with Hyper-V on Windows 2008 R2 has a good change of having seen the issue described in http://support.microsoft.com/kb/974909/en-us

You install the Hyper-V role on a computer that is running Windows Server 2008 R2.

  • You run a virtual machine on the computer.
  • You use a network adapter on the virtual machine to access a network.
  • You establish many concurrent network connections, or there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter is disabled.
Note You have to restart the virtual machine to recover from this issue.

We’ve seen this one on VM’s that have indeed a lot of outgoing traffic.  In our environment the situation looks like this:

  • You can access the VM with Hyper-V Manager or SCVMM but not via RDP as all Network connectivity is lost.  The status the  guest NIS is always “Enabled” but there is no traffic/connectivity
  • You can try to disable the NIC but this tales a  very long time and when you try to enable it again this never succeeds. Disconnecting the NIC form the virtual network and connecting it again doesn’t help either.
  • You need to shut down the host but this takes an extremely long time, so long you really can’t afford to wait if it ever succeeds. It seems to hang at shutting down with a “non whirling whirly”.  So finally you’ll power off the VM and start it up again. Apart from entries related to having not connectivity the event logs are “clean” and there is no indication as to what happened.

Well this exact same issue is back with Windows 2008 R2 SP1. That’s the bad news. The good news is there is a hotfix for it already so you can fix it. You can read up on this issue in Knowledge Base article 2263829  and request the hotfix here. Instructions to get the hotfix are in there as well as a reference to the previous fixes for Windows 2008 R2 RTM.

Consider the following scenario:

  • You install the Hyper-V role on a computer that is running Windows Server 2008 R2 Service Pack 1 (SP1).
  • You run a virtual machine on the computer.
  • You use a network adapter on the virtual machine to access a network.
  • You establish many concurrent network connections. Or, there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter may be disabled.
Notes

  • You must restart the virtual machine to recover from this issue.
  • This issue can also occur on versions of Windows Server 2008 R2 that do not have SP1 installed. To resolve the issue, apply the hotfix that is described in one of the following Microsoft Knowledge Base articles:

    974909 (http://support.microsoft.com/kb/974909/ ) The network connection of a running Hyper-V virtual machine is lost under heavy outgoing network traffic on a Windows Server 2008 R2-based computer
    2264080 (http://support.microsoft.com/kb/2264080/ ) An update rollup package for the Hyper-V role in Windows Server 2008 R2: August 24, 2010

Oh yeah, people often seem confused  as to where to install the hotfix. Does it go on the Hyper-V hosts or and/or on the guest?  It’s a hyper visor bug in Hyper-V so it goes on the hosts. Have a nice weekend.

Kick Starting Your Windows 7 Deployments With Mastering Windows 7 Deployment

I have to hand it to Aidan Finn, he doesn’t stop at sharing information via his blogs or the community. He joined forces with Darril Gibson & Kenneth van Surksum went the extra mile. The wrote a readable, useful book Mastering Windows 7 Deployment about a subject on which consolidated documentation is scarce, scattered around the internet or written badly so you still can’t figure it out or is to boring you just don’t read it. If I need to define the goal of this book: get people a good head start for Windows 7 deployments in a planned and organized fashion.

This is not a book for the absolute newbie who doesn’t know the difference between a local and a domain account. It isn’t targeted at the WDS/MDT experts who’ve solved, fixed and worked around any and all PXE boot, network errors, cryptic WDS or MDT deployment errors & configuration challenges known to man kind. In that case this stuff is known to you (or should be). The point is those experts have already learnt a lot the hard way and they put in a considerable effort to do so. But knowledge needs to be transferred and spread around and to do that you need to cover the basics and work up from there, showing progress and results. The progress and results motivate people.

In that respect, this books get’s you started on that path from chapter one and by page 5 you’re already being guided into auditing & reporting via MAPS to prepare a roll out proposal. The effort put into discussing the Application Compatibility Toolkit (ACT) is important. I remember the work that we needed to do for Vista x64 bit and how that paid off when deploying Windows 7. What surprises me it that a lot of IT Pro’s don’t even know about the ACT, file and registry virtualization or shims. I recommend another blog on this subject http://blogs.msdn.com/b/cjacks/ , Chris Jackson, the “App Compat Guy” and a very good conference speaker on the subject. The scenarios with the User State Migration Tool will benefit system administrators who dread touching end users their PC and the precious data it might contain. If so, I hope you are backing up the data on those workstations, if not than that is really scary.

Perhaps some readers will already be using certain tools touched upon in the book but not others. In that case this is a great way to start with them and see where they fit in and what they can do for you. We did Vista x64 bit deployments in 2008 with WDS; rolled out Windows 7 x64 in 2010 using WDS/MDT and I still found this book interesting enough to buy some copies and add it to the toolkit of my team. What I’d like to add as a useful hint: look into disable rearming by using <SkipRearm>1</SkipRearm> in the unattended XML file you can pass to sysprep as in “/generalize /quiet /unattend:<file_name.xml” so you don’t run into a when you do it more than 4 times on the same image (An error message occurs when you run "Sysprep /generalize" in Windows Vista or Windows 7: "A fatal error occurred while trying to Sysprep the machine").

The Microsoft Deployment Toolkit (MDT) sections point you directly to some gems we found very useful in our deployments. That you can pre stage computers in the MDT database to help make the roll outs as “light touch” as possible is cool, but that you can automate that with the MDT PowerShell module makes it really very valuable. See http://blogs.technet.com/b/mniehaus/archive/2009/05/15/manipulating-the-microsoft-deployment-toolkit-database-using-powershell.aspx for more details. Michael Niehaus is to MDT what Chris Jason is to ACT. As identifier we use the MAC address as we get that on a label on the PC and we can easily get a list of those to mass import them together with creating the computer objects in Active Directory. We also added driver profiles depending on the client make & model. When you combine this with boot from PXE provided by WDS to boot to an MDT WinPE, and remember WDS also gives you multicast, you have a real sweet solution going. This is the route we went last year and has served us well (we came from a pure WDS solutions, and RIS before that when we still did XP rollouts but that was more than 4 years ago Open-mouthed smile … time flies.

Task sequencer is a gem that we indeed also use to roll out certain default software like 7zip, a pdf reader, ISO burner, anti malware, etc. The fact that these are not in the image makes it very easy to deploy newer versions as they come available.

The chapter on KMS, VAMT, volume licensing will be of use to people who have never dealt with it coming from Windows 2003/XP

This book will come into its own for any SME or enterprise departmental system administrator with who needs to be launched swiftly and on his or her way to their targets, which are smooth Windows 7 deployments. A lot of production system administrators are in the progress of looking at Windows 7 and might have a lot of experience with Windows XP and Windows 2003 but not with Windows 2008(R2) and Vista/Windows 7. If you’re in that bracket you’re definitely going to get a kick start with this book and it contains some neat tips and tricks to get over some initial gotchas. Don’t think that this is for big enterprises only. Apart from the system center products most tools are free downloads or a part of the Windows server license you already own.

As always, the only way to understand technologies is to work with them, use them. That’s the way to gain insight, experience, and context. So play with this stuff in a lab. Run into a bunch issues and fix them. If you need to get up to speed with all this stuff then you should dig into this book with a hands on approach. The book will also help you make more sense of other information out there and you’ll be able to put that into context better. As a bonus, I’m pretty sure that anything you learn from it will help you with deploying Windows vNext as well.

Hyper-V Dynamic Memory does not work on a Windows Server 2008 Standard Edition or Windows Web Server 2008 virtual machine (VM)

Here’s a pointer to a Microsoft Knowledge base article on Hyper-v Dynamic Memory not working on Windows 2008 Standard & Web edition Hotfix: Hyper-V Dynamic Memory does not work on a Windows Server 2008 Standard Edition or Windows Web Server 2008 virtual machine (VM) As you probably already know you have a similar issue with Window 2008 R2 Standard and Web edition virtual machines which is fixed by installing SP1 in the guest. For the predecessor of R2 you need to install this hotfix.

The cause is that on these versions of the operating systems the the required memory enlightenment is supported. After installing the hotfix (or SP1 in the case of Windows 2008 R2) memory addition enlightenment is available on these SKU and your good to go.

Windows 2008 R2 SP1 Publicly Available & Update on Error 0x00f0818

Yesterday, late last night in Europe, Windows Server 2008 R2 SP1 became available to the general public. That means it will be pushed via Windows Updates or that you can download it manually here Windows 7 and Windows Server 2008 R2 Service Pack 1 (KB976932)

I’ve been busy doing deployments in the lab but also in production at several locations. The speed of a deployment depends on the host and what’s running on that host. I’ve seen anything between 22 minutes and 65 minutes. For a walk through of an install see my previous blog post on this Upgrading a Hyper-V R2 Cluster to Windows 2008 R2 SP1

I’ve also noticed an increased amount of hits for my blog on error 0x00f0818 when installing the Windows 2008 R2 SP1 Beta. The same solution holds true for RTM. More info on that is available at Windows 2008 R2 SP1 Beta Install Gone Wrong: Service Pack Installation failed with error code 0x800f0818

I had one place where dozens of VMs had this issue so in order to progress quickly we scripted the replacement of the C:WindowsservicingPackages folder content with know good files in bulk. Read the above mentioned article to learn about the security settings you’ll need to address in any automated solution (use takeown F c:WindowsServicingPackages /D y /R & cacls c:WindowsServicingPackages /E /T /C /G “UserName”:F  to take care of business). That saved us the time to check on each affected server individually what packages were involved. This works but test this before using it in production and don’t forget to return the security settings to the default when you’re done. In this case it was a rather large lab environment.