Déjà vu Bug: The network connection of a running Hyper-V virtual machine may be lost under heavy outgoing network traffic on a computer that is running Windows Server 2008 R2 SP1

Posted on by
9

Anyone who’s been doing virtualization with Hyper-V on Windows 2008 R2 has a good change of having seen the issue described in http://support.microsoft.com/kb/974909/en-us

You install the Hyper-V role on a computer that is running Windows Server 2008 R2.

  • You run a virtual machine on the computer.
  • You use a network adapter on the virtual machine to access a network.
  • You establish many concurrent network connections, or there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter is disabled.
Note You have to restart the virtual machine to recover from this issue.

We’ve seen this one on VM’s that have indeed a lot of outgoing traffic.  In our environment the situation looks like this:

  • You can access the VM with Hyper-V Manager or SCVMM but not via RDP as all Network connectivity is lost.  The status the  guest NIS is always “Enabled” but there is no traffic/connectivity
  • You can try to disable the NIC but this tales a  very long time and when you try to enable it again this never succeeds. Disconnecting the NIC form the virtual network and connecting it again doesn’t help either.
  • You need to shut down the host but this takes an extremely long time, so long you really can’t afford to wait if it ever succeeds. It seems to hang at shutting down with a “non whirling whirly”.  So finally you’ll power off the VM and start it up again. Apart from entries related to having not connectivity the event logs are “clean” and there is no indication as to what happened.

Well this exact same issue is back with Windows 2008 R2 SP1. That’s the bad news. The good news is there is a hotfix for it already so you can fix it. You can read up on this issue in Knowledge Base article 2263829  and request the hotfix here. Instructions to get the hotfix are in there as well as a reference to the previous fixes for Windows 2008 R2 RTM.

Consider the following scenario:

  • You install the Hyper-V role on a computer that is running Windows Server 2008 R2 Service Pack 1 (SP1).
  • You run a virtual machine on the computer.
  • You use a network adapter on the virtual machine to access a network.
  • You establish many concurrent network connections. Or, there is heavy outgoing network traffic.

In this scenario, the network connection on the virtual machine may be lost. Additionally, the network adapter may be disabled.
Notes

  • You must restart the virtual machine to recover from this issue.
  • This issue can also occur on versions of Windows Server 2008 R2 that do not have SP1 installed. To resolve the issue, apply the hotfix that is described in one of the following Microsoft Knowledge Base articles:

    974909 (http://support.microsoft.com/kb/974909/ ) The network connection of a running Hyper-V virtual machine is lost under heavy outgoing network traffic on a Windows Server 2008 R2-based computer
    2264080 (http://support.microsoft.com/kb/2264080/ ) An update rollup package for the Hyper-V role in Windows Server 2008 R2: August 24, 2010

Oh yeah, people often seem confused  as to where to install the hotfix. Does it go on the Hyper-V hosts or and/or on the guest?  It’s a hyper visor bug in Hyper-V so it goes on the hosts. Have a nice weekend.

Cluster Validation Bug In Windows 2008 R2 SP1 – Disk has a Persistent Reservation on it

Posted on by
2

Pretty soon after the RTM of Windows 2008 R2 SP1 release we were discussing a bug on the TechNet forum (Hyper-V Cluster issues after applying Win2008 R2 SP1 on a 3 node Cluster!) here. If you have a Windows 2008 R2 SP1 cluster with more than 2 nodes you get the following warning:

List Potential Cluster Disks

Disk with identifier 2sef8cdf has a Persistent Reservation on it. The disk might be part of some other cluster. Removing the disk from validation set

“Normally” you would expect such a warning if the LUN ever belonged to another cluster and it needs the old reservation cleared. To do that you would use following command on the node that throws the warning (where in this example the disk is disk 2 in disk manager/diskpart) and after making sure it is not in use anywhere else in the SAN

"cluster node clusternode1 /clearpr:2"

However this is not the cause here as were most others in this discussion. And I’m pretty no san software or MPIO software is putting a reservation on there either so what is this? A bug? Well yes, it has been confirmed by Microsoft support that is is indeed a bug an that is fix will be made available by April 18th2011 .

This was not a show stopper bug, but it could be one if you needed to add a host to a cluster and confirm all is well and supported. However if you’re certain you’ve done everything right you can choose not to run cluster validation.

I will update this blog with more information when the fix becomes available.

UPDATE:  The hotfix has become available today, April 26th 2011 as announced on the TechNet forum here:

A hotfix is now available that addresses the Win2008 R2 service pack 1 issue with Validate on a 3+ node cluster.  This is KB 2531907.  The KB article and download link will be published shortly, in the mean time you can obtain this hotfix immediately free of charge by calling Microsoft support and referencing KB 2531907. Update 27/05/2011 Here is the link: http://support.microsoft.com/kb/2531907/en-us?sd=rss&spid=14134

Microsoft re-releases Exchange 2010 Service Pack 1 Update Rollup 3 (V3)

Posted on by
Reply

Due to an issue with Blackberry devices Exchange Server 2010 SP1 Update Rollup 3 was withdrawn on March 14th. See my blog on this here.

The issue was identified and fixed so now there is a new release. My environment doe not use Blackberries so we were not impacted, We had already deployed RU3 so all we have to do is apply the current release over that one and we’re good to go.

It’s a good thing it’s back available since if bring back UDP support which is a much asked and need feature for Outlook 2003 and it also fixed the DAG copy GUI bug introduced  with SP1  (September 2010) where you could see and thus mange the copies anymore with EMC.

So all in all good news and we can all move ahead now.

Microsoft iSCSI Software Target 3.3 for Windows Server 2008 R2 available for public download

Posted on by
1

As TechNet subscribers, we had access to Windows Storage Server 2008 with Microsoft iSCSI Software Target 3.2  (also see Jose Barreto’s blog on this here). That was sweet but for one little issue. This SKU cannot be a Hyper-V Host. In order not to lose a physical host in the lab you could edit the MSI installer from the Windows Storage Server 2008 install media where you would delete the SKU check. Problem solved but not very legal so nobody ever did that.  You can install Windows Storage Server in a VM for the lab I know but that becoming very SkyNet like … Virtual servers providing virtual storage for virtual servers … and while a good option to have I like to have a hardware host.

Bring Windows Storage Server 2008 R2 along and Microsoft decided that we could have the iSCSI Software Target 3.3 software without constraints, except that you needed a TechNet/MSDN subscription, to install on W2K8R2. This is the one I’m running in my labs at the moment installed on a Physical Windows Server 2008 R2 Enterprise edition that also is a Hyper-V host. This provides all my iSCSI storage to both physical and virtual clusters. I used it to test MelioFS with FileScaler recently with a 2 node virtual cluster.

Today, Jose Barreto blogged about the public release of iSCSI Software Target 3.3 for Windows Server 2008 R2. This is very good news as now everyone has access to an iSCSI target for labs, testing, POCs, and even production. Thank you, Microsoft. Now with some luck, we could get some SMI-S support for it with SCVMM2012? Please?

If you need some help, Jose Barreto has a bunch of blog posts on configuring the iSCSI target, so I suggest you check out his site. As an added benefit, Microsoft iSCSI Software Target 3.3 setup & configuration is scriptable using PowerShell.