Upgrading a Hyper-V R2 Cluster to Windows 2008 R2 SP1

For all you people waiting to roll out Windows 2008 R2 SP1 to your Hyper-V cluster here’s a quick screenshot rich run through of the process. Some people claim you need to shut down the cluster services and shut down the guests but this is not the case.  You can do a rolling upgrade and your guests can stay on line on the other nodes, just use live migration to get them there. Now I do recommend to upgrade all the nodes tot SP1 as soon as possible and not staying a mixed Windows 2008 R2 / Windows 2008 R2 SP1 situation in your cluster. But this mixed situation makes upgrades of the nodes in the cluster possible without any down time for the guests (if you have live migration), which is the aim of having a high availability cluster.

Walk Through

Live migrate all the guests from the node you wish to upgrade to SP1. Make sure the host is fully patched and disable any antivirus services if you are running any. I always reboot the node before a major upgrade to make sure we have the server in a clean state with  no lingering reboots waiting  or processes can cause issues.

Navigate to the service pack 1 file for Windows 2008 R2, it’s called windows6.1-KB976932-X64.exe and start it up:

You’ll have to accept the license terms:

And then the preparation process starts:

It is now ready to start the upgrade and yes we want it to reboot automatically when needed:

The upgrade process takes a while (about 17 minutes on my lab servers):

 

When it’s done it will reboot and bring you back to the logon screen. Multiple reboots might be needed to complete the upgrade process depending on what’s running on your server. In this case, we are dealing with dedicated Hyper-V nodes.

View when connected to the console

View when connected via RDP

After logging on you are greeted with this window:

And yes this is indeed the case

Reboot included the entire process took about 22 to 23 minutes. In the setup event log you’ll find these messages:

  • Initiating changes for package KB976932. Current state is Absent. Target state is Installed. Client id: SP Coordinater Engine.
  • Package KB976932 was successfully changed to the Installed state.

Note: if an extra reboot is required you’ll see an extra entry in between these stating: A reboot is necessary before package KB976932 can be changed to the Installed state.

When you have a cluster with nodes running both W2K8R2 TM and W2K8R2 SP1, mixed situation so to speak,  you’ll see the following notification in the cluster events:

You can live migrate the guest from the next node to the node already upgraded to SP1 and than repeat the process. You keep doing this until all your nodes are upgraded.

As a final recommendation, I would suggest waiting until you get the SCVMM2008R2 SP1 bits is you use this product before you upgrade your clusters especially when using this with SCOM2007R2 PRO Tips. Otherwise, you don’t need to wait just realize that until you have SP1 for SCVMM2008 R2 you won’t be able to use the new functionality for Hyper-V. In production, I would not recommend using the RC1 for this.

Please do not forget to update your guests with the new SP1 version of the Hyper-V Integration Components. This is needed to be able to use the new features like Dynamic Memory & Remote FX. The Windows 2008 R2 RTM version of the Integration Components is  6.1.7600.16385:

image

You can do this using Hyper-V Manager through selecting “Insert Integration Services Setup Disk”  and running the setup, this will require a reboot.

Click to start the upgrade process.

It will ask to upgrade or repair the previous version:

Work in progress:

Done and asking for a reboot:

SCVMM2008R2 can also be used, here you shut down the guest before updating the virtual guest services as it’s called in SCVMM2008R2. It can be annoying that the nomenclature differs. The good thing here is that you can upgrade multiple guest using VMM2008R2. Hans Vredevoort did a blog post on this here: http://www.hyper-v.nu/blogs/hans/?tag=windows-server-2008-r2-sp1.  After the upgrade you can see that the version of the Integration Components  for Windows 2088 R2 SP1 is  6.1.7601.17514:

Windows 2008 R2 / Windows 7 is RTM today (for real this time)

After an initial false start (https://blog.workinghardinit.work/2011/01/14/windows-2008-r2-windows-7-sp1-rtm-today/) Window 2008 R2/Windows 7 Service Pack 1 has been officially RTMed today . SP1 brings Dynamic Memory& RemoteFX to hyper-V virtualization. I’m probably doing a SQL Server virtualization project and thus I’m very interested in the ability to disable NUMA spanning (watch Ben Armstrong’s Tech Ed 2010 presentation here) http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/06/10/talking-about-dynamic-memory-the-movie.aspx) when beneficial to do so . Which is good news. Until now SQL Server Hyper-V host where better of using machine with lesser CPU sockets & and SSL server VM’s that don’t consume more RAM than that CPU socket can address directly to avoid this.  Until now, for the  environment at hand, I’m leaning to virtualizing SQL Server on it’s own Hyper-V cluster for that reason. It will have to be confirmed in a test environment to see how big the impact is. Systems do differ and get better every year. Perhaps I’ll get back to that subject later. Anyway The bits should be on TechNet/MSDN on February 16th 2001 and available to the general public on February 22 2011. Read the announcement by Microsoft here Windows Server 2008 R2 and Windows 7 SP1 Releases to Manufacturing Today

Keeping The Same Server Name & IP Address When Moving A Domain Controller, DNS, DHCP & WINS Server To New Hardware

Introduction

At one place I provide Infrastructure consulting  & services the supportable life of some servers running Active Directory, DNS, WINS and DHCP has come and gone (+5 years old DELL PE1850) so I was asked to renew them. These are in fact the servers that where upgraded to Windows 2008 R2 some 16 months ago right after Windows 2008 R2 went RTM. See this blog post about that: https://blog.workinghardinit.work/2010/01/22/dell-pe1850-domain-controller-upgrade-to-windows-2008-r2/.

For several reasons not all domain controllers or core infrastructure servers should be virtualized so this was not just a matter or doing a P2V migration and be done with it. That would have made it very easy as not only are they already running on the latest Windows version but they had some extra requirements. They wanted to preserve the server names and IP addresses for these servers. Normally DHCP takes care of the DNS servers for clients. Servers that have static IP addresses without using DHCP reservations can be dealt with via scripting. But al lots of devices that need a DNS Server are a pain to deal with. There are devices that do not accept FQDNs, can’t be DCHP clients. Often a management solution for settings the IP configurations on hardware devices is not in place even when does tools exists (i.e. for network switches). Even if such software is in place more likely than not it doesn’t cover all devices (printers, copiers, scanners, load balancers, …) You’re lucky if they know and have a list of all devices that need their DNS entries changed, sometimes you don’t even have that. So I understand the reasons behind their request. In a perfect world this might not be needed but I’ve seldom see that in reality.

So the names and IP addresses needed to be preserved. That is can be done with some preparation and planning.  There is the approach of restoring the server to new hardware using a Windows backup but that’s not without its own issues. You could use one of the (paying) P2P migration tools that comes with various backup solutions or dedicated disaster recovery tools. To do it for free and in the cleanest possible manner I chose to go with the approach you’ll find below. There are multiple domain controllers with DNS, WISN and DHCP set up for redundancy. This means we can take these out of production one by one to recuperate the names and IP addresses. If you didn’t have that you could even introduce a temporary one (virtualized) to give you that option.

Imagine you have to servers named bert.big.corp (192.168.2.10/24) and ernie.big.corp (192.168.2.20) with Active Directory, DNS, WINS and DHCP running on them. These will be replaced with new hardware whilst maintaining the name and IP address. Basically you stage the new servers with Windows 2008 R2 and get everything installed what you need (Server management software from the hardware vendor, the UPS Software) and that doesn’t depend on computer name, IP address and domain membership. That way they are ready to go. You’ll need to wait to rename the servers and add them to the domain one by one until you’ve demoted the current servers one by one that use that name and IP address. So first you replace Bert and only after that has been completed successfully and has been verified you can replace server Ernie. That way you can recuperate the name and IP address without losing those services at any given time. The end result is two new servers with the name Bert (192.168.2.10) and Ernie (1692.168.2.20) and none of your network devices will need any DNS FQDNs or IP addresses changed. Do note that some devices only allow for one IP address or FQDN so they might lose DNS name resolution for the period in which the old server is taken out of service and the new server is brought into service. This is not the only way to do it. I do not like to rename domain controllers and I have other roles to migrate such as WINS and DHCP. So the DNS & Active Directory migration I do is not like the one described in http://technet.microsoft.com/cs-cz/library/dd379526(WS.10).aspx. Perhaps Microsoft can comment on the best practice of this all, but this is how I approached it. This is the first time I did this for Windows 2008 R2 in production as that hasn’t been out in the wild that long yet.

We’ll run through the process step by step for Bert. This needs to be done for every server that is being replaced. Check your Active Directory, DNS and replication is healthy. If not, fix this first. Also check out DHCP and WINS health. As always you have a good backup of the server and it roles before you attempt any of this.

Mind you can play the order of things to do according to you needs & environment but when using some option in the Server Migration Tools like –Users and –Groups you should pay attention unless you want to wind up with a truckload of users/groups from a DC created as local users and groups on a member server. Oh yeah, I’ve done that. When that happens you’ll love the fact that you know some scripting . So if you do the DHCP migration, do both export and import from a domain controller or from a member server (i.e. after demoting the domain controller). In the former case you’ll just get a bunch of warnings during the import that the users/groups already exist and are therefore skipped.

Decommissioning The Old Server

Demote the domain controller

Move all FSMO roles to another domain controller.  Make sure you also move the time server to the domain controller with the PDC role.

  • If you’ve manually altered the Active Directory replication topology, make sure you take care of that as well. If you let the KCC handle this, you’re good to go as it will do this for you. But once you intervene manually it’s your responsibility.
  • We demote the domain controller
  • Reboot and check all went well.
  • We wait for AD replication or we induce it ourselves.
  • Check AD, DNS & replication, if all is well continue.
DHCP Decommission On The Old Server

There is a great tool to do this. Check out TechNet http://technet.microsoft.com/nl-be/library/dd379545(WS.10).aspx. The benefit of using the Server Migration Tools  is that we can also migrate the IP configuration from the old server to the new one in one step! Pretty cool Open-mouthed smile! The details differ depending what version of Windows you’re running. I’m at Windows 2008 R2 so I can install these tools via an elevated PowerShell prompt using

  • Import-Module ServerManager
  • Add-WindowsFeature Migration (to install the Server Migration Tools)
  • We then take a DHCP backup (GUI or via an elevated command prompt using  Netsh DHCP server backup “c:DHCPBackup”). Make sure the folder structure exists and is empty. Copy the folder structure of to a file share for safe keeping.  Remember you want to give yourself options.
  • Export the DCHP configuration and content using an elevated powershell prompt:
    • Get-Service DHCPServer (running)
    • Stop-Service DHCPserver
    • Get-Service DHCPServer (stopped)
    • Add-PSSnapin Microsoft.Windows.ServerManager.Migration (load the Server Migration commandlets)
    • Export-SmigServerSetting -featureID DHCP -Group -IPConfig -path C:DHCPservermig  -Verbose (not we are working on a DC so we have no local users and hence no use for the –user switch, see TechNet http://technet.microsoft.com/cs-cz/library/dd379483(WS.10).aspx) and copy the folder structure to a file share for safe keeping. You’ll be asked to provide a password for the exported files. You’ll get feedback due to the –verbose switch about the results. Review these and act accordingly. The below screenshot is an illustration of the output.

  • Unauthorize the DHCP Server using an elevated command prompt: Netsh DHCP delete server bert.big.corp  192.168.2.10
  • If you want you can also uninstall the DHCP server role.
WINS Decommission On The Old Server

We won’t be migrating WINS. We’ll just replicate from the other WINS server(s). But we’re a careful bunch and you have to give yourself multiple options to get out of bad situations, as the saying goes: “one is none, two is one”. Do make sure that you adjust the WINS replication topology if you have more than 2 servers so that all of them get the required registration changes. And just in case you ask, yes I know about the Windows 2008 R2 option to dump WINS and use DNS, but they did not need/want this.

  • Take a WINS backup (GUI) and copy the folder structure of to a file share for safe keeping.
  • Stop the WINS service, disable it and copy the WINS database (mdb) over to a file share for safe keeping as well, just in case.
  • Remove the server as WINS replication partner where it was added. It remains in there when you uninstall WINS (old technology). You can actually leave it in there in this case, as a server with that name will be back.
  • If you want you can already uninstall the WINS feature. This requires a reboot.

That’s it we have now successfully stripped the old server Bert from all its previous roles. We can now change the IP address of the old server to 192.168.2.110 and rename the server to BertOld. I give it a restart and check to see if the DNS record is updated. if not a swift restart of the netlogon service might help. It is now a plain member server in the domain that can be taken out of service at later. This means we are ready to bring the new server Bert to life.

Commissioning The New Server

Commissioning DHCP on the new server
  • Rename the new server to Bert.
  • Restart the server.
  • Make Bert a member of the domain.
  • Restart the server.
  • We now have a plain member server in the domain, named Bert on which we also install the Windows Migration Tools using an elevated PowerShell prompt:
    • Import-Module ServerManager
    • Add-WindowsFeature Migration
  • We copy the folder structure from the file share where we earlier copied the exported older DHCP settings and data, back to server Bert, i.e. C:DHCPservermig
  • We import the IP configuration plus the DCHP configuration & database using the following commandlet: Import-SmigServerSetting -featureid DHCP -Group -IPConfig NIC -SourcePhysicalAddress 00-14-72-7D-B3-C3 -TargetPhysicalAddress 44-2C-2B-AF-67-D2 -Force -path C:DHCPservermig –Verbose
  • Start-Service DHCPServer
  • Via an elevated command prompt we authorize Bert as a DHCP server using “netsh DHCP add server bert.big.corp 192.168.2.10”

Voila, we have Bert running as a DHCP server already!

Commissioning Wins on the new server

We Install WINS and just add the desired replication partners and configure the WINS replication schedule and option to your liking. This is the “cleanest” way. A backup restore is also an option but it could transplant old issues.

Manual trigger Replication and check if you are getting the entries you need (netstat –RR, or via the WINS GUI)

Please note that the browser service is not running by default on Windows 2008(R2) so if you need this for you clients you should set the service to automatic and start it.

Commissioning Active Directory and DNS on the new server
  • Make sure you can contact an existing domain controller before promote Bert to become a domain controller, this can be assured by having the first DNS server in your IP Configuration point to an operational DNS Server in the domain.
  • Promote Bert to a domain controller. By default the DNS option is enabled. Take a look at http://technet.microsoft.com/cs-cz/library/dd379526(WS.10).aspx that will walk you through the wizard.
  • Check the health of you Active Directory, replication and name resolution once more and keep a close eye on it the following days. You should be good to go.

Conclusion

Voila, a new Bert server is born. Not too bad and a happy customer. Rinse and repeat for every domain controller you want to replace.  Do make sure you check out the documentation on the Server Migration Tool and its options. Also, if you are running Windows 2008(R2) and got there by upgrading from Windows 2000/2003 consider moving from FRS to DFRS for SYSVOL replication. As mentioned is a previous blog post of mine (https://blog.workinghardinit.work/2011/01/06/dcdiag-exe-problem-on-windows-2008r2-verifyenterprisereferences-indicates-problem-missing-expected-value-points-to-knowledge-base-article-q312862/) there is an absolute brilliant step by step guide to get the move from FRS to DFRS completed without a problem in a series by the storage team at Microsoft . You can find the first of a 5 part blog series over here http://blogs.technet.com/b/filecab/archive/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process.aspx. I prefer to have this done before I introduce the new domain controllers. This helps me with my neurotic urge to have a SYSVOL folder named SYVOL and not SYSVOL-DFRS clip_image002[1]

Shameless Plug For Mastering Hyper-V Deployment By Aidan Finn

In October 2010 Aidan Finn (MVP) his book “Mastering Hyper-V Deployment” was released and in November three copies of this book landed on my desk. I bought them (pre order) via Amazon. Nope I did not get them as a gift or anything. Why Three? Well that’s the number of people I wanted to get up to speed about Hyper-V and virtualization management and operations in a Microsoft environment.

His book takes you along a journey through a Hyper-V project that will teach you about virtualization in all it’s aspects. It also touches on many supporting technologies and products such as System Center Virtual Machine Manager 2008 R2, System Center Essentials 2010, Data Protection manager 2010 and System Center Operations Manager 2007 R2. No one book can be the only source of knowledge and understanding, but using this book as a start for both new and experienced IT Pros to learn about virtualization with Hyper-V will give you the best possible start. Consider it going to an Ivy league college on a scholarship paid for by Aidan’s experience and hard work. The subsidized tuition fee is the price of the book.

We feel a bit sorry that Aidan only got one copy so we made a group picture of the gang of three on the desk of our newest team member. He got a copy of the book together with 4 recycled PC’s and a TechNet subscription to build a lab.

If you know people who want or need to learn about Hyper-V, you’d do well to make sure they get this book and have them set up a lab to play with the technologies. Those efforts will pay off big time when they implement their solutions in the wild. If Ireland is doomed it won’t be because of smart & hardworking Irish IT professionals like Aidan. You see when you design, build and support IT solutions that your customers depend on 24/7 you can not hide behind false promises, you can’t fake away from the fact when “stuff” doesn’t work or hide behind vast amounts of papers & documents void of any substance. Nope, you are responsible for everything and anything you build. Aidan backed and supported by some very knowledgeable colleagues has made that burden a bit lighter for you to bear with this book. Aidan’s blog lives here: http://www.aidanfinn.com/