Windows 2008 R2 SP1 Publicly Available & Update on Error 0x00f0818

Yesterday, late last night in Europe, Windows Server 2008 R2 SP1 became available to the general public. That means it will be pushed via Windows Updates or that you can download it manually here Windows 7 and Windows Server 2008 R2 Service Pack 1 (KB976932)

I’ve been busy doing deployments in the lab but also in production at several locations. The speed of a deployment depends on the host and what’s running on that host. I’ve seen anything between 22 minutes and 65 minutes. For a walk through of an install see my previous blog post on this Upgrading a Hyper-V R2 Cluster to Windows 2008 R2 SP1

I’ve also noticed an increased amount of hits for my blog on error 0x00f0818 when installing the Windows 2008 R2 SP1 Beta. The same solution holds true for RTM. More info on that is available at Windows 2008 R2 SP1 Beta Install Gone Wrong: Service Pack Installation failed with error code 0x800f0818

I had one place where dozens of VMs had this issue so in order to progress quickly we scripted the replacement of the C:WindowsservicingPackages folder content with know good files in bulk. Read the above mentioned article to learn about the security settings you’ll need to address in any automated solution (use takeown F c:WindowsServicingPackages /D y /R & cacls c:WindowsServicingPackages /E /T /C /G “UserName”:F  to take care of business). That saved us the time to check on each affected server individually what packages were involved. This works but test this before using it in production and don’t forget to return the security settings to the default when you’re done. In this case it was a rather large lab environment.

Hyper-V Component Architecture Poster Updated With Windows 2008 R2 SP1 Features

You probably are all familiar with the component posters that Microsoft put out for Several technologies. You have the Exchange 2010 (Exchange Server 2010 Architecture Poster), Windows 2008 R2 (Windows Server 2008 R2 Feature Components Poster) & Hyper-V (Hyper-V Component Architecture).

I have them hanging on the walls around our offices and these make a great tool to discuss the technology and where what is being done or accomplished. Or as a reference when you need to explain some features.  I suggest you download them and have them plotted in a print shop.

Microsoft recently put out a new version of the Hyper-V Component Architecture poster with the Windows 2008 R2 Service Pack 1 features (Windows Server 2008 R2 Hyper-V Component Architecture (with Service Pack 1)). It’s nice to see that they update these so quickly.  We’re plotting this on A0 as we speak.

Upgrading a Hyper-V R2 Cluster to Windows 2008 R2 SP1

For all you people waiting to roll out Windows 2008 R2 SP1 to your Hyper-V cluster here’s a quick screenshot rich run through of the process. Some people claim you need to shut down the cluster services and shut down the guests but this is not the case.  You can do a rolling upgrade and your guests can stay on line on the other nodes, just use live migration to get them there. Now I do recommend to upgrade all the nodes tot SP1 as soon as possible and not staying a mixed Windows 2008 R2 / Windows 2008 R2 SP1 situation in your cluster. But this mixed situation makes upgrades of the nodes in the cluster possible without any down time for the guests (if you have live migration), which is the aim of having a high availability cluster.

Walk Through

Live migrate all the guests from the node you wish to upgrade to SP1. Make sure the host is fully patched and disable any antivirus services if you are running any. I always reboot the node before a major upgrade to make sure we have the server in a clean state with  no lingering reboots waiting  or processes can cause issues.

Navigate to the service pack 1 file for Windows 2008 R2, it’s called windows6.1-KB976932-X64.exe and start it up:

SP1

 

You’ll have to accept the license terms:

SP1-2

 

And then the preparation process starts:

SP1-3

 

It is now ready to start the upgrade and yes we want it to reboot automatically when needed:

SP1-4

The upgrade process takes a while (about 17 minutes on my lab servers):

SP1-6SP1-6(2)SP1-6(3)

 

When it’s done it will reboot and bring you back to the logon screen. Multiple reboots might be needed to complete the upgrade process depending on what’s running on your server. In this case we are dealing with dedicated Hyper-V nodes.

View when connected to the console

image

View when connected via RDP

image

 

After logging on you are greeted with this window:

SP1-7

 

And yes this is indeed the case

SP1-8

Reboot included the entire process took about 22 to 23 minutes. In the setup event log you’ll find these messages:

  • Initiating changes for package KB976932. Current state is Absent. Target state is Installed. Client id: SP Coordinater Engine.
  • Package KB976932 was successfully changed to the Installed state.

Note: if an extra reboot is required you’ll see an extra entry in between these stating: A reboot is necessary before package KB976932 can be changed to the Installed state.

When you have a cluster with nodes running both W2K8R2 TM and W2K8R2 SP1, mixed situation so to speak,  you’ll see the following notification in the cluster events:

SP1-9

 

You can live migrate the guest from the next node to the node already upgraded to SP1 and than repeat the process. You keep doing this until all your nodes are upgraded.

SP1-10

As a final recommendation I would suggest waiting until you get the SCVMM2008R2 SP1 bits is you use this product before you upgrade you clusters especially when using this with SCOM2007R2 PRO Tips. Otherwise you don’t need to wait just realize that until you have SP1 for SCVMM2008 R2 you won’t be able to use the new functionality for Hyper-V. In production I would not recommend using the RC1 for this.

Please do not forget to update your guests with the new SP1 version of the Hyper-V Integration Components. This is needed to be able to use the new features like Dynamic Memory & Remote FX. The Windows 2008 R2 RTM version of the Integration Components is  6.1.7600.16385:

image

 

You can do this using Hyper-V Manager through selecting “Insert Integration Services Setup Disk”  and running the setup, this will require a reboot.

image

 

Click to start the upgrade process:

image

 

It will ask to remove the previous version:

image

 

Work in progress:

image

 

Done and asking for a reboot:

image

 

SCVMM2008R2 can also be used, here you shut down the guest before updating the virtual guest services as it’s called in SCVMM2008R2. It can be annoying that the nomenclature differs. The good thing here is that you can upgrade multiple guest using VMM2008R2. Hans Vredevoort did a blog post on this here: http://www.hyper-v.nu/blogs/hans/?tag=windows-server-2008-r2-sp1.  After the upgrade you can see that the version of the Integration Components  for Windows 2088 R2 SP1 is  6.1.7601.17514:

image

Why I’m No Fan Of Virtual Tape Libraries

After implementing a couple of SAN’s with backup solutions I have come to dislike Virtual Tape Libraries. This is definitely technology that, for us, has never delivered the promised benefits. To add insult to injury it is overly expensive and only good to practice hardware babysitting. When discussing this I’ve been told that I want things to cheap and that I should have more FTE to handle all the work. That’s swell but the business and the people with the budgets are telling me exactly the opposite. That explains why in the brochures it’s all about reduced cost with empty usage of acronyms like CAPEX and OPEX. But when that doesn’t really materialize the message to the IT Pros is to get more personnel and cash. In the best case (compared to calling you a whining kid) they‘ll offer to replace the current solution with the latest of the greatest that has, wonder oh wonder, reduced CAPEX & OPEX. Fool me once shame on you, fool me twice, shame on me.

So one thing I’m not planning to buy, ever again, is a Virtual Tape Library (VLS). Those things have a shelf life of about 2 years max. After that they turn into auto disintegrating pieces of crap you’ll spend babysitting for the rest of the serviceable live. This means regularly upgrading the firmware to get your LUN(s) back, if you get them back that is. This means convincing tech support to come up with a better solution than restarting from scratch when they acknowledge that their OS never cleans up its own log files and thus one day just kicks the bucket. Luckily they did go the extra mile on that one after we insisted and got a workaround without losing al backups. Babysitting also means that replacing the battery kits of all shelves becomes a new hobby. You become so good at it that you have better and faster way of doing it than the junior engineers they send who happily exclaim “so this is what it looks like”. The latter is not a confidence builder. The disks fail at a rate of 1 to 2 per week until you replaced almost all of them. Those things need to be brought down to fix just about anything. That means shutting down the disk shelf’s as well and cutting the power, not just a reboot so yes you need to be in the data center.

There is no RAID 6, no hot spares (global or otherwise). The disks cost an arm and a leg and have specialized hardware to make sure all runs fine and well. But in they are plain7.200 rpm cheap 500 GB SATA disks that cost way too much. The need for special firmware I can understand but the high cost I cannot. The amount of money you pay in support costs and in licensing the storage volume is more than enough to make a decent profit. Swapping disks and battery kits isn’t hard and we do it ourselves as waiting for a support engineer takes more time. We have spares at hand. We buy those when we by the solution. We’ve grown wise that way. We buy a couple units of all failure prone items at the outset of any storage project. Having only RAID 5 means that one disk failure puts you virtual tapes at a very high risk so you need to replace it as soon as possible. Once they shipped us a wrong disk, our VTL went down the drain due to incorrect firmware on disk. They demanded to know how it got in there. Well Mr. Vendor, you put it in yourself a as a replacement for a failed disk. In the first year it often happened they didn’t have more than 1 spare disk to ship. If anyone else has a VLS in your area you’re bound to hit that limit have to wait longer for parts. They must have upped the spare parts budget to have some more on hand just for us as we now get a steady supply in Confused smile.

When you look at the complete picture the cost of storage per GB on a VLS is a much as on 1st tier SAN storage. That one doesn’t fly well. At least the SAN has decent redundancy and is luckily a 100 fold more robust and reliable. Why buy a VLS when you can have a premier tier SAN for the same cost, the VLS functionality? No sir that also never lived up to its promises. It has come to the point that the VTL, due to the underlying issues with the device, are more error prone than our Physical Tape Library. That’s just sad. Anyway, we never got the benefits we expected form those VTL. For disk based backup I don’t want a Virtual Tape Library System anymore. It just isn’t worth the cost and hassle.

Look you can buy 100 TB of SATA storage, put it in couple of super micro disk bays, add a 10Gbps Ethernet to your backup network and you’re good to go. Hey that even gives you RAID 6, the ability to add hot spares etc. You buy some spare disks, controllers, NICS, and perhaps even just build two of these setups. That would give you redundant backup to two times 100 TB for under 60.000 €. A VLS with 100TB, support and licensing will put you back the 5 fold of that. Extending that capacity costs an arm and a leg and you’re babysitting it anyway so why bleed money while doing that?

Does this sound crazy? Is this blasphemy? The dirty little secret they don’t like you to know is that’s how cloud players are doing it. First tier storage is always top notch, but if you talk about backing up several hundreds of terabytes of data, the backup solutions by the big vendors are prohibitively expensive. This industry looks a lot like a mafia racketeering business. Well if you don’t buy it you’ll get into trouble, you’ll lose your data. You won’t’ be able to handle it otherwise. Accidents do happen. The guys selling it even dress like mobsters in their suits. But won’t you miss out on cool things like deduplication? If your backup software supports it you can still have that. The licensing cost for this isn’t that bad a deal when compared to VLS storage costs. And do realize instead of 2*100 TB you could make due to 2* 25 TB Open-mouthed smile Hey that price even dropped even more.

When it comes to provisioning storage our strategy is to buy as much of your storage needs during the acquisition phase. That’s the only time deals can be made. The amount of discounts you’ll get will make you wonder what the markup in this market actually. It must be huge. And storage can’t cost as much as you think to build as they would like us to believe. Last time the storage sales guy even told us they were not making any money on the deal. Amazing how companies giving away their products have very good profit margins, highly paid employees with sales bonuses and 40.000 € cars If one vendor or reseller ever tells you that things will get cheaper with time, they are lying. They are actually saying their profit margins will increase during the life cycle of you storage solution. Look, all 1st tier storage is going to be expensive. The best you can hope for is to get a good quality product that performs as promised and doesn’t let you down. We’ve been fortunate in that respect to our SAN solutions but when it comes to backup solutions I’m not pleased with the state of that industry. Backups are extremely important live savers, but for some reason the technology and products remain very buggy, error prone, labor intensive and become very expensive when the data volumes and backup requirements rise.