Microsoft iSCSI Software Target 3.3 for Windows Server 2008 R2 available for public download

As TechNet subscribers, we had access to Windows Storage Server 2008 with Microsoft iSCSI Software Target 3.2  (also see Jose Barreto’s blog on this here). That was sweet but for one little issue. This SKU cannot be a Hyper-V Host. In order not to lose a physical host in the lab you could edit the MSI installer from the Windows Storage Server 2008 install media where you would delete the SKU check. Problem solved but not very legal so nobody ever did that.  You can install Windows Storage Server in a VM for the lab I know but that becoming very SkyNet like … Virtual servers providing virtual storage for virtual servers … and while a good option to have I like to have a hardware host.

Bring Windows Storage Server 2008 R2 along and Microsoft decided that we could have the iSCSI Software Target 3.3 software without constraints, except that you needed a TechNet/MSDN subscription, to install on W2K8R2. This is the one I’m running in my labs at the moment installed on a Physical Windows Server 2008 R2 Enterprise edition that also is a Hyper-V host. This provides all my iSCSI storage to both physical and virtual clusters. I used it to test MelioFS with FileScaler recently with a 2 node virtual cluster.

Today, Jose Barreto blogged about the public release of iSCSI Software Target 3.3 for Windows Server 2008 R2. This is very good news as now everyone has access to an iSCSI target for labs, testing, POCs, and even production. Thank you, Microsoft. Now with some luck, we could get some SMI-S support for it with SCVMM2012? Please?

If you need some help, Jose Barreto has a bunch of blog posts on configuring the iSCSI target, so I suggest you check out his site. As an added benefit, Microsoft iSCSI Software Target 3.3 setup & configuration is scriptable using PowerShell.

System Center Virtual Machine Manager 2008 R2 SP1 & 2012 Béta Available

Good news, today March 2nd, 2011 System Center Virtual Machine Manager 2008 R2 SP1 went RTM. I’ll update this short post with the download link when it becomes available ==> UPDATE: download it here from Microsoft (Evaluation) or from your TechNet subscription or licensing site. Seems like we got all host & guest updates to W2K8R2SP1 done exactly on time to get this one installed and have a state of the art, up to date infrastructure. Today the 2012 Beta version became available for download (here) and the documentation site went life (here). Things are moving in the system center space. Busy times ahead! Yet 2 more  VMs to test with I the lab … and than Denali is coming.

Key Value Pair Exchange WMI Component Property GuestIntrinsicExchangeItems & Assumptions

Now that Windows 2008 R2 SP1 is being deployed some scripts to check whether the Integration Components (IC) in Hyper-V VM guests are upgraded came back on the radar screen. Host are being upgraded and thus the clients need upgraded IC as well. Now to check this for hundreds or thousands of guest we need some automation. PowerShell comes in handy for this and some neat scripts can be found around the internet. The most concise PowerShell code to do this, that I know of, is the one Peter Noorderijk (great Dutch IT Pro)  uses in his  PowerShell function Get-IntegrationServicesVersion on his blog How to check the version of the Integration Components.  As he provided this script just when I needed one I used it. This worked fine until I ran into an issue with it on some clusters. On two test clusters and two production clusters, it did the job as expected. On one test cluster and one production cluster, we ran into the situation where the output seemed wrong. The screenshot below is an example of this.

The red arrows indicate wrong data for the VMname and ICVersion. What happened here?  Well, when we read out the  GuestIntrinsicExchangeItems property from the WMI object Msvm_KvpExchangeComponent we get back XML. That XML needs to be parsed to display it for human consumption. The function depends on fixed positions containing the correct data. I’ve marked the relevant portions with a red arrow above, they come from$vmkvp[0] en $vmkvp[14] in the script below.

function Get-Integ.rationServicesVersion ($HVhost = $(throw “HVHost required”))
    {
    $kvps = Get-WmiObject -Namespace rootvirtualization -ComputerName $HVHost -Query “Select GuestIntrinsicExchangeItems From Msvm_KvpExchangeComponent”
        foreach ($kvp in $kvps)
         {
         $vmkvp = $Kvp.GuestIntrinsicExchangeItems
         $vmkvp | select-object @{Label=”VMHost”;Expression={$hvhost}},
                                @{Label=”VMName”;Expression={($vmkvp[0]).instance.property[1].value}},        
                                @{Label=”ICVersion”;Expression={($vmkvp[14]).instance.property[1].value}} -first 1
         }    
    }
  
foreach ($hvhost in get-content servers.txt) {Get-IntegrationServicesVersion $hvhost}

And indeed, when we dump the XML for two of the affected servers out to text files you can see the order is indeed different so counting on the exact location in an array is what tripped us up here.

Should this ever happen? Am I making a scripting mistake somewhere? Running a check with a VBScript that parses the XML  using XDOM (just in case my PowerShell skills are the cause of this) confirms the order is different but that the key pairs match up and are correct

D:SysAdminPowerShellScripts>cscript.exe test.vbs

Microsoft (R) Windows Script Host Version 5.8

Copyright (C) Microsoft Corporation. All rights reserved.

Guest OS information for server01

CSDVersion : Service Pack 1

FullyQualifiedDomainName : server01.lab.test

IntegrationServicesVersion : 6.1.7601.17514

NetworkAddressIPv4 : 10.10.100.118

NetworkAddressIPv6 : fe80::a177:729:8840:250%9

OSBuildNumber : 7601

OSEditionId : 7

OSMajorVersion : 6

OSMinorVersion : 1

OSName : Windows Server 2008 R2 Standard

OSPlatformId : 2

OSVersion : 6.1.7601

ProcessorArchitecture : 9

ProductType : 3

RDPAddressIPv4 : 10.10.100.118

RDPAddressIPv6 : fe80::a177:729:8840:250%9

ServicePackMajor : 1

ServicePackMinor : 0

SuiteMask : 272

D:SysAdminPowerShellScripts>cscript.exe test.vbs

Microsoft (R) Windows Script Host Version 5.8

Copyright (C) Microsoft Corporation. All rights reserved.

Guest OS information for server13

FullyQualifiedDomainName : server13.lab.test

OSName : Windows Server 2008 R2 Standard

OSVersion : 6.1.7601

CSDVersion : Service Pack 1

OSMajorVersion : 6

OSMinorVersion : 1

OSBuildNumber : 7601

OSPlatformId : 2

ServicePackMajor : 1

ServicePackMinor : 0

SuiteMask : 272

ProductType : 3

OSEditionId : 7

ProcessorArchitecture : 9

IntegrationServicesVersion : 6.1.7601.17514

NetworkAddressIPv4 : 10.10.100.112

NetworkAddressIPv6 : fe80::c18b:e3f2:7f05:31e4%12

RDPAddressIPv4 : 10.10.100.112

RDPAddressIPv6 : fe80::c18b:e3f2:7f05:31e4%12

When I look at where that data lives in the registry on those servers it all looks exactly the same, neatly ordered buy the RegEdit GUI:

So when getting that data from the Key-Value Pair Exchange WMI component with the property GuestIntrinsicExchangeItems you get a bunch of XML. That has to be parsed to be displayed in a readable fashion. The problem we are seeing is due to the fact that the items in the XML file are not in the same order. Peter’s function assumes it is. However this does not happen to be the case for most virtual machines, the majority is in the expected order. I don’t know why that is or if this is supposed to happen but it doesn’t seem to cause any harm. All is fully functional and operational in Hyper-V Manager, SCVMM 2008R2 … etc. Perhaps an MVP or Microsoft guru can shed some light on this. It seems like a bug waiting to happen if a developer of Hyper-V management software makes the same assumption. Of is this never suppose to happen and do I need to worry? I don’t know Smile I reinstalled the IC on the guests that have a different ordering and live migrated them, but that didn’t change anything

Anyway if you want to make sure you get the correct output we’ll need another approach that doesn’t make assumptions. You can roll your own and get the output customized to your needs but you need to parse the XML using a filter.  An example of this is listed below.

# Filter for parsing XML data
filter Import-CimXml 
{    
# Create new XML object from input  
$CimXml = [Xml]$_    
$CimObj = New-Object -TypeName System.Object     

# Iterate over the data and pull out just the value name and data for each entry   
foreach ($CimProperty in $CimXml.SelectNodes("/INSTANCE/PROPERTY[@NAME='Name']"))      
 {          
     $CimObj | Add-Member -MemberType NoteProperty -Name $CimProperty.NAME -Value $CimProperty.VALUE       
 }  
    
 foreach ($CimProperty in $CimXml.SelectNodes("/INSTANCE/PROPERTY[@NAME='Data']"))     
   
 {         
     $CimObj | Add-Member -MemberType NoteProperty -Name $CimProperty.NAME -Value $CimProperty.VALUE      
 }    


# you send the output from the property to the filter via a pipe
$KvpComponentVMGuest.GuestIntrinsicExchangeItems $vmkvp |Import-CimXml 

Maarten Wijsman (a fellow blogger at http://www.hyper-v.nu like Peter)  has a nice example script here that also uses a filter Import-CimXML. Do note that there are variants on this filter depending on what output you desire that explains the difference between the filters.

Technical Projects, Planning, Skills, Motivation & Psychopaths

When planning a technical project complexity adds up very fast. Take a virtualization project for example; a lot more things than just the hypervisor installation are coming into play. You’ll need to assess a lot of needs and desires about SANs (snapshots, redundancy, replication, FC, iSCSI, FCoE), network (VLAN, 1/10 Gbps Ethernet, redundancy), disaster recovery/business continuity, hypervisors and there capabilities, management of it all and security. That is a lot of stakes and agendas to take into consideration. And then you haven’t even talked to the business managers, the application owners, and developers. Now, this isn’t limited to virtualization, but this is just a nice example of how so many stakes come together in one project.

One of the major mistakes, that is made again and again even up until this day in the second decade in the 21st century is the fact that entire important or even critical IT systems are being put into place with a plan that can be paraphrased as follows “We’ll just set it up and sort of see how it evolves and just wing it from there”. I have been forced to do this quite often. This creates many problems some of which I will address below.

The single worst problem is that you create a vacuum. That can be storage space, bandwidth, ample resources for a huge amount of virtual machines or a mixture of all this. The results however are always the same and is one of two possibilities. Either they really don’t want and need it so it will never be used. You can also achieve this by keeping it hidden so they can’t use it. The other option is the most natural one. In nature, there is a thing called a “horror vacui”. That means that a vacuum unless protected cannot exist, it has to be filled. Empty LUNSs with data, hypervisor hosts with guests, networks with bandwidth, and backup capacity with even more terabytes. You might think the second option is better than the first one as at least the infrastructure is getting used. Unfortunately, the reality is that this is creating a very expensive mess to run, support & troubleshoot. The legacy this creates is not a valuable inheritance but a bank-breaking, efficiency, and effectiveness ruining debt. Stop doing that right now, you are killing your business. You see technology debt is about more than just old hardware and software. It’s about what you build with it or what grows organically with it. Is that a fertile land that sustains the business or a cancer that is killing it?

The way to prevent this is planning done by competent, involved people with experience and context. No plan is perfect, but a plan gives you a framework to achieve the desired result. Even great people make mistakes but they have the skills and attitude to fix them or work around them.

What are some other problems? Wasting money. Take for example a completely oversized server farm. That thing will consume so much money over a three year period in energy and idle capacity that the amount would be sufficient to replace it with new right sized hardware (more bang for the buck, better energy efficiencies in three years) I don’t know about you but those are very disconcerting numbers.

You can also be wasting money and time. And those who know me I loath wasting time. What if the SAN solution you bought doesn’t perform as planned or isn’t the right fit? There goes 500.000 € or you find yourself in the CEO office explaining why you need an extra 400.000 € to get what is really needed. Oh-oh! Do you have money and time to do it all over again or will you be living with that expensive mistake until the current solution is end of life? Do you have to wait until the CFO and CEO have recovered enough from the shock to allow a new attempt? Or perhaps you bought a SAN solution that is enough to run NASA’s workload and you’ve invested 4.000.000 € in a rather expensive data room heater.

Getting a virtualization project wrong can wreak havoc on a business and create a sizable financial hemorrhage. You can say that that’s not your problem but I beg to differ. If the project goes south that means you’ll have to find another job. The IT world where I live is rather small so you might even have to switch to another field as you’ll be forever known as the guy that sunk company X with his little “plan”.

The reverse, being rewarded for your hard work and success is not a given. In the end, they pay you for getting the job done so results are expected, and to Joe Average manager all ICT is a PC with a software packet to install. So for all you eager beavers who think that with this kind of responsibility and risk management comes big reward when you get it right, I suggest you think again. I have witnessed quite the opposite personally. Even when you’re running multiple enterprise SAN’s, networks, infrastructures like SQL Server, Exchange clusters, Hyper-V clusters, geo clusters, load balancers and providing 2nd and 3rd line support for those and taking 24/7 responsibility for the environment the only thing some managers care about is why the PC they never ordered with the software they never ordered can’t be installed tomorrow. “What kind of a chicken shit outfit are you running here” is what they’ll think when you can’t do that. They’ve read the glossy brochure that IT is a commodity and they expect it cheap and always on, much like electricity. In the end some (incompetent) managers act like ungrateful psychopaths. They’ll just abuse you less when you get it right. Don’t expect anything else. Often it’s the ones that are not capable to integrate things they can’t do or don’t understand into their business. They can not value anything that’s beyond their comprehension so they’ll never recognize it. To them, people are, for all practical purposes, resources that are identical, “Full Time Equivalents”. So don’t buy into the hype that there is a skills shortage from that lot and they can’t fill job openings. The volume in which they often waste talent and flush motivation down the drain is shockingly high and indicates that there is no shortage at all or that they can’t recognize skills when they find it and they’ll hire anyone. Surely they didn’t make a mistake so it must be a skills shortage. So you still want to be some hotshot technical architect? Or does a job that only produces open opinions and optional advice on paper sound more attractive. Per hour worked you’ll earn more, run less risk, and have a lot less stress. My advice? Don’t switch fields if you enjoy what you’re doing, switch jobs. The best career advice I ever got was “don’t work with or for assholes”.

Well if you don’t agree with your bosses and you dare go against them you’re surely playing with your job, you could get fired! So? Does living in fear of being fired make good employees? Does not being strong and confident enough to tell your managers they are doing certain things totally wrong or that they are mistaken make for good advisors? The worst thing a boss can have are a bunch of “yes men” around him or her. That boss should be smarter than that. It doesn’t work. Having trust in the abilities and loyalty of your employees does not mean you need to agree on everything. As a boss you’ll make the final decisions, yes, but you’d better listen very carefully to your advisors and staff or you might as well have hired some monkeys. You can train them to say yes all the time, all it takes are some bananas. As an employee, don’t let yourself be treated like a monkey and if they fire you for throwing the banana back, good for you!

So you’d better love technology and building solutions because that means you are intrinsically motivated to go the extra miles. When you are, select a small group of people with the same attitude. You’ll be able to drag the devil himself out of hell with such a team at a very  low cost. Whatever you, do don’t think you can externally motivate or coerce people into achieving this. Charles “Chargin’ Charlie” Beckwith knew that all along when he said “I’d rather go down the river with seven studs than with a hundred shitheads”. And guess what, he wasn’t taught this in some course, by getting a title or by being told this by a manager. He learned it himself by working with the best. These people will keep learning and growing on their own. They don’t need to be told what to do, how to train, what to use, they don’t need nannies & micromanagement. They need an end state and they’ll get it for you. Frankly, that kind of skillset and ability scares the shit out of some bosses as they micromanage actions & items instead of doing their jobs. You can’t use force, treats or authority to make people achievers. In the end, you can cut a diamond, but you cannot create it. Trust me. Putting that amount of pressure on someone that isn’t a diamond only turns them into a heap of crushed remains of what used to be a human being or FTE in your typical HR speak.

“Mate you’re not a conformist” my friend said … you’d better believe I’m not.