Microsoft Management Summit 2012

From the 16th of April until the 20th of April 2012 Microsoft is running animportant conference for anyone who’s involved with systems management in the Microsoft sphere. It is, of cause, the Microsoft Management Summit 2012 (MMS 2012) in Las Vegas (Nevada, USA).

image

This is a conference that is held in very high regard and I’ve heard through the grape vines it’s one of the favorite conferences for Microsoft Employees to attend themselves due to its high quality and focus on the System Center suite. I’ve never had the opportunity to attend before and I would like to go.

It’s very likely that the System Center 2012 Suite of products will be officially launched at MMS 2012 and there will be an abundance of learning opportunities in regards to these. As said above, I’d really love to go and I encourage anyone who can make it to attend. Yes, I know it’s in the United States, so for us non US residents that can mean long and expensive travel and we’ll need a budget to stay in Las Vegas for a week. But it’s the only conference of its kind. There is no MMS Europe and such. Although I have to say that with initiatives like “Best of MMS” TechNet events Microsoft & the community make an effort to deliver content and information to a much larger audience, which is great.

If you’re in the target group for this conference and you’re interested take a look here. They even have cost/benefit sheet to help convince your management Winking smile

Now a lot of you are already be playing with the System Center 2012 Betas and Release Candidates but if you’re not  you might want to get a head start by downloading the System Center 2012 Evaluation Products and perhaps even by joining the Community Evaluation Program for System Center 2012 (Private Cloud) and Configuration Manager 2012.

Download The Next Wave Of System Center Suite Products Release Candidates

As you all probably know by now in 2012 there will be a lot of new tools available for us IT Pros. Windows 8 Server alone holds so many new features and added value that it alone will keep us busy for many years to come. On top of that, the 2012 version of the System Center Suite is being released. I hope most of you have started looking at the betas and the release candidates. If not I encourage you to do so. There are a lot of great improvements that will help us enhance our current setups and operations, not to mention enable us to build a private cloud. For some musings on why you might want to do that, even if you have no public cloud plans yet, I refer you to my blog post The Private Cloud A Profitable Future Proofing Tactic?

To help you get started with testing I include some links to some of the most interesting products.

If you select the “Download All Available System Center 2012 Products” option you’ll get all most current versions of the pre release. To fully test the System Center 2012 Suite products you’ll need Windows Server/Hyper-V but also the current line of System Center Suite (upgrade testing).

The full Windows 2008 R2 SP1 OS is available here http://technet.microsoft.com/evalcenter/dd459137.aspx  and you can find the free Hyper-V Server here http://technet.microsoft.com/evalcenter/dd776191.aspx 

The most relevant current System Center Suite products are available via these links:

To top it all of and get acquainted with “The Cloud” grab the “Deploying and Managing Windows Azure Applications” here http://technet.microsoft.com/evalcenter/hh282846.aspx

There you go. All this should keep you usefully occupied and out of trouble during the end of year holidays and make you an knowledgeable and capable IT Pro, ready for the year to come.

The Private Cloud A Profitable Future Proofing Tactic?

The Current Situation

I’m reading up on the private cloud concept as Microsoft envisions we’ll be able to build ‘m with the suite of System Center 2012 products. The definition of private cloud is something that’s very flexible. But whether we’re talking about the private, hybrid or public cloud there is a point of disagreement between some on the fact that there are people that don’t see self-service (via a portal, with or without an approval process) as a required element to define a *cloud. I have to agree with Aidan Finn on this one. It’s a requirement. I know you could stretch the concept and that you could build  a private cloud to help IT serve it customers but the idea is that customers can and will do it themselves.

The more I look into system center 2012 and it’s advertised ability to provide private clouds the more I like it. Whilst the current generation has some really nice features I have found it lacking in many areas, especially when you start to cross security boundaries and still integrate the various members of the System Center suite. So the advancements there are very welcome. But there is a danger lurking in the shadows of it all. Complexity combined with the amount of products needed. In this business things need to go fast without sacrificing or compromising on any other thing. If you can’t do that, there is an issue. The answer to these issues is not always to go to the public cloud a hundred percent.

While the entire concept might seem very clear us techies (i.e. still lots of confusion to be found) and the entire business crowd is talking about cloud as if it’s a magic potion that will cure all IT related issues (i.e. they are beyond confused, they are lost) there are still a lot of questions. Even when you have the business grasping the concept (which is great) and have an IT team that’s all eager and wiling to implement it (which is fabulous) things are still not that clear on how to start building and/or using one.

In reality some businesses haven’t even stepped into the virtual era yet or only partially at best. Some people are a bit stuck in the past and still want to buy servers and applications with THEIR money that THEY own and is ONLY for them.  Don’t fight that to much The economics of virtualization are so good (not just financially but also in both flexibility & capabilities) that you can sell it to top management rather easily, no matter what. After that approval just sell the business units servers (that are virtual machines), deliver whatever SLA they want to pay for and be done with it. So that problem is easily solved.

But that’s not a cloud yet. Now that I’m thinking of it, perhaps getting businesses to adopt the concept will be the  hardest. You might not think so by reading about private clouds in the media but I have encountered a lot of skepticism and downright hostility towards the concept. No, it’s not just by some weary IT Pros who are afraid to lose their jobs. Sometimes the show stoppers are the business and users that won’t have any of it. They don’t want to order apps or servers on line, they want then delivered for them. I even see this with the younger workforce when the corporate culture is not very helpful. What ‘s up here? Responsibility. People are avoiding it and it shows in their behavior. As long as they want to take responsibility things go well. If not, technical fear masked as “complexity” or issues like “that’s not our job” suddenly appear.

There is more, a lot of people seems at their limit of what they can handle in information overload at every extra effort is too much.  Sometimes it’s because of laziness or perhaps even stupidity? Perhaps it’s a side effect of what Nicolas Carr writes about the: the internet is making us dumber and dumber as a species. But then again, we only have to look at history to learn that, perhaps, we’ve never been that smart. Sure we have achieved amazing things but that doesn’t mean we don’t act incredibly stupid as individuals or as a group. So perhaps things haven’t changed that much. It’s a bit like the “Meet the new boss, same as the old boss” sort of thing. But on the other hand things are often too complex. When things are easy and become an aid in their work people adopt technology fast and happily.

Sometimes the scale of the business is not of that nature that it’s worthwhile top deploy a cloud. The effort and cost versus the use and benefits are totally out of sync.

That’s all nice and well you tell me, but what’s are technologists to advice to their customers?

Fire & Maneuver

The answer is in the sub title. You can’t stand still and do nothing. It will get you killed (business is warfare with gloves on and some other niceties). Now that’s all good to know but how do we keep moving forward and scoring? There will always be obstacles, risks, fears etc. but we can’t get immobilized by them or we come to a standstill, which means falling behind. The answer is quite simple. Keep moving forward.  But how? Do what you need to do. Here’s my approach. Build a private cloud. Use it to optimize IT and to be ready to make use of * clouds at every opportunity. And to put your mind at ease you can do this without spending vast amounts of money that gets wasted. Just provide some scale up and scale out capacity & capability. The capability is free if you do it right. The capacity will cost you some money. But that’s your buffer to keep things moving smoothly. Done right your CAPEX will be less than not doing this. How can this be?

Private Clouds enable Hybrid Clouds

The thing that I like most about the private cloud is that it enables the use of hybrid cloud computing. On the whole and in the long run hybrid clouds might be a transition to public cloud but as I’ve written before, there are scenarios where the hybrid approach will remain. This might not be the case for the majority of businesses but still I foresee a more permanent role for hybrid clouds for a longer time that most trendy publications seem to indicate. I have no crystal ball but if hybrid cloud computing does remain a long term approach to server computing needs we night well see more and better tools to manage this in the years to come. Cloud vendors who enable and facilitate this approach will have a competitive advantage. The only thing you need to keep I mind that private or cloud computing should not bee seen as replacements or alternatives for the public cloud. They don’t have the elasticity, scale and economics of a public cloud. They are however complementary. As such they enable and facilitate the management and consumption of IT services that have to remain on premises for whatever reason.

Selling The Public Cloud

Where private cloud might help businesses who are cloud shy warm up to the concept, I think the hybrid cloud in combination with integrated and easy management will help them make the jump to using public cloud services faster. That’s the reason this concept will get the care and attention of cloud vendors. It’s a stepping stone for the consumption of their core business (cloud computing) that they are selling to businesses.

What’s in it for the business that builds one?

But why would a business I advise buy into this? Well a private cloud (even if used without the self-service component) is Dynamic Systems Initiative (SDI) / Dynamic Data Center on steroids. And as such it delivers efficiency gains and savings right now even if you never go hybrid or public. I’m an avid supported of this concept but it was not easy to achieve for several reasons, one of them being that the technologies used missed some capabilities we need. And guess what, the tools being delivered for the private could can/could fill those voids. By the way, I was in the room at IT Forum 2004 when Bill Gates came to explain the concept and launch that initiative. The demo back then was deploying hundreds of physical PCs. Times have changed indeed! But back to selling the private cloud. Building a private cloud means you’ll be running a topnotch infrastructure ready for anything. Future proofing your designs at no extra cost and with immediate benefits is to good to ignore for any manager/CTO/CIO. The economics are just too good. If you do it for the right reason that is, meaning you can’t serve all your needs in the public cloud as of yet. So go build that private cloud and don’t get discouraged by the fact that it won’t be a definition example of the concept, as long as it delivers real value to the business you’ll be just fine. It doesn’t guarantee your business survival but it will not be for your lack of trying. The inertia some businesses in a very competitive world are displaying makes them look like rabbits trapped in the beams of the car lights. Not to mention government administrations. We no longer seem to have the stability or rather slowness of change needed to function effectively. Perhaps this has always been the case. I don’t know. We’ve never before in history had such a long period of peace & prosperity for such a broad section of the population. So how to maintain this long term is new challenge by itself.

Danger Ahead!

As mentioned above, if there is one thing that can ruin this party it’s is complexity. I’m more convinced of this than ever before. I’ve been talking to some really smart people in the industry over the weekend and everyone seems to agree on that one. So if I can offer some advice to any provider of tools to build a private cloud.  Minimize complexity and the amount of tools needed to get it set up and working. Make sure that if you need multiple building blocks and tools the integration of them is top notch and second to none. Provide clear guidance and make sure it is really as easy to set up, maintain and adapt as it should be. If not businesses are going to get a bloody nose and IT Pros will choose other solutions to get things done.

System Center Virtual Machine Manager 2008 R2 Error 12711 & The cluster group could not be found (0×1395)

The Issues

I recently had to go and fix some issues with a couple of virtual machines in SCVMM 2008 R2. There was one that failed to live migrate with following error:

Error (12711)
VMM cannot complete the WMI operation on server HopelessVm.test.lab because of
error: [MSCluster_ResourceGroup.Name=" df43bf60-7216-47ed-9560-7561d24c7dc8"] The cluster group could not be found.

(The cluster group could not be found (0×1395))
 
Recommended Action
Resolve the issue and then try the operation again

Other than that it looked fine and could be managed with SCVMM 2008 R2. Another one was totally wrecked it seemed. It was in a failed state after an attempted live migration. You couldn’t do anything with it anymore. Repair was “available” but every option there failed so basically that was the end of the game with that VM. Both issues can be resolved with the approach I’ll describe below.

The Cause

After some investigation the cause of this was the fact that this virtual machine had been removed from the failover cluster as a resource was exported & imported using Hyper-V manager on one of the cluster nodes. It was then added back to the failover cluster again to make them high available. All this was done without removing it from SCVMM 2008 R2. By the way, as mentioned above in “The Issues” this can get even worse than just failing live migrations. The same scenario can lead to virtual machines going into a failed state that you can’t repair (retry or undo fail) or ignore and basically you’re stuck at that point. You can’t even stop, start, shutdown the virtual machine anymore, not one single operation works in SCVMM while in the failover cluster GUI and in hyper-v manager everything is fully operational. This is important to note, as the services are fully on line and functional. It’s just in SCVMM that you’re in trouble.

Why did they do it this way? They did it to move the VM to a new CSV. The fact that you delete the VM files when deleting a VM with SCVmm2008R2 made them use Hyper-V manager instead. Now this approach (whatever you think of it) can work but then you need to delete the VM in SCVMM2008R2 after exporting the virtual machine AND before proceeding with the import and making the virtual machine highly available.

People get creative in how to achieve things due to inconsistencies, differences in functionality between Hyper-V Manger and SCVMM 2008R2 (in the latter especially the lack of complete control over naming, files & folders, export/migration behavior) as well as the needs of the failover cluster can lead to some confusing scenarios.

The Supported Fix

Now the easy way to fix this is to export the virtual machine again and delete it in SCVMM 2008 R2. That will remove the virtual machine object from SCVMM, the failover cluster en Virtual Machine Manager. However this virtual machine was so large (50Gb + 750 GB data disk) that there was no room for an export to be made. Secondly an export of such a large VM takes a considerable time and it has to be off line for this operation. This is annoying as SCVMM might be uncooperative at this point, the virtual machine is online en performing it’s duties for the business. So this presented us with a bit of a problem. Stopping the virtual machine, Exporting it using Hyper-V Manager will cause it to go missing in SCVMM 2012 and then you can delete it, importing the virtual machine again and adding it to the failover cluster causes down time.

The Root Cause

Why does this happen? Well when you import a virtual machine into a failover cluster is creates a new unique ID for the virtual machine Resource Group . This happens always. Choosing to reuse an existing ID during import in Hyper-V Manager has nothing to do with this. But VMM uses ID/names to identify a VM, independent of the cluster. So when you did not remove the VM from SCVMM before adding the VM back to the cluster you get a different cluster group ID in the cluster than you have in SCVMM. They both have the same name but there is a disconnect leading to the issues described above.

By the way exporting & importing a VM without first removing the virtual machine from the failover cluster leads to some issues in the Failover cluster so don’t do that either Smile

The “No Down Time” Fix

This is not the first time we need to dive in to the SCVMM database to fix issues. One of my main beef about SCVMM other than inconsistency with the other tools and its lack of control & options in some scenarios is the fact that it doesn’t have enough self-maintenance intelligence & functionality. This leads to the workaround above which are slow and rather annoying or consist of messing around in the SCVMM database, which isn’t exactly supported. Mind you Microsoft has published some T-SQL to clean up such issues themselves. See You cannot delete a missing VM in SCVMM 2008 or in SCVMM 2008 R2 and RemoveMissingVMs. See also my blog SCVMM 2008 R2 Phantom VM guests after Blue Screen post on this subject.

The usual tricks of the trade like refreshing the virtual machine configuration in the failover cluster GUI don’t work here. Neither does the solution to this error described Migrating a System Center Virtual Machine Manager 2008 VM from one cluster to another fails with error 12711. The error is the same but not the cause.

# Add the VMM cmdlets
Add-PSSnapin microsoft.systemcenter.virtualmachinemanager

# Connect to the VMM server
Get-VMMServer –ComputerName MySCVMMServer.test.lab

# Grab the problematic VM and put it into the object $vm
$vm = Get-VM –name “HopelessVM”

#Force a refresh
refresh-vm -force  $vm

In the end we have to fix the mismatch between the VMResourceGroupID in failover cluster and SCVMM by editing the database.

First you navigate to the registry key HKEY_LOCAL_MACHINEClusterGroups on one the cluster nodes, do a find for the problematic VM’s name and grab the name of its key, this is the VMResourceGroupID the cluster knows and works with? So now we have the correct VMResourceGroupID: 0f8cabe4-f773-4ae4-b431-ada5a3c9926c

clip_image002

Now you connect to the SCVMM database and run following query to find the VMResourceGroupID that SCVMM thinks that VM has and that it uses causing the issues

SELECT  VMResourceGroupID  FROM tbl_WLC_VMInstance WHERE ComputerName = 'hopelessVM.test.lab'
GO 

The results:

VMResourceGroupID

————————————————–

df43bf60-7216-47ed-9560-7561d24c7dc8

(1 row(s) affected)

The trick than is to simply update that value to the one you just got from the registry by running:

UPDATE tbl_WLC_VMInstance SET VMResourceGroupID = '0f8cabe4-f773-4ae4-b431-ada5a3c9926c' WHERE VMResourceGroupID = 'df43bf60-7216-47ed-9560-7561d24c7dc8'
GO 

Than you need some patience & refresh the GUI a few times. Things will turn back to normal, but in between you might seem some “missing” statuses appear for your problematic VM. These go away fast however. If not you can always use the Microsoft provided script to remove missing VM’s as mentioned above in RemoveMissingVMs.

Warning

What I described above is something you can do to fix these issues fast and effectively when needed. But I’m not telling you this is the way to go, let alone that this is supported. Make sure you have backups of your VMs, Hosts, SCVMM database etc. It only takes one mistake or misinterpretation to royally shoot yourself in your foot Winking smile. It hurts like hell; recovery is long and seldom complete. On top of that it might generate a vacancy in your company whilst you’re escorted out of the building. Be careful out there.