Dell generation 13 servers & Intel E5 v3 18 core CPUs are upon us in world where per core licensing is reality

Posted on September 8, 2014 by workinghardinit

As I watched the Intel E5 v3 launch event & DELL releasing their next generation servers to the public to purchase there is a clear opportunity for hardware renewal next year. I’m contemplating on what the new Intel E5 v3 18 core processors

and the great DELL generation 13 PowerEdge Servers mean for the Hyper-V and SQL server environments under my care.

For the Hyper-V clusters I’m in heaven. At least for now as Windows is still licensed per socket at the time of writing. vNext has me worried a bit, thinking about what would happen if that changes to core based licensing to. Especially with SQL Server virtualization. I do hope that if MSFT ever goes for per core licensing for the OS they might consider giving us a break for dedicated SQL Server Hyper-V clusters.

For per core licensing with SQL Server Enterprise we need to run the numbers and be smart in how we approach this. Especially since you need Software Assurance to be able to have mobility & failover / high availability. All this at a time you’re told significant cost cutting has to happen all over the board.

So what does this mean? The demise of SQL Server in the Enterprise like some suggest. Nope. The direct competitors of SQL Server in that arena are even more expensive. The alternatives to SQL are just that, in certain scenarios you don’t need SQL (Server) or you can make due with SQL Server Express. But what about all the cases where you do really need it? You’ll just have to finance the cost of SQL Server. If that’s not possible the business case justifying the tool is no longer there, which is valid. As the saying goes, if you can’t afford it, you don’t need it. A bit harsh yes, I realize, but this is not a life saving medicine we’re talking about but a business tool. There might be another reason your SQL Server licensing has become unaffordable. You might be wasting money due to how SQL Server is deployed and used in your environment. To make sure you don’t overpay you need to evaluate if SQL Server consolidation is what is really needed to save the budget.

Now please realize that consolidation doesn’t mean stupidly under provisioning hardware & servers to make budget work out. That’s just plain silly. For some more information on this, please read Virtualizing Intensive Workloads on Hyper-V, Can It Be Done

So what is smart consolidation (not all specific to SQL Server by the way):

You have to avoid physical SQL Server sprawl with a vengeance.
You need to consolidate SQL Servers aggressively.
Virtualize on a dedicated SQL Server Hyper-V cluster if possible
Favor scale out over scale up in the Hyper-V scenario to keep node costs reasonable and allow for affordable expansion.
Use 2 socket servers and replace the hardware faster to keep the number of needed cores down.
- This allows to leverage modern commodity, high performance storage, networking and compute where you can in order to optimize workloads & minimize costs.
- It helps save on power consumption & cooling
- More nodes with lesser cores (scale out approach) reduces VM density per node but also keep the cost of adding a node (with SQL Server per core licensing, or when it comes to that for the OS as well), which is your scaling block with a fixed cost under control. It’s all about balance and it isn’t as easy as it seems.
Play the same game with storage. This can be a harder sale to make internally. Traditionally people hang on to storage longer due to the high CAPEX. I have said it before, storage vendors have to deliver more & better. Even the challengers & hyper converged systems are still too expensive to really get into a short renewal cycle for most organizations.

Be smart about it. A great DBA can make a difference here and some hard core performance tuning is what can save a serious amount of money. If on top of that you have some good storage & network skills around you can achieve a lot. Next to the fact that you’ll have to spend serious money for serious workloads the ugly truth is that consolidation requires you find your peak loads and scale for those with a vengeance. Look, maxing out one server on which one SQL Server is running isn’t that bad. But what if 3 SQL Servers running a peak performance spread over a 3 node Hyper-V cluster dedicated to SQL Server VMs might kill performance all over!

The good news is I have solid ideas,visions, plans and options to optimize both the on premise & cloud of part of networking, storage & compute. Remember that there is no one size fits all. Execution follows strategy. The potential for very performant, cost effective & capable solutions are right there. I cannot give you a custom solution for your needs in a blog post. One danger with fast release cycles is that it requires yearly OPEX end if they cannot guarantee it the shift in design to solutions with less longevity could become problematic if they can’t come up with the money. Cutting some of the “fat” means you will not be able to handle longer periods of budget drought very well. There is no free lunch.

So measure twice & cut once or things can go wrong very fast and become even more expensive.

You might think this sounds a bit pessimistic. No this is an opportunity, especially for a Hyper-V MVP who happens to be a MCDBA Winking smile . The IT skills shortage is only growing bigger all over the planet, so not too much worries there, I won’t have to collect empty bottles for a living yet. The only so called “draw back” here could be that the environments I take care of have been virtualized and optimized to a high extend already. The reward for being good is sometimes not being able to improve things in orders of magnitude. Bad organizations living in a dream world, the ones without a solid grasp of the realities of functional IT in practice, might find that disappointing. Yes the “perception is reality” crowd. Fortunately the good ones will be happy to be in the best possible shape and they’ll invest money to keep it that way. Interesting times ahead.

Manually Merging Hyper-V Checkpoints

Posted on August 29, 2014 by workinghardinit

A Last-ditch Effort

First of all you need to realize this might not work. It’s a last-ditch effort. There is a reason why you have backups (with tested restores) and why you should monitor your environment for things that are not as they should be. Early intervention can pay off.

Also, see blog post on a couple of more preferred actions.

If you have lost checkpoints, you have basically lost data and corruption/data inconsistencies are very much a possibility and reality. If the files have been copied and information about what file is the parent the dates/timestamps are what you have to go by. You might not know for sure if you have them all.

Setting up the demo

For demo purposes, we take a test VM and ad files to indicate what checkpoint we’re at.

We start with ORGINAL.TXT on the desktop and we create a checkpoint, which we rename accordingly.

We add a file called CHECK01.TXT and we create a checkpoint, which we rename accordingly.

We add a file called CHECK02.TXT and we create a checkpoint, which we rename accordingly.

We add a file called NOW.TXT no more checkpoints are taken.

The file names represent the content you’d see disappear if you applied the checkpoint and we have reflected this in the name for the checkpoints.

As we want to merge all the snapshots and end up with a usable VHDX we’ll work back from the most recent differencing disk until all is merged. As you can see this is a straightforward situation and I hope you’ll never be running having to deal with a vast collection of subtrees.

Finding out what are the parents of avhdx files

In this demo, it’s pretty obvious what snapshots exist and what avhdx files they represent. We’ve even shown you the single tree visualized in Hyper-V Manager. In reality, bad things have happened and you don’t see this information anymore. So you might have to find out yourself. This is done via inspect disk in Hyper-V manager. If you’re confused about what the parent is of (a)vhdx files this tool will help you find out or show you what the most recent one was.

Sometimes the original files have been renamed or moved and that it will show you the last known valid parent.

Manually Merging the checkpoints

Remember to make a copy of all files as a backup! Also, make sure you have enough free disk space … you need working space! You might need another shot at this. As we want to merge all the snapshots and end up with a usable vhdx we’ll work back from the most recent differencing disk until all is merged in the oldest one which is the vhdx. You can look at the last modified timestamps to find out the correct order in which to work. The most recent avhdx is the one used in the virtual machine configuration file and locate the information for the virtual hard disk.

The configuration file’s avhdx is the one containing the “NOW” running state of the VM.

Note: You might find some information that you need to rename the extension avhdx to vhdx (or avhd to vhd). The reason for this was that in Windows 2008 Hyper-V Manager did not show avhd files in the Edit virtual disk wizard. You can still do this and it will still works, but you do not need to. Ever since Windows Server 2008 R2 avhd (and with since Windows Server 2012 avhdx) files do show up in Hyper-V Managers Disk edit.

For some insights as to why the order is important read this blog by Ben Armstrong What happens when a snapshot is being merged? [Hyper-V]

WARNING: If you did not start with the most recent one and work your way down, which is the easiest and least confusing way all is not lost. But you will have to reconnect the first more recent (a)vhdx to one to its new parent. This is needed as by merging a snapshot out of order more recent one will have lost it’s will have lost its original parent.

Here’s how to do this: Select reconnect. This is the page you’ll get if you’d start edit disk wizard as all other options are unavailable due to the missing parent.

The wizard will tell you what used to be the parent and allow you to select a new one. Make sure to tick the check box for Ignore ID mismatch or the reconnect will fail as you’re previous out of order merge has created a new a(vhdx). If your in this pickle by renaming (a)vhdx files or after a copy this isn’t needed by the way.

Follow the wizard after that and when your done you can launch the edit disk wizard again and perform a merge. It’s paramount that you do not mix up orders when doing so that you reconnect to the parent this or you’ll end up in a right mess. There are many permutations, keep it simple!. Do it in order. If you start having multiple checkpoint trees/subtrees things can get confusing very fast.

You might also have to reconnect if the checkpoints have lost their connection the what they know to be their last parent for other reasons. In that case, you do this and when that’s done, you merge. Rinse and repeat. The below walk-through assumes you have no reconnects to be done. If so it will tell you like in the example above.

Walk through:

Open the Edit Disk Wizard

Select the most recent avhdx & click “Next”

We choose to merge the avhdx

In our case into its parent disk

Verify the options are correct and click “Finish”

Let the wizard complete

That’s it. You’ve merged the most recent snapshot into it’s parent. That means that you have not lost the most recent state of the virtual machine as when it was running before you shut it down. This can be verified by mounting the now most recent avhdx and looking at the desktop for my user profile. You can see the NOW.txt text file is there!

OK, dismount the avhdx and now it’s rinse and repeat.

You do this over and over again until your merge the last avhdx into the vhdx.

Than you have the vhdx you will use to create a new virtual machine.

Make sure you get the generation right.

Assign memory

Connect to the appropriate virtual switch or not if you’re not ready to do this yet

Use your vhdx disk that’s the remaining result of your merging efforts

When you boot that virtual machine you’ll see that all the text files are there. It’s as if you’ve deleted the checkpoints in the GUI and retained “NOW” in the vhdx.

Last but not least, you can use PowerShell or even DiskPart for this but I found that most people in this pickle value a GUI. Use what you feel most comfortable with.

Thanks for reading and hope this helps someone. Do remember “big boy” rules apply. This is not safe, easy or obvious in each and every situation so you are responsible for everything you do in your environment. If your in to deep, way over your head, etc. call in some expert help.

3 Ways To Deal With Lingering Hyper-V Checkpoints Formerly Known as Snapshots

Posted on August 25, 2014 by workinghardinit

Lingering or phantom Hyper-V checkpoints or snapshots

Once in a while the merging of checkpoints, previously known as snapshots, in Hyper-V goes south. An example of this is when checkpoints are not cleaned up and the most recent avhdx or multiple of these remains in use as active virtual disk/still even as you don’t see them anymore as existing in the Hyper-V Manager UI for example. When that happens you can try looking at the situation via PowerShell to see if that show the same situation. Whatever the cause, once in while I come across virtual machines that have one or more avhdx (or avdh) active that aren’t supposed to be there anymore. In that case you have to do some manual housekeeping.

Now please, do not that in Windows Server 2012(R2) Hyper-V replica is using checkpoints and since Windows Server 2012 R2 backups also rely on this. Just because you see a snapshot you didn’t create intentionally, don’t automatically think they’re all phantoms. They might exits temporarily for good reason Winking smile . We’re talking about dealing with real lingering checkpoints.

Housekeeping

Housekeeping comes in a couple of variants form simply dusting of to industrial cleaning. Beware of the fact that the latter should never be a considered a routine operation. It’s not a normal situation. It’s a last ditch resort and perhaps you want to call support to make sure that you didn’t miss anything else.

Basically you have tree options. In order of the easiest & safest to do first these are:

Create a new checkpoint and delete it. Often that process will take care of merging the other (older) lingering avhd/avhdx files as well. This is the easiest way to deal with it and it’s as safe as it gets. Hyper-V cleans up for you, you just had to give it a kick start so to speak.
Shut down the VM and create a new checkpoint. Export that newly created checkpoint. Yes you can do that. This will create a nicely exported virtual machine that only has the relevant vhd/vhdx files and no more checkpoints (avhd/avhdx). Do note that this vhd/vhdx is dynamically expanding one. If that is not to your liking you’ll need to convert it to fixed. But other than that you can dump the old VM (don’t delete everything yet) and replace it by importing the one you just exported. For added security you could first copy the files for save guarding before you attempt this.
Do manual mergers. This is a more risky process & prone to mistakes. So please do this only on a copy of the files. That way you’ll give Microsoft Support Services a fighting change if things don’t work out or you make a mistake. Also note that in this case you get one or more final VHDX files which you’ll use to create a new virtual machine with to boot from. It’s very hands on.

So that’s the preferred order of things to try/do in regards to safety. The 3rd option, is the last resort. Don’t do it before you’ve tried options 1 and 2. And as said above, if you do need to go for option 3, do it on copies.If you’re unsure on how to proceed with any of this, get an expert involved.

There’s actually another option which is very save but not native to Hyper-V. In the running virtual machine which current state you want to preserve do a V2V using Disk2vhd v2.01. Easy and sort of idiot proof if such a thing exists.

In a next blog post I’ll walk you through the procedure for the 3rd option. So if this is your last resort you can have practiced it before you have to use it in anger. Bit please, if needed, and do make sure it’s really needed as discussed above, try 1 first. If that doesn’t do it. Then try option 2. If that also fails try option 3. Do not that for option 2 and 3 you will have to create a new virtual machine with the resulting VHDX, having the required settings documented will help in this case.

I Can’t Afford 10GBps For Hyper-V And Other Lies

Posted on July 23, 2014 by workinghardinit

You’re wrong

There, I said it. Sure you can. Don’t think you need to be a big data center to make this happen. You just need to think and work outside the box a bit and when you’re not a large enterprise, that’s a bit more easy to do. Don’t do it like a big name brand, traditionalist partner would do it (strip & refit the entire structural cabling in the server room, high end gear with big margins everywhere). You’re going for maximum results & value, not sales margins and bonuses.

I would even say you can’t afford to stay on 1Gbps much longer or you’ll be dealing with the fall out of being stuck in the past. Really some of us are already look at > 10Gbps connections to the servers, actually. You need to move from 1Gbps or you’ll be micro managing a way around issues sucking all the fun out of your work with ever diminishing results and rising costs for both you and the business.

Give your Windows Server 2012R2 Hyper-V environment the bandwidth it needs to shine and make the company some money. If all you want to do is to spent as little money as possible I’m not quite sure what your goal is? Either you need it or you don’t. I’m convinced we need it. So we must get it. Do what it takes. Let me show you one way to get what you need.

Sounds great what do I do?

Take heart, be brave and of good courage! Combine it with skills, knowledge & experience to deliver a 10Gbps infrastructure as part of ongoing maintenance & projects. I just have to emphasize that some skills are indeed needed, pure guts alone won’t do it.

First of all you need to realize that you do not need to rip and replace your existing network infrastructure. That’s very hard to get approval for, takes too much time and rapidly becomes very expensive in both dollars and efforts. Also, to be honest, quiet often you don’t have that kind of pull. I for one certainly do not. And if I’d try to do that way it takes way too many meetings, diplomacy, politics, ITIL, ITML & Change Approval Board actions to make it happen. This adds to the cost even more, both in time and money. So leave what you have in place, for this exercise we assume it’s working fine but you can’t afford to have wait for many hours while all host drains in 6 node cluster and you need to drain all of them to add memory. So we have a need (OK you’ll need a better business case than this but don’t make to big a deal of it or you’ll draw unwanted attention) and we’ve taking away the fear factor of fork lift replacing the existing network which is a big risk & cost.

So how do I go about it?

Start out as part of regular upgrades, replacement or new deployments. The money is their for those projects. Make sure to add some networking budget and leverage other projects need to support the networking needs.

Get a starter budget for a POC of some sort, it will get your started to acquire some more essential missing bits.

By reasonably cheap switches of reasonable port count that do all you need. If they’re readily available in a frame work contract, great. You can get it as part of the normal procedures. But if you want to nock another 6% to 8% of the cost order them directly from the vendor. Cut out the middle man.

Buy some gear as part of your normal refresh cycle. Adapt that cycle life time a bit to suit your needs where possible. Funding for operation maintenance & replacement should already be in place right?

Negotiate hard with your vendor. Listen, just like in the storage world, the network world has arrived at a point where they’re not going to be making tons of money just because they are essential. They have lots of competition and it’s only increasing. There are deals to be made and if you chose the right hardware it’s gear that won’t lock you into proprietary cabling, SPF+ modules and such. Or not to much anyway Smile .

Design options and choices

Small but effective

If you’re really on minimal budget just introduce redundant (independent) stand alone 10Gbps switches for the East-West traffic that only runs between the nodes in the data center. CSV, Live Migration, backup. You don’t even need to hook it up to the network for data traffic, you only need to be able to remotely manage it and that’s what they invented Out Off Band (OOB) ports for. See also an old post of mine Introducing 10Gbps With A Dedicated CSV & Live Migration Network (Part 2/4). In the smallest cheapest scenario I use just 2 independent switches. In the other scenario build a 2 node spine and the leaf. In my examples I use DELL network gear. But use whatever works best for your needs and your environment. Just don’t go the “nobody ever got fired for buying XXX” route, that’s fear, not courage! Use cheaper NetGear switches if that fits your needs. Your call, see my recent blog post on this 10Gbps Cheap & Without Risk In Even The Smallest Environments.

Medium sized excellence

First of all a disclaimer: medium sized isn’t a standardized way of measuring businesses and their IT needs. There will be large differences depending on you neck of the woods .

Build your 10Gbps infrastructure the way you want it and aim it to grow to where it might evolve. Keep it simple and shallow. Go wide where you need to. Use the Spine/Leaf design as a basis, even if what you’re building is smaller than what it’s normally used for. Borrow the concept. All 10Gbps traffic, will be moving within that Spine/Leaf setup. Only client server traffic will be going out side of it and it’s a small part of all traffic. This is how you get VM mobility, great network speeds in the server room avoiding the existing core to become a bandwidth bottleneck.

You might even consider doing Infiniband where the cost/Gbps is very attractive and it will serve you well for a long time. But it can be a hard sell as it’s “another technology”.

Don’t panic, you don’t need to buy a bunch of Nexus 7000’s or Force10 Z9000 to do this in your moderately sized server room. In medium sized environment I try to follow the “Spine/Leaf” concept even if it’s not true ECMP/CLOSS, it’s the principle. For the spine choose the switches that fit your size, environment & growth. I’ve used the Force10 S4810 with great success and you can negotiate hard on the price. The reasons I went for the higher priced Force10 S4810 are:

It’s the spine so I need best performance in that layer so that’s where I spend my money.
I wanted VLT, stacking is a big no no here. With VLT I can do firmware upgrades without down time.
It scales out reasonably by leveraging eVLT if ever needed.

For the ToR switches I normally go with PowerConnect 81XX F series or the N40XXF series, which is the current model. These provide great value for money and I can negotiate hard on price here while still getting 10Gbps with the features I need. I don’t need VLT as we do switch independent NIC teaming with Windows. That gives me the best scalability wit DVMQ & vRSS and allows for firmware upgrades without any network down time in the rack. I do sacrifice true redundant LACP within the rack but for the few times I might really need to have that I could go cross racks & still maintain a rack a failure domain as the ToRs are redundant. I avoid stacking, it’s a single point of failure during firmware upgrades and I don’t like that. Sure I can could leverage the rack a domain of failure to work around that but that’s not very practical for ordinary routine maintenance. The N40XXF also give me the DCB capabilities I need for SMB Direct.

Hook it up to the normal core switch of the existing network, for just the client/server.(North/South) traffic. I make sure that any VLANs used for CSV, live migration, can’t even reach that part of the network. Even data traffic (between virtual machines, physical servers) goes East-West within your Spine/Leave and never goes out anyway unless you did something really weird and bad.

As said, you can scale out VLT using eVLT that creates a port channel between 2 VLT domains. That’s nice. So in a medium sized business you’re pretty save in growth. If you grow beyond this, we’ll be talking about a way larger deployment anyway and true ECMP/CLOS and that’s not the scale I’m dealing with where. For most medium sized business or small ones with bigger needs this will do the job. ECMP/CLOS Spine/leaf actually requires layer 3 in the design and as you might have noticed I kind if avoid that. Again, to get to a good solution today instead of a real good solution next year which won’t happen because real good is risky and expensive. Words they don’t like to hear above your pay grade.

The picture below is just for illustration of the concept. Basically I normally have only one VLT domain and have two 10Gbps switches per rack. This gives me racks as failure domains and it allows me to forgo a lot of extra structural cabling work to neatly provide connectivity form the switches to the server racks .

You have a scalable, capable & affordable 10Gbps or better infrastructure that will run any workload in style.. After testing you simply start new deployments in the Spine/Leaf and slowly mover over existing workloads. If you do all this as part of upgrades it won’t cause any downtime due to the network being renewed. Just by upgrading or replacing current workloads.

The layer 3 core in the picture above is the uplink to your existing network and you don’t touch that. Just let if run until there nothing left in there and you can clean it up or take it out. Easy transition. The core can be left in place or replaces when needed due to age or capabilities.

To keep things extra affordable

While today the issues with (structural) 10Gbps copper CAT6A and NICs/Switches seem solved, when I started doing 10Gbps fibre cabling of Copper Twinax Direct Attach was the only way to go. 10GBaseT wasn’t an option yet and I still love the flexibility of fibre, it consumes less space and weighs less then CAT6A. Fibre also fits easily in existing cable infrastructure. Less hassle. But CAT6A will work fine today, no worries.

If you decide to do fibre, buy OM3, you can get decent, affordable cabling on line. Order it as consumable supplies.

Spend some time on the internet and find the SFP+ that works with your switches to save a significant amount of money. Yup some vendor switches work with compatible non OEM branded SPF+ modules. Order them as consumable supplies, but buy some first to TEST! Save money but do it smart, don’t be silly.

For patch cabling 10Gbps Copper Twinax Direct Attach works great for short ranges and isn’t expensive, but the length is limited and they get thicker & more sturdy and thus unwieldy by length. It does have it’s place and I use them where appropriate.

Isn’t this dangerous?

Nope. Technology wise is perfectly sound and nothing new. Project wise it delivers results, fast, effective and without breaking the bank. Functionally you now have all the bandwidth you need to stop worrying and micromanaging stuff to work around those pesky bandwidth issues and focus on better ways of doing things. You’ve given yourself options & possibilities. Yay!

Perhaps the approach to achieve this isn’t very conventional. I disagree. Look, anyone who’s been running projects & delivering results knows the world isn’t that black and white. We’ve been doing 10Gbps for 4 years now this way and with (repeated) great success while others have to wait for the 1Gbps structural cabling to be replaced some day in the future … probably by 10Gbps copper in a 100Gbps world by the time it happens. You have to get the job done. Do you want results, improvements, progress and success or just avoid risk and cover your ass? Well then, choose & just make it happen. Remember the business demands everything at the speed of light, delivered yesterday at no cost with 99.999% uptime. So this approach is what they want, albeit perhaps not what they say.

Working Hard In IT

My view on IT from the trenches

Tag Archives: Hyper-V