Hot Iron, Cold Steel & Cables Are Still Paramount In The Era Of The Cloud

Cloud, virtualized, on premise, hosted … the people in the field offices need to connect to them and as such hardware is not dead yet Winking smile. Commodities don’t mean obsolete or “in the cloud” only.

clip_image002

Some nice DELL PowerConnect 5548P switches. We’ve been using this line of switches (since the 53XX series) for many years now and with great success for in the datacenter (before we switched to 10Gbps) and campus/client access. They’ve never let us down at a price/value point that make the economies of using them to good to ignore.

Once in a while, we’re out in the field making sure the people can access their apps, services, servers in the cloud, the data center or at a hosting provider. Meaning we get to play with some hardware and we all enjoy that still Smile. Whilst at work at several sites I’m once again confronted with commodities being treated like specialties with the following results:

  • Overly expensive
  • Very little value & capabilities (under delivery)
  • Slow delivery
  • Churning

To avoid wasting you money or allowing it to be wasted you need to use common sense. If you use advisors get a consigliore, not a racketeer.

1Gbps to the desktop and get some extra ports

I’ve talked about getting affordable 10Gbps without compromising capabilities before so here I’ll look at the access/campus side of the story. I still find many organizations rolling out 100Mbps to the desktop for cost reasons and counting ports in orders of one. Two things to keep in mind. Buy 1Gbps and buy some extra.

Buying vast quantities of something you don’t use but does it power is not a good idea. But being a complete scrooge and not having some extra ports is ridiculous. I have seen many thousands of € wasted in meetings about 10 to 40 switch ports too few in new building projects that have > 5000 outlets. The only real saving I see in in electricity used, if that is a major concern where you are at. Organizations spend tens of thousands of  € discussing something that would be fixed by spending a few thousand which would give extra benefits on top. That’s churning people. Creating work and billable hours by overinflating issues & crying wolf to justify the expenditure that’s supposedly needed to stave of disaster.

On top of that when you do ask those architects to do some modern designs like SMB Direct  & DCB they freak out & repeat the above ritual. Chances are you’ll spend 20.000 to 30.00 euro on a 6 month study that says it can’t be done because of cost & the probability the sky will fall on your head, leaving you empty handed an poorer. You should have taken the money and just done it. Their scams defer responsibilities to untraceable entities, lines the pocket of consulting houses and, as no one is going to take responsibility to stop this madness, it just goes on forever whilst on paper everything is done by the book and compliancy to the rules is achieved.  Until the day some joker, frustrated at the lack of a few ports, attached a cheapo 8 ports switch to the outlet, creates a loop and brings down the buildings network affecting many thousands. Because the design didn’t handle that to well … been there, seen it.

I also disagree with the practice of dropping in 100Mbps unless you have really good reasons. Structural cabling is being put in at Cat6A specifications nowadays and CAT5E has been put I for many years. 1Gbps is not a luxury if you do lots of data transfers within an office and have image intensive needs (more and more that is all of us with video, images, all in high res). Google fiber is coming to residential homes … guess what that could mean to services that can be delivered … Heaven forbid you buy 100Mbps because those fancy overpriced VOIP phones only do 100Mbps & you can’t afford the replace them.

With QoS for VOIP and other use cases some extra bandwidth comes in handy as well for. Also don’t forget software installations & automated rollouts of desktops & laptops. Last but not least it helps deal with crappy network behavior of way to many software packets.

On the number of ports and the price per port. We buy the most minimal support on switches possible. They hardly ever die on you and if something goes bad it’s a port perhaps, and even that is rare. So don’t waste money on support contracts. Buy some extra ports. For one you need some wiggle room and you have spare capacity to deal with port or even switch failures. If you need 400 ports, by 10*48 port switches. You have spare capacity and can even afford to lose a switch. If one really fails you most have a “lifetime warranty”. You finance 1Gbps to the desktop by dumping support you won’t need, buying value commodity switches and avoiding the racketeers mentioned above. If you need a network engineer, hire one, a good one.

Than inevitably the cry comes: “you’ll saturate the uplinks”! Not a big issue for the small office (+/- 60 people) setup we did recently but what about a bit larger environments? Todays commodity switches had dual fiber uplink port,10Gbps capable, for a redundant lag. If you build a star design and not a cascade to a more capable core/top switch & you’re golden. It’s also great future proofing as we use access switches for a long time, over 7 years is not an exception, so give yourself some wiggle room.

Cost you say? Again, forgo the expensive market leaders and you’ll get better value for less money that get the job done very well. Cables, even OM3 fiber, is affordable compared to the labor, construction and maintenance of a  > 1000 employee building. Put in enough cabling to allow for 21st century network traffic and make sure working on it is easy. Good principles used at the wrong place in the wrong way are no good to anyone except for the ones making money of this scam.

Some Insights Into How Windows 2012 R2 Hyper-V Backups Work

How Windows Server 2012 R2 backups differ from Windows Server 2012 and earlier

You’ll remember our previous blog about an error when backing up a virtual machine on Windows Server 2012 R2, throwing this error:

Dealing With Event ID 10103 “The virtual machine ‘VM001’ cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID DCFE14D3-7E08-845F-9CEE-21E0605817DC)” In Windows Server 2012 R2

The fix was easy enough, adding a virtual SCSI controller to the virtual machine. But why does it need that now?

Well, this all has to do with the changed way Windows Server 2012 R2 backups work. Before Windows Server 20012 R2 the VSS provider created a VSS snapshot inside the guest virtual machine. That snapshot was exposed to the host, to create a volume snapshot for backup purposes. Right after the volume snapshot has been taken this VSS snapshot inside the guest virtual machine needed to be reverted. The backups then run against that volume snapshot and is consistent thanks to both host & guest VSS capabilities.

For an overview of VSS based backup process in general take a peak at Overview of Processing a Backup Under VSS

Now it is the “Hyper-V Integration Services Shadow Copy Provider” that is being used. When the the host initiates a volume snapshot (Microsoft or hardware VSS provider) the host VSS writer goes in to freeze. This process leverages the Hyper-V Integration Services Shadow Copy Provider  to create the virtual machine checkpoint. After that the volume/LUN/CSV snapshot is taken. When that is done the host VSS writes goes into thaw and the virtual machine checkpoint is deleted. After that the backup runs against the Volume snapshot and at the end that is also deleted. You can follow this process quite nicely in the GUI of your Hyper-V host, you SAN (if you use a Hardware VSS provider).

Dear storage vendors: a great, reliable, fast VSS Hardware Provider is paramount to success in a Microsoft environment. You need to get this absolutely right and out of the door before spending any more time and money on achieving yet more IOPS. Keep scalability in mind when doing this.

Dear backup software vendors: think about the scalability when designing your products. If we have 200 or 500 or a thousand VMs … can we leverage CSV based backups to protect every VM on the LUN or do we need to snap the LUN for every VM backed up? Choice there is good for both data protection schemes and scalability.

At this stage the hardware VSS snapshot is being taken …

image_thumb3

Contrary to common belief this means that the backup will indeed application consistent to the time of the checkpoint as the CSV snapshot being taken is of a consistent checkpoint. It’s the delta in the active avhdx that is only crash consistent, like any running VM by the way. Now pay attention to the screenshot below. The two red arrows are indicating to ntfs source events, two volumes seem to be exposed to the next free drive letters. E: and F: here as C: is the virtual machine OS and D: the DVD.

image_thumb5

Look at the detail. Indeed two. Well it the previous screenshot we only saw one in the CSV path but there are two avhdx files indeed.

image_thumb[1]

Exposing a snapshot on the SAN to a server actually shows us this much better … look here at the avhdx with the GUID and one with “AutoRecovery” in the name. So that makes for two nfts events … and as the backup needs to do this life it requires a vSCSI controller to be present in the virtual machine … and vIDE controller can’t do this.

image_thumb[3]

Anyway, enough under the hood detective work for now, In VEEAM that stage looks like this:

image_thumb7

And on the Compellent it looks like this. The screenshots are from different backups at different times so don’t get confused about the time stamps here. It’s just as illustration of what you can expect to see.

image_thumb12

Now when the CSV snapshot has been taken the virtual machine checkpoint is removed. At that time the backup runs against the CSV snapshot. In our case (hardware VSS provider) this is a snapshot on the SAN that gets exposed in a view and mapped to the off host backup proxy VEEAM server. On the DELL Compellent it looks like this.

image_thumb16

This takes a while to o…but after a while the backup will kick off. Do not that the checkpoint has merged and is no longer visible at this time.

image_thumb18

Once the backup is complete, the mapping is removed, the view deleted and the snapshot expired. So your SAN is left as the backup found it.

There you go. I hope this helped clarify certain things on how Hyper-V guest backups work in Windows 2012 R2. So your backups are still application consistent, just not when you’re running Linux or DOS or NT4.0 as there is no support / VSS for that. However they are based on a  consistent virtual machine snapshot which explains why Hyper-V backups can protect Linux guests very adequately!

Dealing With Event ID 10103 “The virtual machine ‘VM001’ cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID DCFE14D3-7E08-845F-9CEE-21E0605817DC)” In Windows Server 2012 R2

I was doing backups of a Windows 2012 R2 Hype-V cluster recently and it runs only Windows Server 2012 R2 virtual machines. It’s a small but very modern and up to date cluster Smile.

Using VEEAM as backup software I have high expectations and VEEAM did deliver. All went well except for one virtual machine.

image

VEEAM states "Processing Error. Guest processing skipped (check guest OS VSS state and integration components version)". Well all  virtual machines  are W2K12R2 as are the cluster host and all IC components are up to date and backup (volume checkpoint) is enabled.

image

I dove into the Hyper-V log and sure enough I found following event:

The virtual machine ‘VM001’ cannot be hot backed up since it has no SCSI controllers attached. Please add one or more SCSI controllers to the virtual machine before performing a backup. (Virtual machine ID DCFE14D3-7E08-845F-9CEE-21E0605817DC).

As it turns out in in Windows Server 2012 R2 the VM requires a SCSI controller for the backup to function. It doesn’t need to have any storage attached. It just needs one to be there (default). So the fix is easy, just add one.

image

image

Click “Apply” and “OK”. You can now start the virtual machine and that’s it. Once we fixed that it was a squeaky clean backup run.

But why does it need to be there?

Well when we monitor the event logs inside a virtual machine we are backing up we see that during the backup process, very briefly a VHDX get’s mounted inside the guest.

image

To answer this question we need to dive into how Windows Server 2012 R2 backups work as that is different from how it used to be. You can read about that over here when it’s published.