Key Take Aways From MMS2013

Introduction

I’ve parked myself at McCarran International Airport in Las Vegas awaiting the start of my long haul home to Europe. The new terminal is inspiring me to share as I reflect on the past week and on what’s happening at work.

ICT in the 21st Century

A lot is going on and moving in ICT and even more is coming our way. In the Microsoft sphere we got the official heads up at MMS2013 that new features would be publicly discussed at TechEd 2013 (USA or Europe). So you might want to attend that one. I for one think that’s great. We need that information to verify we’re still are on the right track and fine tune our course. Especially in those areas where we can get quick wins with sometimes significant cost savings & benefits.  I could start telling you about all the great sessions and products at MMS2013 whilst quacking like a duck “cloud, cloud, …, cloud, cloud, cloud, … cloud”. But I will not. You can watch it all here.  I will reflect on the key take away.

Cheaper & Faster

Cheaper AND faster are the new mantra or’ “fast is the new cheap”. Cheaper makes everyone happy especially when quality remains high. Faster is sometimes a bit more of a challenge to sell. “New features, already?”  you say. Yes. The nature of our economies and industry is being transformed by the cloud and commoditization. It brings a lot of benefits, especially in a high speed, low drag world.

Fast is actually faster. For many years now any strategy & execution plan that took more than a couple of years was doomed. You get bypassed and your big investments will never live up to their potential. So, apart from the necessary larger and more long term investments, we evolve more and more towards a perpetual improvements & rapid adoption model. Innovation and the subsequent commoditization of it is pushing this. That’s not bad. By making constant smaller (easier to fund) investments that deliver fast results we get to a more adaptable, agile environment for lesser costs. It’s not that all long term, large scale projects are going away but the ratio is shifting. In smart countries this is already being done for building hospitals and other infrastructure that evolves fast. It’s not unique to ICT. Massive projects taking too long and too much funding lead to out of date solutions at the time of delivery at huge costs. Use this approach where needed but forget about it for the other projects. Cloud will be an important tool in all this, not the goal.

A Word of Warning

Fast and cheap shouldn’t translate into mediocre crap at dump pricing that will bite us. It should also keep in mind the ecosystem and don’t act like a shock & awe offensive leaving everything in it’s track in disarray. It needs to fit into a plan with clear goals an knowing where it fits in and helps.  It’s about balance. That’s the art. Knowing what, where, when and with/for who to do it. Not easy. Now let’s hope some of my managers read this blog. It might help them. As the question beckons an answer: who is it that will lead us in this new era? Well not one single person, far from it. It’s a team effort and to lead a team takes competence and some character.

It takes competence and personality

Competence and personality, combined with  applying both these (skills and  drive) diligently in a sustained fashion. That requires a lot of effort, even when no one is watching you, or perhaps better stated, especially then. Do what needs to be done where and when needed. Not because it could get you promoted or more money. That’s the character part. That’s what drives us to learn by participating in our ICT communities, presenting, attending conferences and networking. But also in those hours spend reading, studying and working in the lab alone or with a buddy. That’s what will make us able to handle the though and bad situations you’ll encounter and overcome them. It’s your resourcefulness that will make you seek and find opportunity in adverse conditions. People like the team members amongst whom I have the distinct pleasure of working. You can’t find such synergy if it’s only about personal gain and getting ahead. There is both a broad and deep skill set needed by all involved and doesn’t come easy nor can it be bought. It has to be acquired through work and experience. The transformation of the ICT landscape is uncharted domain for all but a few of us so it’s going to ask a lot of effort, often outside of our comfort zone.

Sure there are cynics who laugh at this and can’t imagine why someone would do all that without personal and immediate reward. Those are the ones we don’t need and who won’t be there at crunch time. Only after the facts they seek the spotlight to poach the glory if things went well or to condemn those that failed whilst trying. Well, the last so called leader who did that doesn’t work with us anymore. Enough said.

Disk to Disk Backup Solution with Windows Server 2012 & Commodity DELL Hardware – Part II

As I blogged in a previous post we’ve been building a Disk2Disk based backup solution with commodity hardware as all the appliances on the market are either to small in capacity for our needs, ridiculously expensive or sometimes just suck or a combination of the above (Virtual Library Systems or Virtual Tape Libraries come to mind, one of my biggest technology mistakes ever, at least the ones I had and in my humble opinion Disappointed smile) .

Here’s a logical drawing of what we’re talking about. We are using just two backup media agent building blocks (server + storage)  in our setup for now so we can scale out.

image

Now in future post I hope to be discussing storage spaces & Windows deduplication thrown into the mix.

So what do we get?

Not to shabby …  > 1TB/Hour

image

To great …

image

In close up you are seeing just 2 Windows 2012 Hyper-V cluster nodes, each being backed up over a native LBFO team of 2*1Gbps NIC ports to one Windows Server 2012 Backup Media Agent with a 10Gbps pipe. Look at the max throughput we got  …

image

Sure this is under optimal conditions, but guess what? When doing backup from multiple hosts to dual backup media servers or more we’re getting very fast backups at very low cost compared to some other solutions out there. This is our backup beast Smile. More bandwidth needed at the backup media server? It has dual port 10Gbps that can be teamed and/or leverage SMB 3.0 multichannel. High volume hosts can use 10Gbps at the source as well.

Lessons learned

  • The Windows 2012 networking improvements rock. Upgrade and benefit from it! We’re seeing great results thanks to Multichannel leveraging RSS and in box NIC teaming (LBFO).
  • A couple of 1Gbps NICS teamed on Windows Server 2012 work really well. Don’t worry about not having 10Gbps on all your hosts.
  • Having 10Gbps on your backup media hosts (target) is great as you’ll be pushing a lot of data to them from multiple (source) hosts.
  • Make sure your backup software supports enough streams before it keels over under the load you’re pushing through. More streams means more concurrent files (read VHDs/VMs) and thus more throughput and allows multichannel to shine over RSS capable NICs.
  • Find the sweet sport for number of disks per node and total IOPS versus the throughput you can send to the backup media agents. 4 Nodes of 50TB might be better than 2 nodes of a 100TB. If you can, experiment a bit to find your optimal backup block size.
  • Isolate your backup network traffic from data traffic either physically or by other means (QOS) and don’t route it all over the place to end up where it needs to be.
  • We’re doing this using Dell PowerConnect 5424 (end of life) /5524 switches … no need for the real  high end very expensive gear to do this. The 10Gbps switch, well yes that’s always high end at the moment.
  • Use JBODS with SAS/Storage spaces & you’ll be fine. Select them carefully for performance. You can use bays like the MD3X00 if you want to replicates the backups somewhere otherwise MD12x0 will do or any other decent JBOD => even cheaper. You can also mix, some building blocks that can replicate & other on Storage Spaces /JBOS. Mix and match with different backup needs means you have flexibility. Note that at TechEd Europe (June 2012), in a session by DELL, they mentioned the need for a firmware update with the MD1200 to optimize performance with Storage Spaces.

It’s all about the money in a smart way!

As I said before, you will not get fired for:

  • Increasing backup throughput at least 4 fold (without dedupe)
  • Increasing backup capacity 3.5 fold (without deduplication)
  • Doing the above for 20% of systems that are replaced & new offerings with specialized appliances (even at hilarious discount rates). That’s CAPEX reduction.
  • This helps pay for the primary storage, DRC site & extra SAN for data replication in case of disaster
  • Make backups faster, more reliable & reduce OPEX (The difference for us is huge and important)
  • Putting an affordable scale up & scale out Disk2Disk backup solution into place to the business can safely handle future backup loads as very acceptable costs.
  • It’s a modular solution which we like. On top of that it’s about as zero vendor lock in as it gets. You can mix servers, bays, switches. Use what you like best for your needs. Only the bays have to remain the same within an individual “building block”.

Cost reduction is one thing but look at what we get whilst saving money… wow!

What am I missing?  Specialized dedupe. Yes, but we’re  going for the poor mans workaround there. More on that later.  As long as we get enough throughput that doesn’t hurt us. And give the cost we cannot justify it + it’s way to much vendor lock in. If you can’t get backups done without, sure you might need to walk that route. But for now I’m using a different path. Our money is better spend in different places.

Now how to get the same economic improvements from the backup software? Disk capacity licensing sucks. But we need a solution that can handle this throughput & load , is reliable, has great support & product information, get’s support for new release fast after RTM (come on CommVault, get a move on) and is simple to use ==> even more money and time saved.

Spin off huge file server project?

Why is support for new releases in backup software important. Because the lack of it is causing me delays. Delays cost me, time, money & opportunities. I’m really interested to covert our large LUN file servers to Windows Server 2012 Hyper-V virtual machines, which I now can rather smoothly thanks to those big VDHX sizes that are possible now and slash the backup times of those millions of small files to pieces by backing them up per VHDX over this setup. But I’ll have to wait and see when CommVault will support VHDX files and GPT disks in guests because they are not moving as fast as a leading (and high cost) product should. Altaro & Veeam have them beaten solid there and I don’t like to be held back.