Consumerization of IT Discussions at BriForum 2011 London

At BriForum 2001 In London I also attended a lot of talks on BYOD and the consumerization of IT. The connection with BriForum is where VDI and user virtualization fit in to facilitate this. Now talk about this subject has been going on for about 5 years now and has been brought up at many TechEd sessions for example.

If that concept works, I say bring it on. Really. I mean it holds so many promises of a better world for everyone involved that we’d be nuts not to try it. I like the concept, but will it work, is it possible? If so where, when and to what extent. Anyway it’s all good stuff until that seems to require lawyers and contracts. Ouch! We’re not too good at dealing with that and I have to say that from my experience contracts are legal documents and are very useful in that arena but it won’t stop people from doing what they can where and when they can. They don’t think about using Hotmail or drop box of being “illegal” or against policy. They just use it. Look at any other corporate security and fair use policy. They are full of holes like a giant Swiss cheese. The ones demanding the policies are the ones doing most of the drilling.

But legalities aside, will it work on a very large scale in most places? Not right now I think. The dependency of the business on the current infrastructure is so big it can’t be replaced yet. So you need a transition and that means adding stuff & new possibilities and facilitating them. So initially it will only add complexity for the service desk. All the talk of not being able to retain the best and brightest might be true but the same goes for the IT personnel. You might retain a better MBA with your iPad & iPhone but you could very well lose some support personnel that go “BOINK” trying to assist a workforce with hundreds of devices and apps. Are devices and toys to be considered as benefits or as a true work instruments? Perhaps it attracts opportunistic gadget freaks instead of the best personnel. Do car policies help attract the best personnel in this day and age? I mean everyone offers it so it’s a level playing field. Perhaps not offering BYOD but providing really valuable environments works better. Flex work, telecommuting, better wages, interesting job content is still a lot better I think. The best people figure out fast that there is more and better to be had in remuneration than a device and your own app preferences.

Sure I know an iPad might attract a college graduate but they already have such high expectations (culture of entitlement) that perhaps this is not the best path to go. Corporate life is not like what you see on TV. They might as well learn that early. It’s not about a group of gorgeous young people acting important and professional whilst doing nothing, drinking rivers of macchiato form Starbucks and having affairs with the equally gorgeous colleagues. To complete the dream illusion they get paid generously for all that and at the end of the year receive a bonus to make a down payment on that city loft. Wake up! And be fair we’re talking top drawer human resources here and there in lies another issue as you’ll need to offer it to everyone in the company because, when you hear the lawyer talking, it opens you up to legal action due to discrimination if you don’t. Where is the differentiator than?

Now I’m not against it the concept. On the contrary I would love to see it work. But I’m afraid it’s not such a good proposition as it is made out to be when done in a structured way and on a large (read companywide) scale. Is it a perk or business value? I don’t have an iPad or an iPhone but I do use my tools and some devices out of corporate control to get my job done, so basically I’m there dudes. The main issue I still need to resolve is get employers to pay for expensive shiny toys I need to get my job done faster and better. The reason I don’t have them because I’m too cheap to buy them myself (so I don’t see the value to get my job done better?). But when the boss pays, well hello iPad! But I’d better not force my hand. I think my boss would say could luck at your next job if I ever told ‘m I takes an iPhone to retain me. But a CEO doesn’t have that problem. He gets a “right away sir” for an answer.

Is this for everyone? I’m not so sure. In the long term perhaps. Today no. I have generation-Y and millennial “kids” in my social circles and guess who’s asked to help them with all the tools, toys and gadgets? Right. They are indeed consumers! If you define digital natives as mere consumers than they fit the bill but I would suggest that the designation “digital natives” implies they can deal with all tech they use themselves at all times. In the end, when all self-service and tech support for their toys fail who do you think the problems ends up? Right. Ever dealt with a gadget junkie that is forced to go “cold turkey” in the blink of an eye? Face it, every helpdesk has to deal with recovering baby pictures, wedding movies, getting routers to work, helping with capturing a movie stream & configuring smartphones … consumers need support and that support has to be paid. Who does it and who pays is a different matter. Aren’t we just shifting it? What about contracts to make clear how does what, where and when? Have you ever work in a service desk in ICT for internal IT? Really? Where is all the “enabling of the business” when you’re waving with a contract as a user ends up at the service desk with a broken BYO device or application that was repaired but did not fix the issue and now they need help to get to the data stored in that obscure application you’ve never seen? And when it’s your manger are you going to put the contract in his or her face? What about the secretary that can make your life hell or heaven depending on how by the book they play? Sounds familiar? Same old, same old. One thing is for sure that cute, charming red head who’s very gadget minded and processes your requests for attending conferences doesn’t have a problem now and never will. No this is not sexist, it’s reality and you can always change the metaphor to reflect your own preferences, you’re totally free to do so Winking smile In essence what I’m saying here with freedom comes responsibility and ownership.

Then there are the practicalities who buys it and how does it get paid. You need have that figured out and organized. How do you deal with the legalities and auditing of licenses? Lawyer heaven Open-mouthed smile  Where are the tools to really manage devices and applications al those different vendors well?

Just some brainstorming and playing devil’s advocate here. Who wants this for work? Geeks. Who wants this a perks? Employees. Who wants this as a business? People selling solutions to manage and facilitate this. What does the business want?  The fact is consumerization of IT is already a reality. It just happens. It will be interesting to see how we all deal with it, why those choices are made and what their effects are. Feel free to chime in via the comments.

The Dilbert® Life Series: The Carbon Copy, Blind Carbon Copy e-mail Pandemic

Disclaimer: The Dilbert® Life series is a string of post on corporate culture from hell and dysfunctional organizations running wild. This can be quite shocking and sobering. A sense of humor will help when reading this. If you need to live in a sugar coated world were all is well and bliss and think all you do is close to godliness, stop reading right now and forget about the blog entries. It’s going to be dark. Pitch black at times actually, with a twist of humor, if you can laugh at yourself.

Have you ever worked in an office where no one ever takes responsibility and all communication is CC’d and BCC’d to an absurd number of people? There are corporate “cultures” that act very different from our own style. Size often has nothing to do with it. But this habit is just another symptom and indication of blatant management failure.

These are organizations where no one feels like they can make decisions or take actions without involving half of the company in some form of meeting or committee. CYA (Cover Your Ass) in action. One of the symptoms is the fact that just about anyone who has (or thinks they have) some important or urgent information sends all mail with some managers in CC or even BCC. Often the middle management acts the same way and before you know it more CC & BCC recipients are involved making the entire mail flow a mess and proving without any doubt that you’re in kinder garden. Decisions are postponed or never made. No one is going to take responsibility for a decision, that much is for sure. So when a decision is finally made it often by the wrong person, too late and probably not the best one. Basically, you have a management structure where no one knows who’s responsible and is utterly dysfunctional

This is also a symptom of another issue: managers without authority. Yes, they are not very good at their area of expertise; they can’t delegate or organize and lack real people skills. These are often found in middle management where they can be used by the upper level. After all, there needs to be someone between the hammer (management) and the anvil (employees). You see authority does not come from your rank or pay grade. It comes from what you know and can do and the support you get from other well-respected managers or leaders. If you need to CC all your bosses and all bosses of the people you’re emailing, this indicates that you’re a whining kid that can’t hack it. And no, simply not using CC or BCC anymore won’t solve that problem. I thought I’d mention this as they tend to think and act rather simplistic. We have a saying: “You salute the rank, not the man. You respect the man, not the rank.”

Anyway, the mail process is as most people in the mail are not involved, don’t care, don’t need and shouldn’t care, and hopefully don’t want to care. Once it got so out of control I added some more people to the CC list and wrote sarcastically at the top of the mail body that we really should make an effort to senselessly involve as many people as we possibly could. Not very nice, I know. Shouldn’t do that, I know. Some got the message, some didn’t. Another solution is to ignore the mail. Really if so many people, including a bunch of managers above and way above me, are in the CC list I would not have the arrogance to assume I have anything to say in the matter and thus I await their proposals or decisions.

The best employee a manager can have is Vanilla Ice. Really “…If there was a problem yo I’ll solve it …”, that’s what a good manager needs and wants. You see your boss has better things to do than micro-manage the details of your incompetence. You know your end state, so all you need to do is figure out what you need and how you’ll achieve it. Results, that’s what your boss really needs, not details, and moaning about how hard middle management is. I know shit flows down and gripes flow up but try to maintain a balance or you’ll find yourself holding a pink slip or being promoted to where you can do the least damage and annoy the least amount of people. I secretly think some people have that as a cunning plan.

But if you’re stuck with a couple of micro-managers, do not despair. You can work around them unless they surround you. In the latter case, break out and run! They deal with urgent and very pseudo important problems that are actually just details which are benign in nature and are not negatively affected by all that overzealous attention. So the trick is to keep it that way. You have to treat them like mushrooms: keep them in the dark and feed them shit. As long as they don’t know any better and keep getting their “data” fix they are lovable. Whatever you do, don’t give them real information or show them the real problems. Micromanagers really can’t handle them. Ambitious ones that get into the light and get gourmet food can become very dangerous. Both to themselves and the organization. Now you do need them to keep them involved and they need to sign and approve work and proposals. So give them finished work, solutions that are ready to go. Forget about involving them in the details or the decision-making process, they’ll just get lost. And guess what, this is like a good boss should work and act so we have a win-win situation for the entire company!

Now you know how to help prevent that e-mail becomes a burden instead of a useful tool. No CC or BCC unless really needed. Go practice it.

Technical Projects, Planning, Skills, Motivation & Psychopaths

When planning a technical project complexity adds up very fast. Take a virtualization project for example; a lot more things than just the hypervisor installation are coming into play. You’ll need to assess a lot of needs and desires about SANs (snapshots, redundancy, replication, FC, iSCSI, FCoE), network (VLAN, 1/10 Gbps Ethernet, redundancy), disaster recovery/business continuity, hypervisors and there capabilities, management of it all and security. That is a lot of stakes and agendas to take into consideration. And then you haven’t even talked to the business managers, the application owners, and developers. Now, this isn’t limited to virtualization, but this is just a nice example of how so many stakes come together in one project.

One of the major mistakes, that is made again and again even up until this day in the second decade in the 21st century is the fact that entire important or even critical IT systems are being put into place with a plan that can be paraphrased as follows “We’ll just set it up and sort of see how it evolves and just wing it from there”. I have been forced to do this quite often. This creates many problems some of which I will address below.

The single worst problem is that you create a vacuum. That can be storage space, bandwidth, ample resources for a huge amount of virtual machines or a mixture of all this. The results however are always the same and is one of two possibilities. Either they really don’t want and need it so it will never be used. You can also achieve this by keeping it hidden so they can’t use it. The other option is the most natural one. In nature, there is a thing called a “horror vacui”. That means that a vacuum unless protected cannot exist, it has to be filled. Empty LUNSs with data, hypervisor hosts with guests, networks with bandwidth, and backup capacity with even more terabytes. You might think the second option is better than the first one as at least the infrastructure is getting used. Unfortunately, the reality is that this is creating a very expensive mess to run, support & troubleshoot. The legacy this creates is not a valuable inheritance but a bank-breaking, efficiency, and effectiveness ruining debt. Stop doing that right now, you are killing your business. You see technology debt is about more than just old hardware and software. It’s about what you build with it or what grows organically with it. Is that a fertile land that sustains the business or a cancer that is killing it?

The way to prevent this is planning done by competent, involved people with experience and context. No plan is perfect, but a plan gives you a framework to achieve the desired result. Even great people make mistakes but they have the skills and attitude to fix them or work around them.

What are some other problems? Wasting money. Take for example a completely oversized server farm. That thing will consume so much money over a three year period in energy and idle capacity that the amount would be sufficient to replace it with new right sized hardware (more bang for the buck, better energy efficiencies in three years) I don’t know about you but those are very disconcerting numbers.

You can also be wasting money and time. And those who know me I loath wasting time. What if the SAN solution you bought doesn’t perform as planned or isn’t the right fit? There goes 500.000 € or you find yourself in the CEO office explaining why you need an extra 400.000 € to get what is really needed. Oh-oh! Do you have money and time to do it all over again or will you be living with that expensive mistake until the current solution is end of life? Do you have to wait until the CFO and CEO have recovered enough from the shock to allow a new attempt? Or perhaps you bought a SAN solution that is enough to run NASA’s workload and you’ve invested 4.000.000 € in a rather expensive data room heater.

Getting a virtualization project wrong can wreak havoc on a business and create a sizable financial hemorrhage. You can say that that’s not your problem but I beg to differ. If the project goes south that means you’ll have to find another job. The IT world where I live is rather small so you might even have to switch to another field as you’ll be forever known as the guy that sunk company X with his little “plan”.

The reverse, being rewarded for your hard work and success is not a given. In the end, they pay you for getting the job done so results are expected, and to Joe Average manager all ICT is a PC with a software packet to install. So for all you eager beavers who think that with this kind of responsibility and risk management comes big reward when you get it right, I suggest you think again. I have witnessed quite the opposite personally. Even when you’re running multiple enterprise SAN’s, networks, infrastructures like SQL Server, Exchange clusters, Hyper-V clusters, geo clusters, load balancers and providing 2nd and 3rd line support for those and taking 24/7 responsibility for the environment the only thing some managers care about is why the PC they never ordered with the software they never ordered can’t be installed tomorrow. “What kind of a chicken shit outfit are you running here” is what they’ll think when you can’t do that. They’ve read the glossy brochure that IT is a commodity and they expect it cheap and always on, much like electricity. In the end some (incompetent) managers act like ungrateful psychopaths. They’ll just abuse you less when you get it right. Don’t expect anything else. Often it’s the ones that are not capable to integrate things they can’t do or don’t understand into their business. They can not value anything that’s beyond their comprehension so they’ll never recognize it. To them, people are, for all practical purposes, resources that are identical, “Full Time Equivalents”. So don’t buy into the hype that there is a skills shortage from that lot and they can’t fill job openings. The volume in which they often waste talent and flush motivation down the drain is shockingly high and indicates that there is no shortage at all or that they can’t recognize skills when they find it and they’ll hire anyone. Surely they didn’t make a mistake so it must be a skills shortage. So you still want to be some hotshot technical architect? Or does a job that only produces open opinions and optional advice on paper sound more attractive. Per hour worked you’ll earn more, run less risk, and have a lot less stress. My advice? Don’t switch fields if you enjoy what you’re doing, switch jobs. The best career advice I ever got was “don’t work with or for assholes”.

Well if you don’t agree with your bosses and you dare go against them you’re surely playing with your job, you could get fired! So? Does living in fear of being fired make good employees? Does not being strong and confident enough to tell your managers they are doing certain things totally wrong or that they are mistaken make for good advisors? The worst thing a boss can have are a bunch of “yes men” around him or her. That boss should be smarter than that. It doesn’t work. Having trust in the abilities and loyalty of your employees does not mean you need to agree on everything. As a boss you’ll make the final decisions, yes, but you’d better listen very carefully to your advisors and staff or you might as well have hired some monkeys. You can train them to say yes all the time, all it takes are some bananas. As an employee, don’t let yourself be treated like a monkey and if they fire you for throwing the banana back, good for you!

So you’d better love technology and building solutions because that means you are intrinsically motivated to go the extra miles. When you are, select a small group of people with the same attitude. You’ll be able to drag the devil himself out of hell with such a team at a very  low cost. Whatever you, do don’t think you can externally motivate or coerce people into achieving this. Charles “Chargin’ Charlie” Beckwith knew that all along when he said “I’d rather go down the river with seven studs than with a hundred shitheads”. And guess what, he wasn’t taught this in some course, by getting a title or by being told this by a manager. He learned it himself by working with the best. These people will keep learning and growing on their own. They don’t need to be told what to do, how to train, what to use, they don’t need nannies & micromanagement. They need an end state and they’ll get it for you. Frankly, that kind of skillset and ability scares the shit out of some bosses as they micromanage actions & items instead of doing their jobs. You can’t use force, treats or authority to make people achievers. In the end, you can cut a diamond, but you cannot create it. Trust me. Putting that amount of pressure on someone that isn’t a diamond only turns them into a heap of crushed remains of what used to be a human being or FTE in your typical HR speak.

“Mate you’re not a conformist” my friend said … you’d better believe I’m not.

Why I’m No Fan Of Virtual Tape Libraries

After implementing a couple of SAN’s with backup solutions I have come to dislike Virtual Tape Libraries. This is definitely technology that, for us, has never delivered the promised benefits. To add insult to injury it is overly expensive and only good to practice hardware babysitting. When discussing this I’ve been told that I want things to cheap and that I should have more FTE to handle all the work. That’s swell but the business and the people with the budgets are telling me exactly the opposite. That explains why in the brochures it’s all about reduced cost with empty usage of acronyms like CAPEX and OPEX. But when that doesn’t really materialize the message to the IT Pros is to get more personnel and cash. In the best case (compared to calling you a whining kid) they‘ll offer to replace the current solution with the latest of the greatest that has, wonder oh wonder, reduced CAPEX & OPEX. Fool me once shame on you, fool me twice, shame on me.

So one thing I’m not planning to buy, ever again, is a Virtual Tape Library (VLS). Those things have a shelf life of about 2 years max. After that they turn into auto disintegrating pieces of crap you’ll spend babysitting for the rest of the serviceable live. This means regularly upgrading the firmware to get your LUN(s) back, if you get them back that is. This means convincing tech support to come up with a better solution than restarting from scratch when they acknowledge that their OS never cleans up its own log files and thus one day just kicks the bucket. Luckily they did go the extra mile on that one after we insisted and got a workaround without losing al backups. Babysitting also means that replacing the battery kits of all shelves becomes a new hobby. You become so good at it that you have better and faster way of doing it than the junior engineers they send who happily exclaim “so this is what it looks like”. The latter is not a confidence builder. The disks fail at a rate of 1 to 2 per week until you replaced almost all of them. Those things need to be brought down to fix just about anything. That means shutting down the disk shelf’s as well and cutting the power, not just a reboot so yes you need to be in the data center.

There is no RAID 6, no hot spares (global or otherwise). The disks cost an arm and a leg and have specialized hardware to make sure all runs fine and well. But in they are plain7.200 rpm cheap 500 GB SATA disks that cost way too much. The need for special firmware I can understand but the high cost I cannot. The amount of money you pay in support costs and in licensing the storage volume is more than enough to make a decent profit. Swapping disks and battery kits isn’t hard and we do it ourselves as waiting for a support engineer takes more time. We have spares at hand. We buy those when we by the solution. We’ve grown wise that way. We buy a couple units of all failure prone items at the outset of any storage project. Having only RAID 5 means that one disk failure puts you virtual tapes at a very high risk so you need to replace it as soon as possible. Once they shipped us a wrong disk, our VTL went down the drain due to incorrect firmware on disk. They demanded to know how it got in there. Well Mr. Vendor, you put it in yourself a as a replacement for a failed disk. In the first year it often happened they didn’t have more than 1 spare disk to ship. If anyone else has a VLS in your area you’re bound to hit that limit have to wait longer for parts. They must have upped the spare parts budget to have some more on hand just for us as we now get a steady supply in.

When you look at the complete picture the cost of storage per GB on a VLS is a much as on 1st tier SAN storage. That one doesn’t fly well. At least the SAN has decent redundancy and is luckily a 100 fold more robust and reliable. Why buy a VLS when you can have a premier tier SAN for the same cost, the VLS functionality? No sir that also never lived up to its promises. It has come to the point that the VTL, due to the underlying issues with the device, are more error prone than our Physical Tape Library. That’s just sad. Anyway, we never got the benefits we expected form those VTL. For disk based backup I don’t want a Virtual Tape Library System anymore. It just isn’t worth the cost and hassle.

Look you can buy 100 TB of SATA storage, put it in couple of super micro disk bays, add a 10Gbps Ethernet to your backup network and you’re good to go. Hey that even gives you RAID 6, the ability to add hot spares etc. You buy some spare disks, controllers, NICS, and perhaps even just build two of these setups. That would give you redundant backup to two times 100 TB for under 60.000 €. A VLS with 100TB, support and licensing will put you back the 5 fold of that. Extending that capacity costs an arm and a leg and you’re babysitting it anyway so why bleed money while doing that?

Does this sound crazy? Is this blasphemy? The dirty little secret they don’t like you to know is that’s how cloud players are doing it. First tier storage is always top notch, but if you talk about backing up several hundreds of terabytes of data, the backup solutions by the big vendors are prohibitively expensive. This industry looks a lot like a mafia racketeering business. Well if you don’t buy it you’ll get into trouble, you’ll lose your data. You won’t’ be able to handle it otherwise. Accidents do happen. The guys selling it even dress like mobsters in their suits. But won’t you miss out on cool things like deduplication? If your backup software supports it you can still have that. The licensing cost for this isn’t that bad a deal when compared to VLS storage costs. And do realize instead of 2100 TB you could make due to 2 25 TB. Hey that price even dropped even more.

When it comes to provisioning storage our strategy is to buy as much of your storage needs during the acquisition phase. That’s the only time deals can be made. The amount of discounts you’ll get will make you wonder what the markup in this market actually. It must be huge. And storage can’t cost as much as you think to build as they would like us to believe. Last time the storage sales guy even told us they were not making any money on the deal. Amazing how companies giving away their products have very good profit margins, highly paid employees with sales bonuses and 40.000 € cars If one vendor or reseller ever tells you that things will get cheaper with time, they are lying. They are actually saying their profit margins will increase during the life cycle of you storage solution. Look, all 1st tier storage is going to be expensive. The best you can hope for is to get a good quality product that performs as promised and doesn’t let you down. We’ve been fortunate in that respect to our SAN solutions but when it comes to backup solutions I’m not pleased with the state of that industry. Backups are extremely important live savers, but for some reason the technology and products remain very buggy, error prone, labor intensive and become very expensive when the data volumes and backup requirements rise.