Innovation As A Service

Disclaimer: The Dilbert® Life series is a string of post on corporate culture from hell and dysfunctional organizations running wild. This can be quite shocking and sobering. A sense of humor will help when reading this. If you need to live in a sugar coated world were all is well and bliss and think all you do is close to godliness, stop reading right now and forget about the blog entries. It’s going to be dark. Pitch black at times actually, with a twist of humor, if you can laugh at yourself.

Innovation As A Service for sale

Innovation! Get your fresh, real innovation right here ladies and gentlemen! Buy 3 innovations and pay only two! That’s right, the economies of scale at your fingertips. Get your innovation right here and now at pennies to the dollar!

You cannot buy innovation as a service. Maybe that’s why this website is painfully accurate. There is no such thing.

clip_image002

Results and success don’t come as the outcome of a process you can buy and follow. Creating some incubators, handing out some funds or price money isn’t going to help you innovate one bit.

This is especially true when you’re an organization that hasn’t got a culture to stimulate it. Not because culture will always determinate an outcome but because if your culture doesn’t evolve to enable what you need it probably isn’t a cause but a result. The result of lacking a real strategy.

When you’re not a small, agile, talent and motivation driven company it’s hard to become one by acquiring the resources (ideas, talent, motivation, products) you need. Many try but fail. Some succeed but even then it’s a time limited success. Innovation is an ongoing activity. Success is a temporary result. It’s not a long term, well defined and understood process or product. If it was it wouldn’t be innovative.

When you’re you need to optimize your processes or replace them if they can be optimized.

The organizational paradox

Innovation, or any “hot” item in business is like sex. The more people talk about it and want it the less it’s going to happen. This is frustrating too many so they try to facilitate or force some success. This leads to disappointing, second rate, clumsy and poor experiences compared to what it’s supposed to be.

As production, distribution and services are being disrupted by automation or even automatic / autonomous technology we see that profits come under stress. That means more with less, reduction of the work force & optimizing processes to reduce cost & optimize profits. This leads to more and more commodity driven businesses and organizations that are trying to compete on cost. - Dilbert by Scott Adams

A magic word has appeared that’s supposed to deliver a prosperous future for all of us. All this while we save ourselves into poverty and reduce our value into nothing more than 1 or 2 % better efficiencies than any other organizations at delivering commodities. If you don’t have a factual (Energy, “to big to fail” banking, …) or legal (notaries, IRS, …) monopoly you have very little margin as an excuse for your existence. So now we all have to be innovative. Never mind that you’re in the least innovative business on earth. Innovation it is as everyone is doing it, has or wants it. So must we or we’re clearly “out of touch”.

Get in the ring!

Pretending to be a boxer is lots of fun until you have to get into the ring and fight. When you step into the ring and your opponent smacks you in the face even your best plans fall apart. Let alone that sorry excuses of a plan you have in lieu of training, skills and motivation.

It’s very hard for established, highly regulated companies to really innovate. They’re about avoiding risks, optimizing processes & following procedures. The aim is to get the best predictable results with the fewest cost & overhead as long as this doesn’t endanger the processes and regulations. It just isn’t compatible with agility and innovation.

Now some large organizations can deal with this and make it a success.  It requires truly great strategies, strategists and some serious skills to make them materialize. Not a small feat to pull off! But way to many can’t. Nothing has eternal live. It’s even worse when such organizations make their innovation islands or incubators dependent on centralized, slow moving, process driven services & processes. That’s like swimming with an anvil tied around your ankle. Good luck!

DilbertInnovation

Let small organizations exist. Give them the freedom and independency they need to function. This entire mindset of centralized large entities that are driven by economies of scale with a giant focus (or at least the pretense of that) on efficiency of what already exists is mind numbing to most of us. Let alone to the best and brightest. It only “rewards” the accountant mindset. Trying to fix this with “pockets” of innovation and agility is like a reservation. It keeps a species alive artificially but it doesn’t make that species have a future and be all it can be to fulfill its potential It rewards opportunists who claim to be pioneers. Those that want the benefits and rewards that comes with that status without having to do the hard work. It’s lip service, smoke and mirrors.

The fallacy of bimodal IT in this regard is that a schizophrenic organization, with only two opposite extremes, is supposed to transition the fruits between those two “just like that”. The entire process, the evolution required and the gradual cultural shifts to bring the results of agility and innovation to the process driven highly regulated side of the business is never ever discussed or mentioned.

That “integration” is supposed to happen naturally & apparently without issues or effort. That’s a very hard order to fill It’s as if you can zip up the files of your project and e-mail them to the regulated part of the business and that’s all what’s needed. At best they’ll give you SharePoint Online and claim they’ve dealt with that issue as they don’t use attachments anymore. And no, a “Chief Integration Officer” is not going to help. But I’m sure you’ll find plenty of candidates to fill that function when you attach a higher pay grade to it. But just like innovation we’ll see the call for “bimodal” it along with many practitioners, consultants and coaches selling it. The successes will be rare and hard to find.

Video Spam in my WordPress blog comments?

Last night I noticed that a lot of my comments seemed to have random Vimeo video’s embedded into the comments. Sometimes one, some times many. At first you think somebody would use embedding Vimeo embedding code/tags to do so but it was very wide spread over many comments over the years.

image

So was I hacked? Nope. I did find no evidence of that and I also did not find any hidden code/tags. It turns out to be a Jetpack 3.9.5 bug and the fix was to either disable or get a new version of Jetpack that would fix the issue: Jetpack 3.9.5 and 3.9.6: Maintenance Releases

image

As 3.9.6 was available today I upgraded to that version to see if that would also fix this annoying issue.

image

After updating to Jetpack 3.9.6 the issue has indeed gone. Good news!

The Hyper-V Amigos Episode 10

It’s with great pride that the Hyper-V Amigos ride again and for The Hyper-V Amigos Episode 10 they dive into what’s new and improved in Windows Server 2016 Failover Clustering.

image

Well OK we only discuss a few subjects in this web cast as there is only a limited amount of time. I’ll present an overview of during my session at the German Cloud and Datacenter conference on May 12th in Germany. An hour is not enough for a deep dive into everything but we will build on our session we did at the Technical Summit (November 2014) in Germany on Improvements in Failover Cluster 2012 R2 ad get you up to speed so you can select what to investigate further.

Until then, enjoy the webcast and I hope it helps prepare you for what’s coming and entices you to join us at the Cloud and Datacenter Summit in Germany on May 12th! And if clustering alone is not enough to bring you over check out the agenda and you might realize what great gathering of experts is happing at the conference. Just look at the content, the breath and depth of the cloud and datacenter technologies being discussed is vast!

The Cluster and 0.Cluster Registry Hives

The cluster database

In a Windows Server Cluster the cluster database is where the cluster configuration gets stored. It’s a file called CLUSDB with some assisting files (CLUSDB.1.container, CLUSDB.2.container, CLUSDB.blf) and you’ll find those in C:\Windows\Cluster (%systemroot%\Cluster).

clip_image002

But the cluster database also lives in a registry hive that gets loaded when the cluster service gets started. You’ll find under HKEY_LOCAL_MACHINE and it’s called Cluster. You might also find a 0.Cluster hive on one of the nodes of the cluster.

clip_image004

The 0.Cluster hive gets loaded on a node that is the owner of the disk witness. So if you have a cloud share or a file share witness this will not be found on any cluster node. Needless to say if there is no witness at all it won’t be found either.

On a lab cluster you can shut down the cluster service and see that the registry hive or hives go away. When you restart the cluster service the Cluster hive will reappear. 0.Cluster won’t as some other node is now owner of the disk witness and even when restarting the cluster service gets a vote back for the witness the 0.Cluster hive will be on that owner node.

If you don’t close the Cluster or 0.Cluster registry hive and navigate to another key when you test this you’ll get an error message thrown that the key cannot be opened. It won’t prevent the cluster service from being stopped but you’ll see an error as the key has gone. If you navigate away, refresh (F5) you’ll see they have indeed gone.

So far the introduction about the Cluster and 0.Cluster Registry Hives.

How is the cluster database kept in sync and consistent?

Good so now we know the registry lives in multiple places and gets replicated between nodes. That replication is paramount to a healthy cluster and it should not be messed with. You can see an DWORD value under the Cluster Key called PaxosTag (see https://support.microsoft.com/en-us/kb/947713 for more information). That’s here the version number lives that keep track of any changes and which is important in maintaining the cluster DB consistency between the nodes and the disk witness – if present – as it’s responsible for replicating changes.

clip_image006

You might know that certain operations require all the nodes to be on line and some do not. When it’s require you can be pretty sure it’s a change that’s paramount to the health of the cluster.

To demonstrate the PaxosTag edit the Cluster Networks Live Migration settings by enabling or disable some networks.

clip_image008

Hit F5 on the registry Cluster/0.cluster Hive and notice the tag has increased. That will be the case on all nodes!

As said when you have a disk witness the owner node of the witness disk also has 0.Cluster hive which gets loaded from the copy of the cluster DB that resides on the cluster witness disk.

clip_image010

As you can see you find 0.hive for the CLUSTERDB and the equivalent supporting files (.container, .blf) like you see under C:\Windows\Cluster on the cluster disk un the Cluster folder. Note that there is no reason to have a drive letter assigned to the witness disk. You don’t need to go there and I only did so to easily show you the content.

Is there a functional difference between a disk witness and a file share or cloud witness?

Yes, a small one you’ll notice under certain conditions. Remember a file share of cloud witness does not hold a copy of the registry database. That also means there’s so no 0.Cluster hive to be found in the registry of the owner node. In the case of a file share you’ll find a folder with a GUID for its name and some files and with a cloud witness you see a file with the GUID of the ClusterInstanceID for its name in the storage blob. It’s bit differently organized but the functionality of these two is exactly the same. This information is used to determine what node holds the latest change and in combination with the PaxosTag what should be replicated.

The reason I mention this difference is that the disk witness copy of the Cluster DB is important because it gives a disk witness a small edge over the other witness types under certain scenarios.

Before Windows 2008 there was no witness disk but a “quorum drive”. It always had the latest copy of the database. It acted as the master copy and was the source for replicating any changes to all nodes to keep them up to date. When a cluster is shut down and has to come up, the first node would download the copy from the quorum drive and then the cluster is formed. The reliance on that quorum copy was a single point of failure actually. So that’s has changed. The PaxosTag is paramount here. All nodes and the disk witness hold an up to date copy, which would mean the PaxosTag is the same everywhere. Any change as you just tested above updates that PaxosTag on the node you’re working on and is replicated to every other node and to the disk witness.

So now when a cluster is brought up the first node you start compares it’s PaxosTag with the one on the disk witness. The higher one (more recent one) “wins” and that copy is used. So either the local clusterDB is used and updates the version on the disk witness or vice versa. No more single point of failure!

There’s a great article on this subject called Failover Cluster Node Startup Order in Windows Server 2012 R2. When you read this you’ll notice that the disk witness has an advantage in some scenarios when it comes to the capability to keep a cluster running and started. With a file share or cloud witness you might have to use -forcequorum to get the cluster up if the last node to be shut down can’t be started the first. Sure these are perhaps less common or “edge” scenarios but still. There’s a very good reason why the dynamic vote and dynamic witness have been introduced and it makes the cluster a lot more resilient. A disk witness can go just a little further under certain conditions. But as it’s not suited for all scenarios (stretched cluster) we have the other options.

Heed my warnings!

The cluster DB resides in multiple places on each node in both files and in the registry. It is an extremely bad idea to mess round in the Cluster and 0.Cluster registry hives to clean out “cluster objects”. You’re not touching the CLUSDB file that way or the PaxosTag used for replicating changes and things go bad rather quickly. It’s a bad situation to be in and for a VM you tried to remove that way you might see:

  • You cannot live or quick migrate that VM. You cannot start that VM. You cannot remove that VM from the cluster. It’s a phantom.
  • Even worse, you cannot add a node to the cluster anymore.
  • To make it totally scary, a server restart ends up with a node where the cluster service won’t start and you’ve just lost a node that you have to evict from the cluster.

I have luckily only seen a few situations where people had registry corruption or “cleaned out” the registry of cluster objects they wanted to get rid of. This is a nightmare scenario and it’s hard, if even possible at all, to recover from without backups. So whatever pickle you get into, cleaning out objects in the Cluster and/or 0.Cluster registry hive is NOT a good idea and will only get you into more trouble.

Heed the warnings in the aging but still very relevant TechNet blog Deleting a Cluster resource? Do it the supported way!

I have been in very few situations where I managed to get out of such a mess this but it’s a tedious nightmare and it only worked because I had some information that I really needed to fix it. Once I succeeded with almost no down time, which was pure luck. The other time cluster was brought down, the cluster service on multiple nodes didn’t even start anymore and it was a restore of the cluster registry hives that saved the day. Without a system state backup of the cluster node you’re out of luck and you have to destroy that cluster and recreate it. Not exactly a great moment for high availability.

If you decide to do muck around in the registry anyway and you ask me for help I’ll only do so if it pays 2000 € per hour, without any promise or guarantee of results and where I bill a minimum of 24 hours. Just to make sure you never ever do that again.