Anti Virus & Hyper-V Reloaded

The anti virus industry is both a blessing and a curse.  They protect us from a whole lot of security threats and at the same time they make us pay dearly for their mistakes or failures. Apart from those issues themselves this is aggravated that management does not see the protection it provides on a daily basis. Management only notices anti virus when things go wrong, when they lose productivity and money. And frankly when you consider scenarios like this one …

Hi boss, yes, I know we spent a 1.5 million Euros on our virtualization projects and it’s fully redundant to protect our livelihood. Unfortunately the anti virus product crashed the clusters so we’re out of business for the next 24 hours, at least.

… I can’t blame them for being a bit grumpy about it.

Recently some colleagues & partners in IT got bitten once again by McAfee with one of there patches (8.8 Patch 1 and 8.7 Patch 5). These have caused a lot of BSOD reports and they put the CSVs on Hyper-V clusters into redirected mode (https://kc.mcafee.com/corporate/index?page=content&id=KB73596). Sigh. As you can read here for the redirected mode issue they are telling us Microsoft will have to provide a hotfix. Now all anti virus vendors have their issue but McAfee has had too many issues for to long now.  I had hoped that Intel buying them would have helped with quality assurance but it clearly did not. This only makes me hope that whatever protection against malware is going to built into the hardware will be of a lot better quality as we don’t need our hardware destroying our servers and client devices. We’re also no very happy with the prospect or rolling out firmware & BIOS updates at the rate and with the risk of current anti virus products.

Aidan Finn has written before about the balance between risk & high availability when it comes to putting anti virus on Hyper-V cluster hosts and I concur with his approach:

  • When you do it pay attention to the exclusion & configuration requirements
  • Manage those host very carefully, don’t slap on just any update/patches and this includes anti virus products of cause

I’m have a Masters in biology from they days before I went head over heals into the IT business. From that background I’ve taken my approach to defending against malware. You have to make a judgment call, weighing all the options with their pros and cons. Compare this to vaccines/inoculations to protect the majority of your population. You don’t have to get a 100% complete coverage to be successful in containing an outbreak. Just a sufficiently large enough part including your most vulnerable and most at risk population. Excluding the Hyper-V hosts from mandatory anti virus fits this bill. Will you have 100% success, always? Forget it. There is no such thing.

KB2616676 Patching Hiccup Discovered by Out of Sync Cluster Nodes

I was investigating an issue on a Windows 2008 R2 SP1 cluster and as part of my check list I ran the cluster validation. Than came out clean but for the fact that it complained about an update that was missing on some of the nodes.

That update was Microsoft Security Advisory: Fraudulent digital certificates could allow spoofing or KB2607712 Not that these cluster nodes are web clients but this is not good and we need to have this fixed for both security & cluster supportability reasons.

But neither WSUS or Windows Update indicate that there is an update available for these nodes. So I download the patch manually and try to install it. Then I get the response: ‘This update is not applicable to your computer’

No good! Now we need to find out what’s up. After some searching we find other people with this issue in the Microsoft forums: KB2607712 does not download to clients.

As it turns out KB2607712 was erroneously marked as superseded by KB2616676. This means that if that update is approved, or installed, the download/installation of KB2607712 is blocked. I check this on the nodes involved and this is indeed the case.

No please now that the forum reply states “erroneously marked as superseded” which means that BOTH updates are needed. The work around is to:

  • uninstall/unapprove KB2616676
  • install/approve KB2607712
  • reinstall/approve  KB2616676  again after you clients/host have KB2607712 installed.

There should be a revision of KB2616676 coming in the future that’s to include of KB2607712, meaning that KB2607712 will truly be supersede by it. As of this writing that revised version is not released yet so you’re left with the workaround until now.

Piece of advice. Keep your cluster nodes patched but do it in a well organized matter so they remain in sync.  Don’t just do half of the nodes. The good thing that came out of this that we discovered that some other servers/clients did not get the update for KB2607712 due to this. So now the company can address this issue using the workaround. I did the manual uninstall/reinstall workaround for the cluster nodes. For their clients an other servers  I suggested they’d go the WSUS way.