CryptoWall 3.0 Strikes To Close for Comfort

Instead of testing Windows Server 2016 TPv4 a bit more during “slow” hours we got distracted from that a bit CryptoWall 3.0 strikes to close for Comfort. Last week we, my team and I, had to distinct displeasure of having to tackle a “ransomware” infection inside a business network. Talk about petting a burning dog.

We were lucky on a few fronts. The anti malware tools got the infection in the act and shut it down. We went from zero and 100 miles per hour and had the infected or suspect client systems ripped of the network and confiscated.  We issue a brand new imaged PC in such incidents. No risks are taken there.

Then there was a pause … anything to be seen on the anti malware tools? Any issues being reported?  Tick tock … tick tock … while we were looking at the logs to see what we were dealing with. Wait Out …

Contact! The first reports came in about issues with opening files on the shares and soon the service desk found the dreaded images on subfolders on those shares.


Pucker time as we moved to prevent further damage and started an scan & search for more encrypted files and evidence of damage. I’m not going to go into detail about what, why, when and how. As in all fights you have to fight as you are. No good wishing for better defenses, tools, skills or training. At that moment you do what you think you need to do to contain the situation, clean up, restore data and hope for the best.

What can I say? We got lucky. We did our best. I’d rather not have to do that again. We have multiple types of backup & restore capabilities and that was good. But you do not want to call all data lost beyond a point and start restoring dozen of terabytes of corporate data to a last know good without any insight on the blast radius and fall out of that incident.

The good thing was our boss was on board to do what needed and could be done and let us work. We tried to protect our data while we started the cleanup and restores where needed. It could have been a lot uglier, costlier and potentially deadly. This time our data protection measures saved the day. And at least 2 copies of those were save from infection. Early detection and response was key. The rest was luck.

Crypto wall moves fast. It attempts to find active command and control infrastructure immediately. As soon as it gets it public key from the command and control server that it starts using to encrypt files. The private key securely hidden behind “a pay wall” somewhere in a part of the internet you don’t want to know about. All that happens in seconds. Stopping that is hard. Being fast limits damage. Data recovery options are key. Everyday people are being trapped by phishing e-mails with malicious attachments, drive by downloads on infected website or even advertisement networks.

Read more on CryptoWall 3.0 here  Details on how to protect and detect depend on your anti malware solution. It’s very sobering, to say the least.

It makes me hate corporate apps that require outdated browsers even more. Especially since we’ve been able to avoid that till now. But knowing all to well forces are at work to introduce those down grade browsers with “new” software. Insanity at its best.

Windows Server 2016 TPv4 Hyper-V brings virtual machine configuration version 7

When building a Windows Server  2016 TPv4 Hyper-V cluster this weekend I noticed that we now have a new version of the virtual machine configuration.

When we migrate (rolling cluster upgrade, move to new cluster or host, import on new cluster or host) virtual machines to  Windows Server 2016 Hyper-V from Windows Server 2012 R2, the virtual machine’s configuration file isn’t automatically upgraded. In the past it was, which blocked moving back to a previous edition of Hyper-V. Now we can do this until we manually update the virtual machine configuration version.  This block going back but it enables our new virtual machine features. Version 5.0 is the one that’s compatible with Windows Server 2012 (R2) Windows Server 2016. Version 6.2 was what we had in TPv3 and could only run on Windows Server 2016. Windows Server 2016 TPv4 Hyper-V brings virtual machine configuration version 7.

When you have virtual machines that come from  Technical Preview v3 and you had updated the virtual machine configuration of your virtual machines or created brand new ones these would be at version 6.2. Since I do not consider it wise to keep testing these on a version of a previous preview I updated them all to version 7.


The code below grabs all VMs on all cluster nodes (even the none clustered VMs), shuts them down, updates the configuration version and starts them again. It’s just a quick example.


Now do NOT do this to virtual machines with configuration version 5 that you might want to move back / import to a Windows Server 2012 R2 Hyper-V host. But if you know you’ll be testing with the new features, have a blast, like me here on the TPv4 lab cluster.


I’m still looking for the features version 7.0 enables, probably nested virtualization is one of those features I’m guessing. Happy testing!

Recent Changes In My Technology Community Life

Microsoft Most Valuable Professional

There have been some recent changes in my technology community life. As an MVP I have been assigned to the Cloud and Datacenter Management award category. This reflects the fact that we all touch on a lot more technologies than the expertise we have received or award for. In my case Hyper-V means I also do networking, storage,  high to continuous availability (clustering, network load balancing), data protection, IAAS as well as Identity Management (authentication/authorization) both on premises and on Azure.


In that spirit we attended the MVP Summit 2015, which was a great experience and confirmed what Scott Guthrie stated above, we are “most valuable professionals”.

Veeam Vanguard

Another award is decorating my home office. It’s the inaugural member edition of the Veeam Vanguard Award we received at VEEAMON 2015 in Las Vegas that we attended.


That conference was a blast by the way. Breakout sessions, white boarding sessions, presenting on Hyper-V related technologies and lots of networking with smart and engaged technologists. We also sat down with some CEOs of  2 companies and helped them determine an upgrade path for their hyper-V environments for the next 12 to 18 months. We  even some real world troubleshooting in one of the attendees environment. I’d like to think we delivered value for all involved and we got to learn a lot ourselves.

I liked what they shared about Veeam Backup & Replication v9 that’s in development. And their announcement for Veeam Backup for Linux was well received. You can preregister for that here

NIC Firmware/Driver Updates Reset Your RSS/VMQ optimizations

When optimizing your RSS/VMQ settings for maximum performance you’ll normally (I hope) do this in PowerShell. Save that script with some comments on why you configure it that way and make it part of your Hyper-V host deployment scripts

Why? Automation is king but you’ll need it again for sure. Why? Well there is this “tendency” that NIC firmware/driver updates reset your RSS/VMQ optimizations back to their defaults.That’s a bit of a bummer if you have to redo all the work instead of having a script ready to go. I have seen many a deployment where the configuration was missing after firmware/driver upgrades so please, check!


Figure: Where has my optimized configuration gone after a driver/firmware upgrade?

The good news is this isn’t a show stopper issue as things will keep working, but without your optimizations and with VMQ, depending on your NIC team setup for the vSwitch issues might occur. When doing NIC teaming for your virtual switch it’s important to get it right.  With switch dependent teaming (LACP/Static) the NICs in the team need to use overlapping processor sets (Min Queues). When doing switch independent teaming the NICs in the team need to use non-overlapping processor sets. So you need to configure each NIC in your team to use the different processors (Sum of Queues).

On top of that you might want to / should separate RSS/VMQ cores from each other. SMB Direct for CSV/LM will also help achieve this as there we leverage CPU offloading to the NIC.