Veeam Vanguard nominations are now open for 2018!

Just a quick blog post on the Veeam Vanguard program. The nominations for 2018 are open! That means that if you know people who would make a Veeam Vanguard you can nominate them. You can even nominate yourself, that’s perfectly fine. It’s not frowned upon, but it also doesn’t change anything in terms of evaluation for the program.

veeamvanguardnewlogo

Rick blogged on this yesterday on the Veeam blog in “Veeam Vanguard nominations are now open for 2018!” and gave some more insight of what the program is, tries to achieve and does. He also discusses the selection. The key take-away is that you cannot study for this and that it is not some kind of certification or such. Some of the current Vanguards were quoted on how they look at the program and one thing is constant in that. The fact that the people in these programs are contributors to the global tech community and it’s about sharing and helping others getting the best out of their environment and their investment in Veeam. It also helps Veeam as they get a very communicative group of people to give them feedback on their offerings, both products and services. It’s just one more tool that helps them get things right of fix thing when they got it wrong. Likewise understanding Veeam and their products better for us helps us make better decisions on design, implementation and operation of them.

You can have a look at the current lineup of Veeam Vanguards over here.

clip_image004

You’ll find a short video on the program on that page as well. So go meet the Vanguards and find their blog, their communities and follow @VeeamVanguard and the hash tag #VeeamVanguard to see what’s going on.

clip_image006

So, people, this is the moment if you want to nominate someone or yourself to join the Veeam Vanguards in 2018. You have time until December 29th 2017 to do so. I have always felt honored to be selected and have found memories of the events I was able to go to and I to this day I’m happy to be active in the Veeam Vanguard ecosystem. It’s a fine group of professionals in a program of a great company.

Webinar: 3 Emerging Technologies that will change the way you use Hyper-V

I have the pleasure of doing a webinar on October 24th 2017 with two fellow MVPs. They are Andy Syrewicze  from Altaro who organize the webinar and Thomas Mauer who’s well know expert in the tech community and beyond on Cloud and Data Center technologies.

The subject of the webinar is 3 Emerging Technologies that will change the way you use Hyper-V. It’s a panel style discussion amongst the 3 of us technology and trends that effect everyone in this business.

  • Public cloud computing platforms such as Azure and AWS
  • Azure Stack and the complete abstraction of Hyper-V
  • Containers and microservices: why they are game changers

image

There will be time for questions and discussions as we expect the subject to be of great interest to all. For my part I’ll try to look at the bigger picture of the technologies both from a product a service perspective as well as from a strategic point of view and a part of a doctrine to achieve an organizations goals.

Quick Fix Publish : VM won’t boot after October 2017 Updates for Windows Server 2016 and Windows 10 (KB4041691)

If you had WSUS (or SCCM) running tonight with auto approval on you might have woken up this morning to virtual machines that can boot anymore.

image

Great, another update gone wrong. Time to restore from backup as that can be the fasted way to restore services when in a pickle and if you have a good solutions for that in place. For the others you can do what I did is below. Actually a couple of us MVPs were on this issue at a number of sites as our fist task this morning. But first the root cause.

Well read this link Express update delivery ISV support and you have all you need. Basically the delta and the full cumulative update of October (KB4041691 – https://support.microsoft.com/en-us/help/4041691)  ended up in WSUS without you explicitly putting it there. That should not happen, normally the delta is not published for it to be downloaded and heaven forbid auto approved.  You could also have manually approved everything without really knowing what and why. Not a great idea at all.

image

So your VM get’s offered both of them and that is BAD!

image

Normally you get into this pickle if you some how managed to install both of these yourself or via other tools (see the link above), which you shouldn’t do.

Now if you don’t have decent restore capabilities from backups or snapshots there is another way out by removing the updates.

Boot into the problematic VM and select troubleshoot

image

Select to open the command prompt and stay away from any other auto repair options.

image

Microsoft advises to get rid of the SessionsPending reg key. To do so load the software registry hive as follows:

reg load hklm\temp c:\windows\system32\config\software

Delete the SessionsPending registry key, if it exists by running:

reg delete “HKLM\temp\Microsoft\Windows\CurrentVersion\Component Based Servicing\SessionsPending” /v Exclusive

Unload the software registry hive:

reg unload HKLM\temp

Run dism /image:c:\ /get-packages to find the updates installed that caused the issue

image

The yellow one are the ones of interest and you can see the first one never even got an install time/

We now use DISM to remove these updates.  Do first create the C:\Temp folder with MD temp if it doesn’t exist yet!

dism /image:c:\ /remove-package /packagename:myproblematicpackagetoremove /scratchdir:c:\temp

image

When done, close the command prompt, shut down the VM and then start it.

image

It will take a while but if will succeed and you’ll be greeted by a logon screen. Good luck!

Important: Do not try any other repair options or removing the updates with DISM might fail. We choose to remove all 3 updates from tonight to make sure. It might suffice to remove the delta one alone but we wanted to have an VM back as it was last night so more testing can be done before it is deployed again.

So, basically, don’t auto approve updates blindly, but test, validate & roll out in phases. Have great backup and TESTED restores. All by all we were only bitten in the lab, a couple of test/dev VMs and some of our infra VMs. Most of these are redundant and are patched stagger so our services were never badly effected. That gave us time to trouble shoot and investigate and warn our colleagues. As you can see here the issue was a delta update that made it into WSUS and was installed together with the full CU. Just manually downloading the CU and testing it would not have given you the heads up. About an issue. This is a reminder you need to test your real live situation and processes as realistically as possible. When you’re done with testing and cleaning up any fallout of this issue, make sure to patch your systems again!

Update: this also goes for Windows 10 Updates

Also see fellow MVP Mikael Nystrom blog post  https://deploymentbunny.com/2017/10/11/the-october-2017-update-inaccessible-boot-device/

Update: we now also have the official MSFT response & fix for each and every scenario right here https://support.microsoft.com/en-us/help/4049094/windows-devices-may-fail-to-boot-after-installing-october-10-version-o

I am presenting at VeeamON 2017

I’m travelling to New Orleans for VeeamON2. If you don’t know what that is, please check it out here. I can recommend this conference. Both the attendees and presenters are all very active users of Veeam products and the workloads Veeam protects in real live. That makes for excellent sharing of experiences, insights and knowledge with your peers.

SM banner-Presenters

I have the distinct honor of presenting a joint session with Luca Dell’Oca (@dellock6 / http://www.virtualtothecore.com/en/) and Carsten Reachfahl (@hypervserver / https://www.rachfahl.de/). The presentation is called: Throw your backups into ANY window and is on Wednesday, May 17 | 13:30-14:30.

Choosing a storage solution for your backups can be a daunting task: Windows or Linux servers, SMB shares, SAN, NAS, deduplication appliances … But block cloning, a new feature in Windows 2016 and leveraged by Veeam Backup & Replication™, is promising to change this. Available for ReFS 3.1 file systems, this technology allows for insanely reduced transform times and spaceless GFS backups. Or at least, this is what marketing has told us so far, but how good is it in reality? Is an expensive and complex Storage Spaces Direct the only way to consume all the amazing new features? How can I design my new backup repository with these new options in mind? What about encryption and Veeam Scale-out Backup Repository™? Didier Van Hoye, Carsten Rachfahl (both Microsoft MVPs and Veeam Vanguards) and Luca Dell’Oca (Veeam cloud architect) have joined forces to bring you from-the-field information, tips, tricks and ideas to build your next Veeam backup repository with real-life tests and feedback gained from deploying this new powerful combination into multiple environments.

This session is complimentary to the other ones given at VeeamON 2017, both the breakout sessions as well as some of the session the Microsoft MVPs are presenting at the boot. Those sessions combined will send you home with ideas and options on how to leverage Veeam in creative ways to achieve the best possible solution for your needs. Personally I’ll be discussing some of the options you have to get get high available backup targets leveraging ReFSv3.1 in brown field scenarios when  a brand new Storage Spaces Direct deployment is not option or when you don’t run Windows Server Windows Server 2016 yet.

Next to that and between attending interesting sessions I’ll be available at the Veeam and Microsoft boots if you want to have questions or want to discuss the technologies. At the Microsoft boot I’ll be presenting a demo focused walk through on how to on Discrete Device Assignment in Windows Server 2016.