I made Veeam Vanguard 2018!

While attending the Microsoft MVP Global Summit 2018 I received notification that I was renewed as a Vanguard in 2018. This is my forth year, as I’m one of the inaugural members in 2015.

clip_image001

The Veeam Vanguard group is a collection of smart, hardworking IT experts that have a healthy interest in data protection and availability. No matter what you build in IT to support your business or customers it requires to be protected against down time. You also need the ability to perform disaster recovery and deliver business continuity for those days things are not going smoothly. Those requirements keeps these technologist busy and honest. They have to deliver on those requirements and they can’t talk their way out of not being able to do that when needed. The result is that this group of experts is very experienced and knowledgeable in both their specialties and in how to protect their workloads. Being part of the Veeam Vanguards means sharing that experience and knowledge and tapping in to their collective brain power. I’m happy and proud the be a Veeam Vanguard as it is a great learning experience and it helps me to deliver even more value to my employers and all Veeam customers. It’s win-win all over. Thank you Veeam for the opportunity and recognition.

Veeam Vanguard nominations are now open for 2018!

Just a quick blog post on the Veeam Vanguard program. The nominations for 2018 are open! That means that if you know people who would make a Veeam Vanguard you can nominate them. You can even nominate yourself, that’s perfectly fine. It’s not frowned upon, but it also doesn’t change anything in terms of evaluation for the program.

veeamvanguardnewlogo

Rick blogged on this yesterday on the Veeam blog in “Veeam Vanguard nominations are now open for 2018!” and gave some more insight of what the program is, tries to achieve and does. He also discusses the selection. The key take-away is that you cannot study for this and that it is not some kind of certification or such. Some of the current Vanguards were quoted on how they look at the program and one thing is constant in that. The fact that the people in these programs are contributors to the global tech community and it’s about sharing and helping others getting the best out of their environment and their investment in Veeam. It also helps Veeam as they get a very communicative group of people to give them feedback on their offerings, both products and services. It’s just one more tool that helps them get things right of fix thing when they got it wrong. Likewise understanding Veeam and their products better for us helps us make better decisions on design, implementation and operation of them.

You can have a look at the current lineup of Veeam Vanguards over here.

clip_image004

You’ll find a short video on the program on that page as well. So go meet the Vanguards and find their blog, their communities and follow @VeeamVanguard and the hash tag #VeeamVanguard to see what’s going on.

clip_image006

So, people, this is the moment if you want to nominate someone or yourself to join the Veeam Vanguards in 2018. You have time until December 29th 2017 to do so. I have always felt honored to be selected and have found memories of the events I was able to go to and I to this day I’m happy to be active in the Veeam Vanguard ecosystem. It’s a fine group of professionals in a program of a great company.

Troubleshooting Veeam B&R Error code: ‘32768’. Failed to create VM recovery snapshot

I recently had to move a Windows Server 2016 VM over to another cluster (2012R2 to 2016 cluster)  and to do so I uses shared nothing live migration. After the VM was happily running on the new cluster I kicked of a Veeam backup job to get a first restore point for that VM. Better safe than sorry right?

image

But the job and the retries failed for that VM. The error details are:

Failed to create snapshot Compellent Replay Manager VSS Provider on repository01.domain.com (mode: Veeam application-aware processing) Details: Job failed (‘Checkpoint operation for ‘FailedVM’ failed. (Virtual machine ID 459C3068-9ED4-427B-AAEF-32A329B953AD). ‘FailedVM’ could not initiate a checkpoint operation: %%2147754996 (0x800423F4). (Virtual machine ID 459C3068-9ED4-427B-AAEF-32A329B953AD)’). Error code: ‘32768’.
Failed to create VM recovery snapshot, VM ID ‘3459c3068-9ed4-427b-aaef-32a329b953ad’.

Also when the job fails over to the native Windows VSS approach when the HW VSS provider fails it still does not work. At first that made me think of a bug that sued to exist in Windows Server 2016 Hyper-V where a storage live migration of any kind would break RCT and new full was needed to fix it. That bug has long since been fixed and no a new full backup did not solve anything here. Now there are various reasons why creating a checkpoint will not succeed so we need to dive in deeper. As always the event viewer is your friend. What do we see? 3 events during a backup and they are SQL Server related.
image

image

image

On top of that the SQLServerWriter  is in a non retryable error when checking with vssadmin list writers.

image

It’s very clear there is an issue with the SQL Server VSS Writer in this VM and that cause the checkpoint to fail. You can search for manual fixes but in the case of an otherwise functional SQL Server I chose to go for a repair install of SQL Server. The tooling for hat is pretty good and it’s probably the fastest way to resolve the issues and any underlying ones we might otherwise still encounter.

After running a successful repair install of SQL Server we get greeted by an all green result screen.

image

So now we check vssadmin list writers again to make sure they are all healthy if not restart the SQL s or other relevant service if possible. Sometime you can fix it by restarting a service, in that case reboot the server. We did not need to do that. We just ran a new retry in Veeam Backup & Replication and were successful.

There you go. The storage live migration before the backup of that VM made me think we were dealing with an early Windows Server 2016 Hyper-V bug but that was not the case. Trouble shooting is also about avoiding tunnel vision.

VeeamOn 2017 Points of Interest

Introduction

I’m back form attending, speaking, learning and sharing experiences and knowledge at VeeamON 2017 (and DELL EMC World before). It was a blast and I had the opportunity to engage in very interesting discussions with experts from around the globe.

image

As it was a Veeam event it wil be no surprise that we got some very interesting information about the new Veeam offerings now as well as in the near future. Points of particular interest to me are:

  • Veeam backup for file shares. Really this might solve my entire dubio around virtualizing very large capacity clustered files shares (100-200TB) I have to protect. I’m looking forward to testing and leveraging the various restore options like File share rollback. Handy when ransomware just struck.
  • I like what Veeam is doing for disaster recovery in Microsoft’s Azure public cloud. Veeam’s Direct Restore and new Power Network (PN) in order to facilitate and automate the disaster recovery process.
  • The Veeam agent that can protect Windows ad Linux based physical servers and endpoints, along with applications running in Microsoft Azure, AWS and other public clouds tied into Veeam Backup & Replication. We will also get support for failover clusters with this. Something I have been lobbying for!
  • They support native object storage support using Amazon S3, Amazon Glacier, Microsoft Azure Blob etc.
  • They announced improved and extended Office 365 protection including OneDrive for Business and SharePoint Online. One of those improvements is very handy with multiple tenants.
    Ramsomware did something very significant beyond reminding everyone of the importance of recoverable backups and that is reigniting the interest in tape as a backup medium. The inherent “air gap” that tape offers has become more interesting to many people as ransomware can also delete or encrypt backups. So the 3-2-1 rule has never been more important and is being extended by additional rules of thumb. The product to investigate for me is Starwind Virtual Tape Library (VTL). What I like is that I can have an air gapped backup integrated with Veeam in Amazon AWS. Even while my entire business might run in Azure, this separates my data protection technology and location form my production / development environment. Ideal for maximum isolation to protect us form both external and insider threats and risks while avoiding the need to deal with physical tapes. This is and remains a major concern for operational costs and RTO.

Conclusion

The new capabilities are very welcome to help solve the challenges we have now and the ones we see coming in the near future. We have plenty of ideas and plans to build the next generation of data protection and data availability solutions. Whatever the need, on-premises, IAAS, PAAS, SAAS, private/hybrid/public cloud, the need to protect data against loss and down time is there in one form or another. That is and remains a primary responsibility of any business regardless of the technology. As always, my fellow MVPs and Vanguards are ready, willing and able to get the job done.