Why You Need an Immutable Backup Repository, And Why Veeam’s Just Enough OS and VBR appliance are the Smart Choice

Posted on by
Reply

Introduction

Let’s be honest: if you’re still relying on traditional backup strategies without immutability in today’s threat landscape, you’re playing a game of Russian roulette. Ransomware isn’t just a buzzword. It’s a business model. Insider threats aren’t hypothetical; they’re happening. And when the proverbial shit hits the fan, your backups are either your lifeline or your liability. Then there are wipers, who want to destroy data and your business, nothing else. They don’t ask for ransom payments, blackmail you to stop them from exposing your confidential files, or threaten to harm your personnel physically. Destruction and mayhem are all they care about. You need protection!

So, how do you prepare for that? You need a hardened repository, providing immutability and protection from deletion! Not just any Windows or generic Linux box with some tweaks, but a purpose-built, security-first solution. And suppose you want to avoid reinventing the wheel while staying compliant and operationally sane. In that case, the Veeam Hardened Repository ISO or its successor, Veeam Just Enough OS (Veeam JEOS), is the recommended approach.

Now, while I focus on the why related to Veeam’s Hardened Repository ISO, it is worth noting that an immutable repository does not exist in isolation. The 3-2-1-1-0 rule, CPU core and memory sizing, redundancy, high availability, IOPS, throughput, and storage and networking capacity matter! However, when it comes to the 3-2-1-1-0 rule, I have always stated that I don’t count the production workload as a copy. However, that one immutable copy is something I’m gradually changing into zero non-immutable and deletable copies.

Additionally, hardening any role in your backup fabric is now a must. Everything is a target, including your employees, via social engineering.

Hardened Linux Repository with immutability

Using a Hardened Linux Repository with immutability should be mandatory. None of this is about being paranoid; it’s about being prepared. Sure, you can laugh at me, say it is overkill or too expensive. Laughing is healthy, so keep doing that. But listen to me. It is not overkill and is not more costly. It is not even more cumbersome, except for the inevitable extra steps in a zero-trust workflow. There is even a bonus: when ransomware strikes, listening to me might keep that smile on your face!

You may have seen my blog post, Revised script for decrypting datacenter credentials from the Veeam Backup & Replication database | Working Hard In IT. That post does not mean Veeam or Windows cryptography implementations are inherently flawed; it highlights the inevitable consequences of having root access to your system. Hence, you can guess that you require any server role in your Veeam backup environment to be hardened as much as possible. Veeam is therefore also providing a Veeam Software Appliance (VeeamSoftwareAppliance_13.0.0.12109.BETA2.iso).

When you build your own Veeam Hardened Linux Repository, you must take technical measures and establish a process flow to service genuine requests and protect against both external and internal malicious actions. All that is taken care of by the Veeam appliance approach. Not too shabby, not too shabby at all!

A hardened Linux repository is a tactical and strategic asset in a backup fabric. It gives you a fighting chance and serves as an ark of Noah to start over from. Below, we will discuss why it should be a mandatory component in your architecture.

Immutability is essential

If your backups can be deleted, encrypted, or tampered with, you don’t have backups, but “hope”. You have a false sense of security. Immutability ensures that your backup data is locked down and protected from malware or rogue administrators.

Pre-Hardened OS

Security isn’t just about firewalls and antivirus. It’s about reducing your attack surface. A pre-hardened OS turns off unnecessary services, enforces strict access controls, and aligns with best practices from the outset. That means a lot of work and worrying that you don’t have to do.

STIG Compliance

Want to sleep better at night? Align with government-grade security standards. STIG compliance ensures your repository is secure, and you can reference Veeam to support that claim when needed.

Ransomware Resilience

Ransomware loves backup data. It’s the first thing attackers go after. A hardened repository isolates your backups and enforces immutability, making it a fortress against encryption attempts.

Auditability & Compliance

GDPR, HIPAA, and ISO 27001 compliance isn’t optional. Hardened repositories support forensic analysis, secure logging, and system integrity checks. You’re not just protected; you can prove to an auditor. Yes, compliance is a thing, and while the actual protection comes before compliance reports, we cannot ignore that.

Operational Stability

Misconfigurations are the silent killers of IT. A hardened repo minimizes that risk. With pre-applied security settings, even teams without deep Linux chops can deploy confidently.

Maintenance without effort

Security updates and patches? Streamlined. Veeam handles the OS and repo updates, so you don’t have to babysit your infrastructure. I still need to determine if the ISO can also handle firmware updates for you.

Insider Threat Mitigation

Not every threat comes from outside. Role-based access, BMC port protection, and restricted shell access help prevent internal sabotage, whether accidental or intentional.

Strategic Value

All the above is not just a technical and operational advantage. It’s a business win. A hardened repo ensures your backups are a reliable recovery point, even when everything else goes sideways. It is your Ark of Noah! And guess what? Have redundant Arks! One is none, two is one 😉.

Why Veeam’s Just Enough OS ISO Is a Game-Changer

You could build your own hardened Linux repo. I’ve done it. It works. But it’s not for everyone. Veeam’s Hardened Repository ISO (VeeamHardenedRepository_2.0.0.8_20250117.iso) streamlines the process, automates the hardening, and provides a vendor-backed solution that’s ready for production.

The future Veeam Just Enough OS hardened repository (VeeamJEOS_13.0.0.12109.BETA2.iso) is well locked down, and privileged actions require security officer approval. While that is essential in a zero-trust world, it also means you must have your processes streamlined and communication lines open. When people need to reset a password or require root access for troubleshooting, they cannot wait until the next business day when the security officer is at work, let alone a week, because somebody has to bring it up at the weekly CISO approval board.

Look, you can roll your hardened repository if you have the skills, time, and appetite for ongoing maintenance. I have done that and might still do so depending on the environment and requirements. However, if you’re looking for a secure, compliant, and low-maintenance solution that works, Veeam Hardened Repository ISO or its successor, Veeam Just Enough OS, is the answer. By starting today, testing these solutions, you gain insights and experience using them, and will be optimally prepared for when Veeam Backup & Replication v13 becomes available. The Veeam Hardened Repository ISO has experimental support, enabling its use in production environments. At the very least, you can store one backup copy on it today. If you are interested in this for future use, consider Veeam Just Enough OS (VeeamJEOS_13.0.0.12109.BETA2.iso) as part of the future V13 release. However, that one is not yet production-ready. But it won’t be long now when we look at the post by Anton Gostev on LinkedIn! At the time of writing this post, it should be less than a month.

Conclusion

The above is not paranoia, and it’s not just about ticking boxes for compliance. It’s about building a backup strategy that survives real-world threats. And in that world, immutability isn’t optional. It’s your insurance policy. Look, I have seen the devastation ransomware causes. It is a horrible place to be. I don’t want you to be in that world of hurt. However, we cannot prevent it. You are a target, and you will get hit. It is a question of when, not if. So make sure you have the means to come out on top!

Revised script for decrypting datacenter credentials from the Veeam Backup & Replication database

Posted on by
1

Introduction

In a previous article (Protecting your Veeam Backup and Replication Server is critical | Working Hard In IT), I discussed my script for decrypting the datacenter credentials from the Veeam Backup & Replication database. Since then, that PowerShell code has been published dozens of times all over the internet in various articles.

However, three relevant things have changed since my original blog post:

  1. Veeam v12.1 introduced a new encryption method.
    Firstly, in Veeam 12.1, the method of encrypting passwords has changed. That means the old script no longer works (always) as it only uses the legacy method.
  2. Veeam published its encryption and decryption methods.
    Secondly, Veeam has published the methods used to encrypt and decrypt passwords in the spirit of full disclosure and to preempt anyone who attempts to claim that Veeam is insecure. Those individuals or companies demonstrate only ignorance and malicious intentions. The good news is that the article has all the information we need to write a new script.
  3. Veeam now supports PostgreSQL, in addition to Microsoft SQL Server.
    Finally, Veeam now also supports MySQL as a database, in addition to Microsoft SQL. That means we need to ensure that we can retrieve the necessary data from both database types.

Background Info & approach

I based the script on information found in the Veeam KB article “How to Recover Account Credentials From the Veeam Backup & Replication Database” (https://www.veeam.com/kb4349).

Instead of having two scripts, my old one and a newer one. I decided to create one that would work on VBR v12 and lower, as well as on VBR 12.1 and higher.

What Changed in Encryption

Until version 12, Veeam used its internal .NET static method:

[Veeam.Backup.Common.ProtectedStorage]::GetLocalString($encryptedPassword)

That method leverages the native Microsoft Data Protection API (DPAPI) under the hood. It was part of the Veeam.Backup.Common.dll and worked well up to version 12. In v12.1 and beyond, this method no longer exists. Instead, Veeam now leverages the native Microsoft Data Protection API (DPAPI), directly:

[System.Security.Cryptography.ProtectedData]::Unprotect($bytes, $salt, ‘LocalMachine’)

Since both leverage the native Microsoft Data Protection API, I figured I could also use the [System.Security.Cryptography.ProtectedData]::Unprotect static method to decrypt those legacy passwords as long as I don’t try to leverage the optionalEntropy parameter for them. The good news is that in the KB article, Veeam provides instructions on how to differentiate between the legacy and new types of password encryption. That allows me to write logic to determine the version and execute the corresponding decryption method accordingly.

By the way, once you update a password on v12.1 or up, it will be encrypted with the new method. As time passes, by rotating the passwords, legacy encryption phases out.

The new script

I did not want to maintain two separate scripts, one for the legacy password decryption method and one for the newer one. That’s why  I’ve consolidated everything into a single, unified PowerShell script. It supports:

  • Supports VBR v10 through v12.3+ and decrypts Veeam credentials from registry and database.
    • The Veeam Backup & Replication encryption salt in the registry lives here: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\Data.

    • The Veeam database info in the registry lives here:
      Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\DatabaseConfigurations\
  • Per-user counters and clean output formatting
  • Supports MSSQL and PostgreSQL configurations
  • Handles multiple password formats:
    • ‘v12 and lower’
    • ‘v12.1 and up (with encryption salt)’
  • Optional filtering by username
  • Optional export to file (`Veeam_Credentials.txt` on Desktop)
  • Graceful error handling and informative console output

The script runs on Windows only, because DPAPI is a Windows-native feature. With VBR v13 introducing Linux-based deployments, this script won’t work in those environments. That’s a different challenge for another day.

Getting the script

You can find the script on GitHub at https://github.com/WorkingHardInIT/Revised-script-for-decrypting-datacenter-credentials-from-the-Veeam-Backup-Replication-database. You will also find the documentation there.

Why do I need this script?

The IT world, like everywhere else, is not a perfect place, and I need a way to deal with imperfection. It is that simple. If we are honest, we all know that  IT environments aren’t always in pristine condition. Whether it’s a lab, a forgotten backup server, or an entire backup fabric for a production environment abandoned by a previous IT partner, credentials are often missing. Documentation is sparse. And when disaster strikes, you need access, fast.

My script has already helped IT teams recover access to critical systems when no one else could. I know because I’ve seen it happen. Before Veeam ever published its KB article, my original script was quietly saving the day in real-world scenarios.

Conclusion

Knowledge is power. And while power inherently allows abuse, hiding knowledge under the guise of “security” is just security theater. Security through obscurity is not security but window dressing.

That’s why I’m glad Veeam documented their credential encryption methods. It empowers administrators to recover access responsibly. And it exposes the charlatans who twist transparency into baseless accusations of insecurity. I just felt compelled to create a handy, functional script around it that I can use when needed.

If someone uses this information to claim Veeam is irresponsible, they could not be more wrong. They prove themselves to be untrustworthy. To me, they’ve lost their reputation and credibility.

This script isn’t about hacking. It’s about recovery, accountability, and clarity. And if it helps you regain control of your environment when all else fails, then it’s done its job.

Veeam Hardened Repository ISO: Overview and Requirements

Posted on by
Reply

Introduction

Readers of my blog and other articles will know that I am a strong advocate of immutable backups, and Veeam delivers this functionality through its Linux Hardened Repository. I have several articles on how to set this up, secure it, add MFA, extend and repair XFS volumes, and more. I have designed and run many successful deployments in production.

In my latest designs, I have introduced a process flow to ensure that backups are not only immutable but also undeletable. The way to do this is to disallow root/sudo access to key personnel who are not involved in daily operations, and who must agree to allow and grant access under the 4-eyes principle. Why? To ensure no one, accidentally or otherwise, makes preventable, bad decisions.

Still, I notice that many people are hesitant to use it, as the perceived complexity of Linux deters them. Veeam has been addressing this perception, which is partially real and partially driven by fear, by providing the Veeam Hardened Repository ISO to simplify deployment and maintenance. Today, we will be looking into that

The Veeam Hardened Repository ISO

The Veeam Hardened Repository ISO (abbreviated to VHRISO on the forums) is a preconfigured, bootable ISO image based on the Rocky Linux distribution, developed and maintained by Veeam. It delivers a Managed Hardened Repository solution designed to simplify deployment and enhance security for backup infrastructures.

This solution caters to the masses to provide better security for all:

  • Minimize the need for Linux expertise during the setup process.
  • Provide a hardened operating system with advanced security configurations applied by default.
  • Ensure secure and compliant backup storage aligned with industry standards.

Security Foundation

The operating system embedded in VHRISO is pre-hardened using guidelines from the Security Technical Implementation Guides (STIGs), maintained by the Defense Information Systems Agency (DISA) for Rocky Linux. All this ensures that even if immutability is enabled, misconfigurations are less likely to compromise the system.

Support Status

As of October 29, 2024, VHRISO transitioned from Community Preview to Experimental Support status. That means that production use is officially supported.

You can open support cases for issues, except those related to the ISO Installer and Configurator Tool, which fall under experimental SLA terms.

Only unmodified versions of VHRISO deployed on compliant hardware are eligible for support.

Veeam announced that it will integrate the standalone ISO into the platform in the next release of Veeam Backup & Replication, V13.

The main points of that announcement are:

  • Veeam will integrate the standalone ISO into the platform via the new “Just Enough OS” (JeOS) ISO, which will enable deployment of various backup roles, including the hardened repository.
  • Centralized Updates: JeOS will manage and update the OS and Veeam components across all backup infrastructure roles, simplifying maintenance with automatic patching during scheduled windows.
  • Easier Provisioning: V13 removes the need for complex passwords in setting up a hardened repository. It will use thumbprint verification and a temporary PIN code for repairing with backup servers.
  • Host Management Web UI: A new web interface will provide an easy-to-use management tool for JeOS and Veeam settings, with security safeguards to minimize exposure.
  • Full Support for Managed Repositories: Managed hardened repositories deployed from the V13 JeOS ISO will now be fully supported, moving beyond experimental status.

See Anton Gostev’s announcement here: https://www.linkedin.com/posts/askgostev_weve-been-getting-many-questions-about-our-activity-7312464807171923969-q1YR/

Latest Release

On January 29, 2025, Veeam released Build 2.0.0.8, available via:

You can find it in the Veeam Customer Portal https://www.veeam.com/download_add_packs/vmware-esx-backup/hardened-repository/

Trial Downloads section under: Additional Downloads > Extensions and Other > Veeam Hardened Repository ISO

System Requirements

To ensure compatibility and optimal performance, you must meet the following prerequisites:

Software Requirements

  • Veeam Backup & Replication version 12.2 or later

Hardware Requirements

  • You must use hardware from the Red Hat compatibility list or the Certified Quality and Independent organization certified hardware list
  • Enable UEFI Secure Boot
  • Do NOT install third-party security software
  • Only hardware RAID controllers are supported
    • Software RAID, Intel VMD VROC, and FakeRAID are not supported
    • RAID controllers must have write-back cache enabled
  • Use internal or direct-attached storage only

Storage Configuration

  • Minimum of two storage volumes:
    • One for the OS (≥100 GB)
    • One or more for data (must be larger than OS volume)
  • The smallest disk must be identifiable (e.g., 100 GB + 101 GB is valid; 2x 100 GB + 1x 200 TB is invalid)
  • Recommended: Dual-parity RAID configuration

Network Requirements

  • Standard backup repository ports must be open
  • You must allow direct or HTTP proxy access to repository.veeam.com on port 443 for:
    • OS and security updates
    • GPG key renewal (failure to update will require complete OS reinstallation)

Security Best Practices

  • Secure the Baseboard Management Controller (BMC) port using firewalls and strong credentials
  • Avoid deploying VHRISO on virtual machines due to:
    • Increased attack surface via hypervisor
    • Risk of backup inaccessibility during host outages

New Features in Build 2.0.0.8

  • Repair Mode: Reinstall the OS while preserving data partitions.
  • Live Boot: Built-in diagnostics and performance testing.
  • Zero-Touch Installation: Fully automated deployment using Kickstart.
  • IPv6 DHCP Support: Enhanced connectivity options.
  • Enhanced Ping Limits: Rate-limited pings for better troubleshooting.
  • Improved Workflow: Clearer installation steps and safeguards against accidental disk formatting.

Conclusion

The Veeam Hardened Repository ISO aims to provide hardened and immutable repositories in as many deployments as possible. I think they are making progress in achieving this goal. I believe that every Veeam Backup Fabric deployment, whether small or large, should have hardened repositories with immutable backup copies. That is my more recent stance. I used to do it for at least one copy, as that worked out well with refresh projects, but I want to end up with all repositories and backup data copies being immutable and stored on a hardened repository.

I am currently building a lab for the Veeam Hardened Repository ISO to gain experience with it and be well-prepared for the arrival of Veeam Backup & Replication V13. I hope to share some information on that later.

The rejuvenated push for excellence by Veeam for Hyper-V customers

Posted on by
Reply

Introduction

As an observer of the changes in the hypervisor market in 2024 and 2025, you have undoubtedly noted considerable commotion and dissent in the market. I did not have to deal with it as I adopted and specialized in Hyper-V from day one. Even better, I am pleased to see that many more people now have the opportunity to experience Hyper-V and appreciate its benefits.

While the UI management is not as sleek and is more fragmented than that of some competitors, it offers all the necessary features available for free. Additionally, PowerShell automation enables you to create any tooling you desire, tailored to your specific needs. Do that well, and you do not need System Center Virtual Machine Manager for added capabilities. Denying the technical capabilities and excellence of Hyper-V only diminishes the credibility and standing of those who do so in the community.

That has been my approach for many years, running mission-critical, real-time data-sensitive workloads on Hyper-V clusters. So yes, Microsoft could have managed the tooling experience a bit better, and that would have put them in an even better position to welcome converting customers. Despite that, adoption has been rising significantly over the last 18 months and not just in the SME market.

Commotion, fear, uncertainty, and doubt

The hypervisor world commotion has led to people looking at other hypervisors to support their business, either partially or wholesale. The moment you run workloads on a hypervisor, you must be able to protect, manage, move, and restore these workloads when the need to do so arises. Trust me, no matter how blessed you are, that moment comes to us all. The extent to which you can handle it, on a scale from minimal impact to severe impact, depends on the nature of the issue and your preparedness to address it.

Customers with a more diverse hypervisor landscape means that data protection vendors need to support those hypervisors. I think that most people will recognize that developing high-quality software, managing its lifecycle, and supporting it in the real world requires significant investment. So then comes the question, which ones to support? What percentage of customers will go for hypervisor x versus y or z? I leave that challenge to people like Anton Gostev and his team of experts. What I can say is that Hyper-V has taken a significant leap in adoption, as it is a mature and capable platform built and supported by Microsoft.

The second rise of Hyper-V

Over the past 18 months, I have observed a significant increase in the adoption of Hyper-V. And why not? It is a mature and capable platform built and supported by Microsoft. The latter makes moving to it a less stressful choice as the ecosystem and community are large and well-established. I believe that Hyper-V is one of the primary beneficiaries of the hypervisor turmoil. Adoption is experiencing a second, significant rise. For Veeam, this was not a problem. They have provided excellent Hyper-V support for a long time, and I have been a pleased customer, building some of the best and most performant backup fabrics on our chosen hardware.

But who are those customers adopting Hyper-V? Are they small and medium businesses (SME) or managed service providers? Or is Hyper-V making headway with big corporate enterprises as well? Well, neither Microsoft nor Veeam shares such data with me. So, what do I do? Weak to strong signal intelligence! I observe what companies are doing and what they are saying, in combination with what people ask me directly. That has me convinced that some larger corporations have made the move to Hyper-V. Some of the stronger signals came from Veeam.

Current and future Veeam Releases

Let’s look at the more recent releases of Veeam Backup & Replication. With version 12.3, support for Windows Server 2025 arrived very fast after the general availability of that OS. Hyper-V, by the way, is getting all the improvements and new capabilities for Hyper-V just as much as Azure Local. That indicates Microsoft’s interest in making Hyper-V an excellent option for any customer, regardless of how they choose to run it, be it on local storage, with shared storage, on Storage Spaces Direct (S2D), or Azure Local. That is a strong, positive signal compared to previous statements. Naturally, Hyper-V benefits from Veeam’s ongoing efforts to resolve issues, enhance features, and add capabilities, providing the best possible backup fabric for everyone. I will discuss that in later articles.

Now, the strong signal and very positive signal from Veeam regarding Hyper-V came with updates to Veeam Recovery Orchestrator. Firstly, Veeam Recovery Orchestrator 7.2 (released on February 18th, 2025) introduced support for Hyper-V environments. What does that tell me? The nature, size, and number of customers leveraging Hyper-V that need and are willing to pay for Veeam Recovery Orchestrator have grown to a point where Veeam is willing to invest in developing and supporting it. That is new! On the Product Update page, https://community.veeam.com/product-updates/veeam-recovery-orchestrator-7-2-9827, you can find more information. The one requirement that sticks out is the need for System Center Virtual Machine Manager. Look at these key considerations:

  • System Center Virtual Machine Manager (SCVMM) 2022 & CSV storage registered in SCVMM is supported.
  • Direct connections to Hyper-V hosts are not supported.

But not that much later, on July 9th, 2025,  in Veeam Recovery Orchestrator 7.2.1 (see https://community.veeam.com/product-updates/veeam-recovery-orchestrator-7-2-1-10876), we find these significant enhancements:

  1. Support for Azure Local recovery target: You can now use Azure Local as a recovery target for both vSphere and Hyper-V workloads, expanding flexibility and cloud recovery options.
  2. Hyper-V direct-connected cluster support: Extended Hyper-V functionality enables support for direct-connected clusters, eliminating the need for SCVMM. This move simplifies deployment and management for Hyper-V environments.
  3. MFA integration for VRO UI: Multi-Factor Authentication (MFA) can now be enabled to secure logins to the VRO user interface, providing enhanced security and compliance. Microsoft Authenticator and Google Authenticator apps are supported.

Especially 1 and 2 are essential, as they enable Veeam Recovery Orchestrator to support many more Hyper-V customers. Again, this is a strong signal that Hyper-V is making inroads. Enough so for Veeam to invest. Ironically, we have Broadcom to thank for this. Which is why in November 2024, I nominated Broadcom as the clear and unchallenged winner of the “Top Hyper-V Seller Award 2024” (https://www.linkedin.com/posts/didiervanhoye_broadcom-mvpbuzz-hyperv-activity-7257391073910566912-bTTF/)

Conclusion

Hyper-V and Veeam are a potent combination that continues to evolve as market demands change. Twelve years ago, I was testing out Veeam Backup & Replication, and 6 months later, I became a Veeam customer. I am still convinced that for my needs and those of the environments I support, I have made a great choice.

The longevity of the technology, which evolves in response to customer and security needs, is a key factor in determining great technology choices. In that respect, Hyper-V and Veeam have performed exceptionally well, striking multiple bullseye shots without missing a beat. And missing out on the hypervisor drama, we have hit the bullseye once more!