Veeam Hardened Repositories on DELL R740XD2 Servers

Posted on April 15, 2022 by workinghardinit

Veeam Hardened Repositories on DELL R740XD2 Servers

Recently I got the opportunity to build Veeam Hardened Repositories on DELL R740XD2 Servers. Those repositories were needed to deploy a Veeam Scaleout Backup Repository (SOBR). Yes, Linux systems leveraging XFS for Fast Clone and immutability.

Technologies used

Veeam Backup & Replication

First of all, you need Veeam Backup & Replication (VBR) v11a or later. I run VBR on Windows Server 2022 at the time of writing.

Linux operating system

Next to that, I use Ubuntu 20.04 LTS for the Veeam hardened repositories. While the release of 22.04 LTS imminent at the time of writing this, with OEM hardware support as a requirement I stick to 20.04 for now. The file system is XFS with Fast Cloning enabled.

Servers with Direct Attached Storage (DAS)

I leverage RAID 60 on Dell EMC R740-DX2 servers. The servers boot from mirrored BOSS SSDs and have a PERC H740P raid controller with 8GB of cache and 26 3.5″ HDD attached. We need to optimize for both cost and capacity, therefore we use 3.5″ drives. With disk sizes ranging from 8TB to 16TB, this brings us real-world usable storage from 145TB to 290TB with two global hot spares.

Veeam Hardened Repositories on DELL R740XD2 Servers — DELL R740XD2 (Image courtesy of DELL)

Now the RAID 60 is one big virtual disk group containing all disks bar the 2 assigned as global hot spares. This makes sure we engage all disks to help with IOPS, latency, throughput. The 8Gbps helps smooth things out. Depending on the scale of your deployment you can create one VEEAM SOBR extent per server of carve multiple VDISKS out of the available storage.

You also have the option to leverage LVM on linux if that suits your need, but in that case I do it only for volume management, dat protection comes from the physical RAID controller.

There is not right or wrong way there. Select what suits you best, but the golden rule is to keep it simple.

Networking

Networking is 1Gbps for DRAC/Host connectivity and dual 10Gbps or 25Gbps for backup traffic. The switches are Dell EMC PowerSwitch S52XXF-ON series. Awesome kit!

Extra Security

Finally, I provide some extra security. I use DUO as an MFA provider as they have an excellent pam_duo module. We often use our smartphones for MFA. But there are plenty of use cases for using security keys. For those, I like the FEITIAN biometric models such as the K27 (USB-A )and the K26 (USB-C).

BioPass FIDO2 Biometric Fingerprint USB-A SecurityKey – K27

The Journey of building Veeam Hardened Repositories on DELL R740XD2 Servers

Maybe you are interested in how I set up these Veeam Hardened Repositories on DELL R740XD2 Servers? If so, you might be in luck. I hope to blog about this journey for both my own reference to share the experience. This will take several blog posts and in those, I will highlight different parts of the solution. If you want to learn more about the Veeam hardened Repository I recommend you read my blog series I did last year:

Veeam Hardening Linux Repository – Part 1 | StarWind Blog (starwindsoftware.com)., Veeam Hardening Linux Repository – Part 2 | StarWind Blog (starwindsoftware.com), Veeam Hardening Linux Repository – Part 3 | StarWind Blog (starwindsoftware.com)

As I realize not all of you will get your hands on such hardware I have a PowerShell script that creates Hyper-V virtual machines to use in the lab and practice with. Those VMs emulate the DELL hardware setup.

Create virtual machines for a Veeam hardened repository lab

Posted on April 12, 2022 by workinghardinit

Introduction

In this blog post, I will give you a script to create virtual machines for a Veeam hardened repository lab.

Some of you have asked me to do some knowledge transfer about configuring a Veeam hardened repository. For lab work virtualization is your friend. I hope to show you some of the Ubuntu Linux configurations I do. When time permits I will blog about this and you can follow along. I will share what I can on my blog.

Running the script

Now, if you have Hyper-V running on a lab node or on your desktop or laptop you can create virtual machines for a Veeam hardened repository lab with the PowerShell script below. Just adjust the parameters and make sure you have the Ubuntu 20.04 Server ISO in the right place. The script creates the virtual machine configuration files under a folder with the name of the virtual machine in the path you specify in the variables The VM it creates will boot into the Ubuntu setup and we can walk through it and configure it.

Pay attention to the -version of the virtual machine. I run Windows Server 2022 and Windows 11 on my PCs so you might need to adjust that to a version your Hyper-V installation supports.

Also, pay attention to the VLAN IDs used. That suits my lab network. It might not suit yours. Use VLAN ID 0 to disable the VLAN identifier on a NIC.

Clear-Host
$VMPrefix = 'AAAA-XFSREPO-0'
$Path = "D:\VirtualMachines\"
$ISOPath = 'D:\VirtualMachines\ISO\ubuntu-20.04.4-live-server-amd64.iso'
$NumberOfCPUs = 2
$Memory = 4GB
$vSwitch = 'DataWiseTech'
$NumberOfVMs = 2
$VlanIdTeam = 2
$VlanIDSMB1 = 40
$VlanIdSMB2 = 50
$VmVersion = '10.0'

ForEach ($Counter in 1..$NumberOfVMs) {
    $VMName = $VMPrefix + $Counter
    $DataDisk01Path = "$Path$VMName\Virtual Hard Disks\$VMName-DATA01.vhdx"
    $DataDisk02Path = "$Path$VMName\Virtual Hard Disks\$VMName-DATA02.vhdx"
    Write-Host -ForegroundColor Cyan "Creating VM $VMName in $Path ..."
    New-VM -Name $VMName -path $Path -NewVHDPath "$Path$VMName\Virtual Hard Disks\$VMName-OS.vhdx" `
        -NewVHDSizeBytes 65GB -Version 10.0 -Generation 2 -MemoryStartupBytes $Memory -SwitchName $vSwitch| out-null

    Write-Host -ForegroundColor Cyan "Setting VM $VMName its number of CPUs to $NumberOfCPUs ..."
    Set-VMProcessor –VMName $VMName –count 2

    Write-Host -ForegroundColor Magenta "Adding NICs LAN-HOST01, LAN-HOST02, SMB1 and SMB2 to $VMName"
    #Remove-VMNetworkAdapter -VMName $VMName -Name 'Network Adapter'

    Rename-VMNetworkAdapter -VMName $VMName -Name 'Network Adapter' -NewName LAN-HOST-01
    #Connect-VMNetworkAdapter -VMName $VMName -Name LAN -SwitchName $vSwitch
    Add-VMNetworkAdapter -VMName $VMName -SwitchName DataWiseTech -Name LAN-HOST-02 -DeviceNaming On
    Add-VMNetworkAdapter -VMName $VMName -SwitchName $vSwitch -Name SMB1 -DeviceNaming On
    Add-VMNetworkAdapter -VMName $VMName -SwitchName $vSwitch -Name SMB2 -DeviceNaming On
    
    Write-Host -ForegroundColor Magenta "Assigning VLANs to NICs LAN-HOST01, LAN-HOST02, SMB1 and SMB2 to $VMName"
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName LAN-HOST-01 -Access -VLANId $VlanIdTeam
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName LAN-HOST-02 -Access -VLANId $VlanIdTeam  
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName SMB1 -Access -VLANId $VlanIdSMB1
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName SMB2 -Access -VLANId $VlanIdSmb2

    Set-VMNetworkAdapter -VMName $VMName -Name LAN-HOST-01 -DhcpGuard On -RouterGuard On -DeviceNaming On -MacAddressSpoofing On -AllowTeaming On
    Set-VMNetworkAdapter -VMName $VMName -Name LAN-HOST-02 -DhcpGuard On -RouterGuard On -MacAddressSpoofing On -AllowTeaming On
    Set-VMNetworkAdapter -VMName $VMName -Name SMB1 -DhcpGuard On -RouterGuard On -MacAddressSpoofing Off -AllowTeaming off
    Set-VMNetworkAdapter -VMName $VMName -Name SMB2 -DhcpGuard On -RouterGuard On -MacAddressSpoofing Off -AllowTeaming off

    Write-Host -ForegroundColor yellow "Adding DVD Drive to $VMName"
    Add-VMDvdDrive -VMName $VMName -ControllerNumber 0 -ControllerLocation 8 

    Write-Host -ForegroundColor yellow "Mounting $ISOPath to DVD Drive on $VMName"
    Set-VMDvdDrive -VMName $VMName -Path $ISOPath

    Write-Host -ForegroundColor White "Setting DVD with $ISOPath as first boot device on $VMName"
    $DVDWithOurISO = ((Get-VMFirmware -VMName $VMName).BootOrder | Where-Object Device -like *DVD*).Device
    
    Set-VMFirmware -VMName $VMName -FirstBootDevice $DVDWithOurISO `
    -EnableSecureBoot On -SecureBootTemplate MicrosoftUEFICertificateAuthority

    Write-Host -ForegroundColor Cyan "Creating two data disks and adding them to $VMName"
    New-VHD -Path $DataDisk01Path -Dynamic -SizeBytes 150GB | out-null
    New-VHD -Path $DataDisk02Path -Dynamic -SizeBytes 150GB | out-null

    Add-VMHardDiskDrive -VMName $VMName -ControllerNumber 0 `
    -ControllerLocation 1 -ControllerType SCSI  -Path $DataDisk01Path

    Add-VMHardDiskDrive -VMName $VMName -ControllerNumber 0 `
    -ControllerLocation 2 -ControllerType SCSI  -Path $DataDisk02Path

    $VM = Get-VM $VMName 
    write-Host "VM $VM  has been created" -ForegroundColor green
    write-Host ""
}

Conclusion

In conclusion, that’s it for now. Play with the script and you will create virtual machines for a Veeam hardened repository lab in no time. That way you are ready to test and educate yourself. Don’t forget that you need to have sufficient resources on your host. Virtualization is cool but it is not magic.

Some of the settings won’t make sense to some of you, but during the future post, this will become clear. These are specific to Ubuntu networking on Hyper-V.

I hope to publish the steps I take in the coming months. As with many, time is my limiting factor so have patience. In the meanwhile, you read up about the Veeam hardened repository.

Response and feedback about the Veeam hardened repository presentation

Posted on March 24, 2021 by workinghardinit

Veeam hardened repository presentation

It is great to get so much response and feedback about the Veeam hardened repository presentation. So, first of all, I thank all those reaching out to me in regards to the TechNine virtual user group session I gave on this subject.

Most of you did so via the contact e-mail on my blog and not via a comment on the original blog post here. Many of you asked for a recording. Let’s address that first.

Where is the recording?

There is no recording. The TechNine user group tries to bring people together for the events and promote interactive discussions afterward. Hence, no recording is available. They do this to stimulate participation.

I did not make this presentation exclusively for TechNine, but they did get the world premiere. The good news is that I will be giving this session again and I will even be doing a webcast with my fellow Microsoft MVP and Veeam Vanguard Carsten Rachfahl about this. In the webcast, we will discuss the hardened repository at length and it will be published online.

So, nothing to worry about, you will get other opportunities to attend and you will have a recoding of the content reasonably soon.

Reaching out to me

Lately, I have noticed that my readers and social media followers seem to have gotten increasingly shy and do not ask their questions publicly via the blog comments section or social media.

That poses a challenge to me. While I would like to help you all individually, that approach just doesn’t scale. I have a job, family, life, and interest to pursue. I just cannot allocate the time to do so.

Veeam hardened repository presentation — I do explain this on my contact page!

Please ask you questions in the comments section of the relevant blog post and I will normally get to it. The benefit is that that public answer can help others as well with limited effort from myself. If the answer to a super interesting question is lengthy, I can decide to turn it into a blog post. That also helps all people out there. Thank you for being respectful of my time and sharing with the community yourself!

The Veeam hardened repository at the TechNine User group

Posted on March 9, 2021 by workinghardinit

The Veeam hardened repository

Just a quick blog post to let you all know I will be presenting on the Veeam hardened repository in Veeam Backup & Replication v11. I will do this at the TechNine user group meeting on March the 23rd. It is the world premiere of this presentation, so grab your seat and be able to tell your grandchildren you were there!

There is a tremendous amount of new and enhanced capabilities in v11, both small and big ones. But this one stands out because no matter who you are, a small business or a fortune 500 global enterprise, we are all at risk from ransomware and wipers. And guess what, the Veeam hardened repository gives you yet another option in your arsenal to make sure you can recover from such a disaster.

And end to end solution

In VBR v11 Veeam now offers complete end to end immutability for all your backup copies if you desire to do so. You can bring your own storage and so it leaves you the flexibility and freedom of choice you are used to with Veeam.

Like many of you, I like to stay up to date with new technology and implement it to solve real-world problems. Often coo tech is very forward-looking and foundational. This translates into “it is very early days yet”, “the code exists but you cannot use it yet”.

Well, this is not something that shows you the promises of tomorrow and then leaves you to go back empty-handed. That always feels like a bit of a letdown when you are dealing with your real-world, real-time issues. The Veeam hardened repository is available now to help address the challenges you are facing today! It doesn’t get any better than that. I will show you the cookie, you can have the cookie and you can eat the cookie! Want to play with it yourself? You can, right now with the free community edition of Veeam Backup & Replication v11!

Register!

What do you need to do? Register of cause! Do it right here. Make sure you show up in time and stay around for an interactive discussion afterward. We start at 20:00 on March 23rd, 2021. And yes it is all virtual as we plan to get rid of the coronavirus first and then get together in real life again.

I will explain why you need it, how to set it up, and I will dive under the hood to show how it gets the job done. You can now protect your hypervisor and supported backups end to end with immutability in the hardened repository, giving you the extra security and protection you might need to recovery from a ransomware event. There will be demos! You can read up on it here but I will go behind the scenes with you!

Working Hard In IT

My view on IT from the trenches

Tag Archives: Hardened Repository

Veeam Hardened Repositories on DELL R740XD2 Servers