Veeam Hardened Repositories on DELL R740XD2 Servers

Veeam Hardened Repositories on DELL R740XD2 Servers

Recently I got the opportunity to build Veeam Hardened Repositories on DELL R740XD2 Servers. Those repositories were needed to deploy a Veeam Scaleout Backup Repository (SOBR). Yes, Linux systems leveraging XFS for Fast Clone and immutability.

Technologies used

Veeam Backup & Replication

First of all, you need Veeam Backup & Replication (VBR) v11a or later. I run VBR on Windows Server 2022 at the time of writing.

Linux operating system

Next to that, I use Ubuntu 20.04 LTS for the Veeam hardened repositories. While the release of 22.04 LTS imminent at the time of writing this, with OEM hardware support as a requirement I stick to 20.04 for now. The file system is XFS with Fast Cloning enabled.

Servers with Direct Attached Storage (DAS)

I leverage RAID 60 on Dell EMC R740-DX2 servers. The servers boot from mirrored BOSS SSDs and have a PERC H740P raid controller with 8GB of cache and 26 3.5″ HDD attached. We need to optimize for both cost and capacity, therefore we use 3.5″ drives. With disk sizes ranging from 8TB to 16TB, this brings us real-world usable storage from 145TB to 290TB with two global hot spares.

Veeam Hardened Repositories on DELL R740XD2 Servers
DELL R740XD2 (Image courtesy of DELL)

Now the RAID 60 is one big virtual disk group containing all disks bar the 2 assigned as global hot spares. This makes sure we engage all disks to help with IOPS, latency, throughput. The 8Gbps helps smooth things out. Depending on the scale of your deployment you can create one VEEAM SOBR extent per server of carve multiple VDISKS out of the available storage.

You also have the option to leverage LVM on linux if that suits your need, but in that case I do it only for volume management, dat protection comes from the physical RAID controller.

There is not right or wrong way there. Select what suits you best, but the golden rule is to keep it simple.

Networking

Networking is 1Gbps for DRAC/Host connectivity and dual 10Gbps or 25Gbps for backup traffic. The switches are Dell EMC PowerSwitch S52XXF-ON series. Awesome kit!

Veeam Hardened Repositories on DELL R740XD2 Servers
The DELL S5224F-ON as TOR swithes provide 25Gbps for the backup traffic (Image courtesy of DELL)

Extra Security

Finally, I provide some extra security. I use DUO as an MFA provider as they have an excellent pam_duo module. We often use our smartphones for MFA. But there are plenty of use cases for using security keys. For those, I like the FEITIAN biometric models such as the K27 (USB-A )and the K26 (USB-C).

BioPass FIDO2 Biometric Fingerprint USB-A SecurityKey – K27
BioPass FIDO2 Biometric Fingerprint USB-A SecurityKey – K27

The Journey of building Veeam Hardened Repositories on DELL R740XD2 Servers

Maybe you are interested in how I set up these Veeam Hardened Repositories on DELL R740XD2 Servers? If so, you might be in luck. I hope to blog about this journey for both my own reference to share the experience. This will take several blog posts and in those, I will highlight different parts of the solution. If you want to learn more about the Veeam hardened Repository I recommend you read my blog series I did last year:

Veeam Hardening Linux Repository – Part 1 | StarWind Blog (starwindsoftware.com)., Veeam Hardening Linux Repository – Part 2 | StarWind Blog (starwindsoftware.com), Veeam Hardening Linux Repository – Part 3 | StarWind Blog (starwindsoftware.com)

As I realize not all of you will get your hands on such hardware I have a PowerShell script that creates Hyper-V virtual machines to use in the lab and practice with. Those VMs emulate the DELL hardware setup.

Create virtual machines for a Veeam hardened repository lab

Introduction

In this blog post, I will give you a script to create virtual machines for a Veeam hardened repository lab.

Create virtual machines for a Veeam hardened repository lab
The script has just created two virtual machines for you

Some of you have asked me to do some knowledge transfer about configuring a Veeam hardened repository. For lab work virtualization is your friend. I hope to show you some of the Ubuntu Linux configurations I do. When time permits I will blog about this and you can follow along. I will share what I can on my blog.

Running the script

Now, if you have Hyper-V running on a lab node or on your desktop or laptop you can create virtual machines for a Veeam hardened repository lab with the PowerShell script below. Just adjust the parameters and make sure you have the Ubuntu 20.04 Server ISO in the right place. The script creates the virtual machine configuration files under a folder with the name of the virtual machine in the path you specify in the variables The VM it creates will boot into the Ubuntu setup and we can walk through it and configure it.

Pay attention to the -version of the virtual machine. I run Windows Server 2022 and Windows 11 on my PCs so you might need to adjust that to a version your Hyper-V installation supports.

Also, pay attention to the VLAN IDs used. That suits my lab network. It might not suit yours. Use VLAN ID 0 to disable the VLAN identifier on a NIC.

Clear-Host
$VMPrefix = 'AAAA-XFSREPO-0'
$Path = "D:\VirtualMachines\"
$ISOPath = 'D:\VirtualMachines\ISO\ubuntu-20.04.4-live-server-amd64.iso'
$NumberOfCPUs = 2
$Memory = 4GB
$vSwitch = 'DataWiseTech'
$NumberOfVMs = 2
$VlanIdTeam = 2
$VlanIDSMB1 = 40
$VlanIdSMB2 = 50
$VmVersion = '10.0'

ForEach ($Counter in 1..$NumberOfVMs) {
    $VMName = $VMPrefix + $Counter
    $DataDisk01Path = "$Path$VMName\Virtual Hard Disks\$VMName-DATA01.vhdx"
    $DataDisk02Path = "$Path$VMName\Virtual Hard Disks\$VMName-DATA02.vhdx"
    Write-Host -ForegroundColor Cyan "Creating VM $VMName in $Path ..."
    New-VM -Name $VMName -path $Path -NewVHDPath "$Path$VMName\Virtual Hard Disks\$VMName-OS.vhdx" `
        -NewVHDSizeBytes 65GB -Version 10.0 -Generation 2 -MemoryStartupBytes $Memory -SwitchName $vSwitch| out-null

    Write-Host -ForegroundColor Cyan "Setting VM $VMName its number of CPUs to $NumberOfCPUs ..."
    Set-VMProcessor –VMName $VMName –count 2

    Write-Host -ForegroundColor Magenta "Adding NICs LAN-HOST01, LAN-HOST02, SMB1 and SMB2 to $VMName"
    #Remove-VMNetworkAdapter -VMName $VMName -Name 'Network Adapter'

    Rename-VMNetworkAdapter -VMName $VMName -Name 'Network Adapter' -NewName LAN-HOST-01
    #Connect-VMNetworkAdapter -VMName $VMName -Name LAN -SwitchName $vSwitch
    Add-VMNetworkAdapter -VMName $VMName -SwitchName DataWiseTech -Name LAN-HOST-02 -DeviceNaming On
    Add-VMNetworkAdapter -VMName $VMName -SwitchName $vSwitch -Name SMB1 -DeviceNaming On
    Add-VMNetworkAdapter -VMName $VMName -SwitchName $vSwitch -Name SMB2 -DeviceNaming On
    
    Write-Host -ForegroundColor Magenta "Assigning VLANs to NICs LAN-HOST01, LAN-HOST02, SMB1 and SMB2 to $VMName"
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName LAN-HOST-01 -Access -VLANId $VlanIdTeam
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName LAN-HOST-02 -Access -VLANId $VlanIdTeam  
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName SMB1 -Access -VLANId $VlanIdSMB1
    Set-VMNetworkAdapterVlan -VMName $VMName -VMNetworkAdapterName SMB2 -Access -VLANId $VlanIdSmb2

    Set-VMNetworkAdapter -VMName $VMName -Name LAN-HOST-01 -DhcpGuard On -RouterGuard On -DeviceNaming On -MacAddressSpoofing On -AllowTeaming On
    Set-VMNetworkAdapter -VMName $VMName -Name LAN-HOST-02 -DhcpGuard On -RouterGuard On -MacAddressSpoofing On -AllowTeaming On
    Set-VMNetworkAdapter -VMName $VMName -Name SMB1 -DhcpGuard On -RouterGuard On -MacAddressSpoofing Off -AllowTeaming off
    Set-VMNetworkAdapter -VMName $VMName -Name SMB2 -DhcpGuard On -RouterGuard On -MacAddressSpoofing Off -AllowTeaming off

    Write-Host -ForegroundColor yellow "Adding DVD Drive to $VMName"
    Add-VMDvdDrive -VMName $VMName -ControllerNumber 0 -ControllerLocation 8 

    Write-Host -ForegroundColor yellow "Mounting $ISOPath to DVD Drive on $VMName"
    Set-VMDvdDrive -VMName $VMName -Path $ISOPath

    Write-Host -ForegroundColor White "Setting DVD with $ISOPath as first boot device on $VMName"
    $DVDWithOurISO = ((Get-VMFirmware -VMName $VMName).BootOrder | Where-Object Device -like *DVD*).Device
    
    Set-VMFirmware -VMName $VMName -FirstBootDevice $DVDWithOurISO `
    -EnableSecureBoot On -SecureBootTemplate MicrosoftUEFICertificateAuthority

    Write-Host -ForegroundColor Cyan "Creating two data disks and adding them to $VMName"
    New-VHD -Path $DataDisk01Path -Dynamic -SizeBytes 150GB | out-null
    New-VHD -Path $DataDisk02Path -Dynamic -SizeBytes 150GB | out-null

    Add-VMHardDiskDrive -VMName $VMName -ControllerNumber 0 `
    -ControllerLocation 1 -ControllerType SCSI  -Path $DataDisk01Path

    Add-VMHardDiskDrive -VMName $VMName -ControllerNumber 0 `
    -ControllerLocation 2 -ControllerType SCSI  -Path $DataDisk02Path

    $VM = Get-VM $VMName 
    write-Host "VM $VM  has been created" -ForegroundColor green
    write-Host ""
}

Conclusion

In conclusion, that’s it for now. Play with the script and you will create virtual machines for a Veeam hardened repository lab in no time. That way you are ready to test and educate yourself. Don’t forget that you need to have sufficient resources on your host. Virtualization is cool but it is not magic.

Some of the settings won’t make sense to some of you, but during the future post, this will become clear. These are specific to Ubuntu networking on Hyper-V.

I hope to publish the steps I take in the coming months. As with many, time is my limiting factor so have patience. In the meanwhile, you read up about the Veeam hardened repository.

Attending VeeamON 2022

I am attending VeeamON 2022

Yes, I am attending VeeamON 2022. So should you! I mean it. Data protection is becoming ever more diverse. That means you need to keep up and invest in yourself. That is what I do, nearly every day. I want to, I need to and I like to do so.

Attending VeeamON 2022
https://www.veeam.com/veeamon

The landscape has fragmented due to locations such as on-prem, hybrid, cloud, service models like IaaS, PaaS, SaaS, and technologies such as virtual machines and containers. And that is only scratching the surface of the challenges we face while protecting our data. We often have a wide mix of the above and technology trends evolve fast.

Keep learning

The key is to keep learning. That takes a never-ending commitment and effort. We learn in many different ways and VeeamON caters to all of them. Theory, practical guidance, hands-on labs, exams, interaction with peers and industry experts. You name it, VeeamON has it!

Education

Education leads towards a better understanding so you can analyze challenges, design solutions, see relations, and understand dependencies. It is acquiring knowledge that is used to become better at your job by using the technologies optionally in the ecosystem where they need to deliver their value.

Training

Training is getting ready to deploy and operate solutions. It is very focused on specific jobs at hand. That doesn’t necessarily make it easy. On the contrary, it can be difficult but well-trained people can make the hard look easy and look smart because the knowledge and skill have been drilled into them. It is that simple, but again, that doesn’t make it easy.

Networking

Exchanging ideas, experiences, solutions, techniques with others helps us all learn and grow. It builds professional relations that source the common brain of the community so everyone gains. It helps your clients, your employers, your colleagues, and yourself grow and learn. That is good for your job, your career, or your business, whichever it happens to be in your case.

Join me in attending VeeamOn 2022

Join me and the excellent crew Veeam is bringing to bear at VeeamON 2022. You can join online or in-person in Las Vegas. Register here! Online is free, bar the investment of your time. But trust me, you are not a second-class citizen, it is a real and valuable conference. If you are attending in person, be ready for an immersive experience!

Las Vegas – oh my, I want to attend VeeamON 2022 and go on a long road trip after.

I would love to go in person and enjoy the immersion in a world of expertise and learning at the conference, but alas, it will not be this year. If you can attend, do whatever it takes to convince your boss or yourself, it is a rich and rewarding experience, that pays itself back in no time. If you can’t make it, don’t despair, join online like I will. Know that there will be other chances to attend and if the boss is the biggest issue, there are better bosses out there ;-).

I made Veeam Vanguard 2022!

This morning I got an e-mail informing me that my 2022 Veeam Vanguard Renewal Nomination has been approved. Yes, I made Veeam Vanguard 2022! That’s is awesome news. While for the past two years, with Corona and Covid-19, community life has been different, it has most certainly not withered away. Quite the contrary, and the Veeam Vangaurd program has not been different. First, Veeam has made a great effort to support the community through all this. Secondly, the one in-person/hybrid event that was possible during all this (the Veeam Vanguard Summit 2021 in Prague) their care for us and our health was exemplary.

But that is not what being a Veeam Vanguard is all about, it does show that the community works to bring people together. Working, learing, sharing together builds community. And with the pandemic hopefully subsiding we will not have missed a single beat.

I am a Veeam Vanguard 2021
I am a Veeam Vanguard once more in 2022!

The Veeam Vanguard Program

I invite you to read all about the Veeam Vanguard Program here. If you are working with Veeam solution in any way, shape or form and like to share your knowledge it might be something for you! Veeam Vanguards come from all walks of life and backgrounds. What unites them in this community is their expertise in different verticals and in different disciplines. They all contribute to the community at large in different ways.

Honored, happy and humbled

As is often the case with awards and recognition it makes honored to be recognized, proud while also feeling humble. You must realize we all stand on the shoulders of giants. In the process of becoming an expert there is always the realization of how much we don’t know. Add in some “imposter syndrome” and there is no room for hubris. There is however joy of being part of the Veeam Vanguard programs. It means having the opportunity to spend my time learning along people that are the best in the business. Thank you Veeam and a big shout out the Rick, Nikola, Kseniya and the entire Product Strategy Team at Veeam for making this possible.