Veeam Hardened Repositories on DELL R740XD2 Servers

Veeam Hardened Repositories on DELL R740XD2 Servers

Recently I got the opportunity to build Veeam Hardened Repositories on DELL R740XD2 Servers. Those repositories were needed to deploy a Veeam Scaleout Backup Repository (SOBR). Yes, Linux systems leveraging XFS for Fast Clone and immutability.

Technologies used

Veeam Backup & Replication

First of all, you need Veeam Backup & Replication (VBR) v11a or later. I run VBR on Windows Server 2022 at the time of writing.

Linux operating system

Next to that, I use Ubuntu 20.04 LTS for the Veeam hardened repositories. While the release of 22.04 LTS imminent at the time of writing this, with OEM hardware support as a requirement I stick to 20.04 for now. The file system is XFS with Fast Cloning enabled.

Servers with Direct Attached Storage (DAS)

I leverage RAID 60 on Dell EMC R740-DX2 servers. The servers boot from mirrored BOSS SSDs and have a PERC H740P raid controller with 8GB of cache and 26 3.5″ HDD attached. We need to optimize for both cost and capacity, therefore we use 3.5″ drives. With disk sizes ranging from 8TB to 16TB, this brings us real-world usable storage from 145TB to 290TB with two global hot spares.

Veeam Hardened Repositories on DELL R740XD2 Servers
DELL R740XD2 (Image courtesy of DELL)

Now the RAID 60 is one big virtual disk group containing all disks bar the 2 assigned as global hot spares. This makes sure we engage all disks to help with IOPS, latency, throughput. The 8Gbps helps smooth things out. Depending on the scale of your deployment you can create one VEEAM SOBR extent per server of carve multiple VDISKS out of the available storage.

You also have the option to leverage LVM on linux if that suits your need, but in that case I do it only for volume management, dat protection comes from the physical RAID controller.

There is not right or wrong way there. Select what suits you best, but the golden rule is to keep it simple.

Networking

Networking is 1Gbps for DRAC/Host connectivity and dual 10Gbps or 25Gbps for backup traffic. The switches are Dell EMC PowerSwitch S52XXF-ON series. Awesome kit!

Veeam Hardened Repositories on DELL R740XD2 Servers
The DELL S5224F-ON as TOR swithes provide 25Gbps for the backup traffic (Image courtesy of DELL)

Extra Security

Finally, I provide some extra security. I use DUO as an MFA provider as they have an excellent pam_duo module. We often use our smartphones for MFA. But there are plenty of use cases for using security keys. For those, I like the FEITIAN biometric models such as the K27 (USB-A )and the K26 (USB-C).

BioPass FIDO2 Biometric Fingerprint USB-A SecurityKey – K27
BioPass FIDO2 Biometric Fingerprint USB-A SecurityKey – K27

The Journey of building Veeam Hardened Repositories on DELL R740XD2 Servers

Maybe you are interested in how I set up these Veeam Hardened Repositories on DELL R740XD2 Servers? If so, you might be in luck. I hope to blog about this journey for both my own reference to share the experience. This will take several blog posts and in those, I will highlight different parts of the solution. If you want to learn more about the Veeam hardened Repository I recommend you read my blog series I did last year:

Veeam Hardening Linux Repository – Part 1 | StarWind Blog (starwindsoftware.com)., Veeam Hardening Linux Repository – Part 2 | StarWind Blog (starwindsoftware.com), Veeam Hardening Linux Repository – Part 3 | StarWind Blog (starwindsoftware.com)

As I realize not all of you will get your hands on such hardware I have a PowerShell script that creates Hyper-V virtual machines to use in the lab and practice with. Those VMs emulate the DELL hardware setup.

Your organization does not allow you to add your account to Microsoft Authenticator

Your organization does not allow you to add your account to Microsoft Authenticator

I was testing a bunch of scenarios with passwordless authentication in Azure Active Directory on a weekend. Things were looking good. I created some test accounts and played with a bunch of permutations to see how things behaved, Think about Conditional Access policies in combination with authentication methods, etc. The aim was to have multiple passwordless authentication options per user for redundancy. On top of that, I want to have this for multiple accounts (separation of duties). That latter requirement tripped me up.

I succeeded at most of my goals. But at one moment I received the following error trying to register the Microsoft Authenticator app on my phone for one of my test users. Warning “Account not added” and the message “Your organization does not allow you to add your account to Microsoft Authenticator” What’s going on here?

Your organization does not allow you to add your account to Microsoft Authenticator

Passwordless sign-in with the Microsoft Authenticator app

First of all, before you can create this new strong credential, there are prerequisites. One prerequisite is that you must register the device on which you installed the Microsoft Authenticator app within the Azure AD tenant to an individual user. In that requirement lies the answer to our error message.

Set up phone sign-in
Device Registration and Set screen lock
Your organization does not allow you to add your account to Microsoft Authenticator
I already have this device registered for another account

Currently, you can only register a device in a single tenant. This means you can enable only one work or school account in the Microsoft Authenticator app for passwordless sign-in.

So we can only use our smartphone with the Microsoft Authenticator app in a single-tenant, with a single user. And that’s why I got the error. I already had another test user on that phone set up sign-in without a password. My device is already registered for another user in Azure AD in that tenant. There can be only one.

Do note that you can still use the authenticator app as an MFA method with your password. It is the passwordless scenario that doesn’t work under these conditions.

Achieving my goals

This is annoying when testing but it can also be annoying in real life. I tend to have multiple accounts in an Azure AD. I log in with a different account depending on what work I need to do and what roles/rights this requires. That’s why I like FIDO2 security keys with biometrics as a passwordless option.

What I need is a passwordless solution I can use with multiple accounts in the same and other tenants. That, I can do this with my FIDO2 BioPass security key from FEITIAN just fine. I can register my security key with multiple accounts and be on my way. With one smartphone with the Microsoft Authenticator app installed, you cannot add multiple accounts on the phone for passwordless authentication (device registration) at this moment in time. That’s what the error message means to tell you but the wording confused me for a while.

Configure a FEITIAN FIDO2 BioPass security key

Introduction

In this blog post, we will configure a FEITIAN FIDO2 BioPass security key. As you might know from a previous blog post, FEITIAN FIDO2 security keys – Working Hard In ITWorking Hard In IT, I have a FEITIAN FIDO2 BioPass security key. That’s the one I use to test scenarios in the lab. What I write here will work with any biometric security key with the native tools. Only the vendor-specific tools will differ.

Figure 1: The FEITIAN BioPass FIDO2 security key

Before you can use your FEITIAN security key, we need to set it up. That is a pretty straightforward process. You can use the native Windows 10 or Windows 11 tools or download the tool the FIDO2 vendor provides. Both work perfectly well, but the vendor tool often offers more capabilities.

Initial configuration

Below I will show you how to use the Windows 10 native built-in tool to configure a FEITIAN FIDO2 BioPass security key. That tool is available on Windows 19H1 and onwards and in Windows 11. After that, I will peek at the FEITIAN tool (see 未标题-1 (ftsafe.com)). You can download the FEITIAN tool from the Microsoft Store.

You can also use the built-in Chrome tools to do this, which is not only applicable on an Apple or Linux device but also on a Windows Server 2019 OS with the desktop experience where the built-in tool is not available.

Figure 2: Chrome can manage security keys

In Windows Server 2022, however, you do have built-in tools available. You can find it in Settings under Accounts, Sign-in options,  Security Key.

Figure 3: Windows Server 2022 allows you to manage security keys with a built-in tool

Windows 10 built-in tool

But let’s run over how to do this with the built-in tool in Windows 10 or Windows 11.

Type “Window Security” in the search bar and click on “Windows Security.”

In the left pane, click on “Account Protection.”

Figure 4: Under Account Protection, Windows Hello, you can manage your sign-in options

Under “Windows Hello,” click on “Manage sign-in options.”

Figure 5: Manage your security keys

Depending on what type of client you have and if you have Windows Hello capable devices (camera, fingerprint reader, and such) and a supported environment for it, specific options will be available or not. For our use case, a FIDO2 security key, we are interested in the bottom one, “Security Key.” Select it and click on the “Manage” button.

If you have not yet inserted your security key, the tool will ask you to do so. That screen will go away once you have inserted your security key. If you have already inserted it, you will not see this screen.

Figure 6: You really do have to plug in your security key

It will ask you to touch your security key. Don’t worry; this does not require your fingerprints yet. Which is logical as they are not there yet.

Figure 7:Touch your security key when asked

First of all, we now need to add the Security Key PIN to your security key when it is a new one or one that you have reset. That PIN helps secure your key from undesired use and can be anything between 4 and 64 characters.

Figure 8: An unconfigured security key requires you to add a PIN first

A note about your PIN

Note that a PIN does not have to be limited to 4 digits; it can be more. You can use numbers, letters, special characters, etc. That means that “My1stPIN!” is acceptable. But please do not use “1111”. You get the idea. Use something sensible and reasonably secure. It is worth noting that it will be locked out if you type in your key incorrectly too many times. You’ll need to reset your security key, which causes a bit of a hassle, as you can imagine. So choose wisely and make it something you can remember and type in correctly quickly. Also, it is wise to have a backup MFA device (smartphone, 2nd security key).

Figure 9: Enter a  sensible PIN

After you have set the PIN, you can enter one or more fingerprints. For example, I usually register two fingerprints of both hands. That makes sure I can log in when my thumb is in a band-aid and helps easily access the security key when I plug it into either my client device’s right or left side.

Figure 10: It takes 4 recordings to store your fingerprint.
Figure 11: Tab that security key gently 4 times
Figure 12: Success, your fingerprint is registered.

You can register different fingerprints now or come back and add them later.

Figure 13: Done!. Close the setup tool.

That’s it. The FEITIAN FIDO2 security key is now ready for use with any service that supports it.

FEITIAN BioPass FIDO2 Manager

If you have downloaded the FEITIAN BioPass FIDO2 Manager tool from the Microsoft store, you will find it works similarly but with some extra advantages.

Figure 14: BioPass FIDO2 Manager from the Microsoft Store

For one, you can easily list the fingerprints and test them. That way, you can figure out which ones to delete or replace. Quite handy, but it gets even better!

Figure 15: Give your fingerprints a more straightforward name

The cool thing is that you can double-click the fingerprint entries and rename them. That comes in very handy! However, I want to see this capability natively in the Windows built-in tool as well.

I will show you how to use it with your personal Microsoft account and Twitter in a subsequent blog post. With Microsoft, you have the option of going 100% passwordless, and you can delete your password. With other services, this is not always possible yet. But don’t worry, having MFA is helping you avoid 99,9% of unauthorized access to your accounts already. Why? Because most “hacks” of your identity are not real hacks or breaking into the service, but people logging into your account via stolen, derived, or guessed passwords. MFA blocks that.

FEITIAN FIDO2 security keys

FEITIAN FIDO2 security keys

I requested a lab trial sample of some FEITIAN FIDO2 security keys as they offered them to interested and qualifying parties for testing purposes. I was interested in their biometric security keys. So I reached out to see if I qualified, and they sent me two securities for testing in the lab. One is the K26 BioPass FIDO2® with a USB-C interface, which has, you guessed it, biometrics, meaning fingerprints.  The other one is the iePass FIDO® with both a  USB-C and lightning interface. This one has no biometrics but works with touch and makes a good choice for Apple devices. Now, the focus for these security keys is most often professional use cases. Still, I also wanted to point out that you can leverage a security key for your personal online accounts.

FEITIAN FIDO2 security keys
Figure 1: FEITIAN security keys

Yes, you as an individual should also be serious about protecting your online presence. For many of us, if not most, our smartphone is the primary MFA device we use. But I am also interested in an alternative.

FEITIAN

FEITIAN has a wide range of FIDO2 security keys for the many different needs and budgets out there. For me, biometrics is a must for the best possible security. However, they also offer other models, including versatile FEITIAN FIDO2 security keys that offer multiple interfaces like USB, NFC, and Bluetooth. That makes them more widely employable, but as said, I am focusing my efforts on biometric capable ones.

When it comes to biometrics, FEITIAN is the first to offer me that capability. Hence they caught my interest. In addition, ad far as I know, FEITIAN was the first vendor to achieve the FIDO Biometric Component Certification on April 29, 2021.  That is a requirement to qualify for FIDO Level 3 and higher Certification. Next to that, they are very responsive to my communications and feedback. So far, so good! I  know of one other vendor that has biometric FIDO2 keys available, that’s TrustKey. I know Yubikey has had them coming but so far they are not available.

Why a FIDO2 security key?

I usually use a smartphone as my primary MFA tool. A smartphone offers push notification MFA challenges that are easy to approve, allows TOTP code to access services, and can receive single-use passcodes via SMS or e-mail. On top of that, a good smartphone is fingerprint protected. That is a lot of flexibility on a single device most of us carry around daily anyway.

Redundancy for your smartphone

You can use a second smartphone for redundancy, but I use a FIDO2 security dongle where possible. I hang it on my key chain, and even when I merely forgot my phone that day, I have the security keys as a second option handy. That is far more likely to work than counting on having that second phone in my pocket.

So why use a FIDO2 security key? Well, firstly, when using MFA, and most certainly when going ultimately passwordless, you need a second way of accessing your account. I don’t just mean your emergency recovery key or such, but a backup device to answer your MFA challenges. I do this just in case you forget or lose or damage your smartphone or FIDO2 security key and can’t wait for all that to be fixed, replaced, and otherwise handled.

FIDO2 keys as primary and backup MFA solution

Secondly, sometimes FIDO2 security keys are the primary choice, depending on the variety of organizations’ needs, processes, and approaches to dealing with MFA.

Biometrics

I prefer security keys with biometrics. Together with their PIN code, the registered fingerprints provide a device that remains very secure, even when lost. It is also still secure when you leave the security key on your device. Merely touching it is not enough. The fingerprint needs to match even when your device or service does not prompt for your PIN. That keeps your kids or colleagues out of your accounts when you are not around. Secondly, you can register the fingerprints of another person you trust. That comes in handy if access is needed and you are incapacitated. A use case for this is for break glass accounts to Azure, for example.

Where to use FIDO2 security keys?

People use FIDO2 security keys most often to secure applications, services in professional settings (Azure AD, O365, websites of various professional services, etc.). That said, I use them for my personal security needs as well where ever I can. They are my second MFA device next to my smartphone. I like them, and I promote them to friends and family as I explain to them about MFA and the passwordless future. Yes, I do that. It is a requirement. That requirement materializes as mandatory security training when they dare ask me to help them select a new laptop or computer.

So in a few upcoming blog posts, I will discuss how to set them up and use them with various personal and professional services. Think about Microsoft’s Outlook, Teams, OneDrive, and services like Twitter, my WordPress blog, and my personally hosted IAAS VM for RDP or console access.

Figure 2: That’s me logging in to Twitter with my FEITIAN BioPass security key.

You can also use them with your client device or on a server with the right MFA provider or security vendors software and protect access when services like Windows Hello are not available to you for whatever reason.

I have found the FEITIAN FIDO2 security keys easy to use and to work reliably. In addition, they appear to be of high quality and solid enough to survive on my keychain.

Mind you, you cannot use a FIDO2 security key everywhere yet, which I find a pity. It shows security still has some work to do. But, where I can use a FIDO2 security key in combination with a smartphone authenticator app.

No matter what, use MFA wherever you can

Remember that with MFA, you are far less likely to become a victim of unauthorized access to your clients and services. It makes so much sense it should be the default for everyone today. It is an essential step on the road to an actual passwordless world in a zero-trust environment. We are getting closer to that world as recently, Microsoft allows you to remove the password from your personal Microsoft account when you have MFA in place.

Disclaimer

FEITIAN did not sponsor me or otherwise reward me for writing this blog post. However, they did provide me with the two FIDO2 security keys, which I appreciate as it helps me test and show scenarios in the lab and at presentations.  For that, I would like to thank them.