Change the service account on a Veeam hardened repository

Posted on by
Reply

Introduction

In Veeam Backup & Replication (VBR) v11 we have the new hardened repository host. I Have also shared some other findings in previous blogs. You add this to the Veeam managed servers via single-use credentials. These are not stored on the VBR server. That account only needs permission on the repository volumes after adding the repository server to the VBR managed servers. Let’s investigate how to change the service account on a Veeam hardened repository in case we ever need to. Warning, this early info and not official guidance.

Where is that account configured for use with Veeam?

Naturally, we created the account for adding the hardened repository to the Veeam managed servers. We also set the correct permission on the backup repository volumes. After adding the repository host to Veeam managed servers, with the single-use credential method, we removed this account from the sudo group and that’s it.

The VBR server itself does not hold the credentials. The credentials only live on the repository host. But where is that configured? Well under Under /opt/veeam/transport you will find a config file VeeamTransportConfig.

Change the service account on a Veeam hardened repository
Under /opt/veeam/transport you find a config file VeeamTransportConfig

Open it up in your favorite editor and take a look.

Change the service account on a Veeam hardened repository
ServiceUser is where the user account is configured

Change the service account on a Veeam hardened repository

From my testing, you can simply change the user account in the VeeamTransportConfig file if you ever need to. When you do, save the file and restart the veeamtransport service so it takes effect.

sudo service veeamtransport restart

When you change this service user you must also take care of the permissions on the repository folder(s).

We change the ownership recursively for the drive mount to that user.

sudo chown -R newveeamrepouser: newveeamrepouser /mnt/veeamxfsrepo01-02

The permissions for the user should still be what it needs to be (chmod 700)

Now, due to the immutability of the backup files changing the owner will fail on those files but that is OK.

Change the service account on a Veeam hardened repository
Don’t panic, you’ll be fine. Backups & restores will work.

For reading and restoring then tests show that this still works. Backup wise this will run just fine. It is all pretty transparent from the Veeam Backup & Replication side of things. I used to think I would need to run a full backup afterward but Veeam seems to handle this like a champ and I do not have to take care of that it ssems. We’ll try to figure out more when V11 is generally available.

Now, you still might want to give Veeam support a call when you want or need to do this. Remember, this is just informational and sharing what I learn in the lab.

Conclusion

Normally you will not need to change this service account. But it might happen, either because of a policy that mandates this, etc. That is the reason I wanted to find out how to do this and if you can do this without breaking the backup cycles. I also think this is why you can convert an existing Linux repository to a hardened one.

Immutability of Linux files on the Veeam hardened Linux repository

Posted on by
4

Introduction

Without any doubt, I find the hardened Linux repository Veeam introduces in Veeam Backup & Replication v11 one of the most fascinating new features to get my hands on. In the ever-escalating battle with ransomware and wipers, this is a very valuable option to have in your defensive arsenal. So, I grabbed the Beta 2 and got to work in the lab over the holidays to investigate and find out some details about the Immutability of Linux files on the Veeam hardened Linux repository.

Immutability of Linux files on the Veeam hardened Linux repository

It’s quite easy to find the file attribute “i” that marks a file as immutable.

lsattr -a
Immutability of Linux files on the Veeam hardened Linux repository
lsattr -a also shows the hidden files

or

lsattr -a -l
Immutability of Linux files on the Veeam hardened Linux repository
lsattr -a -l list out the full name of the attribute.

Where is the information about the immutability actually stored? I mean, that “i” attribute is one thing but how do the Linux host and Veeam know from what time period this immutability is valid. In the end, the service has to clear it and know when to do this. Or is this only stored in the Veeam database or both?

How does it now from when till when a file must be immutable?

Digging around in the files and folders of the Veeam repository, I soon found the lock file “.veeam.x.lock” (see the green arrow in the image above) that is created by the veeamimmureposvc service. The owner is root, hence it is not created by the Veeam transport service. The veeamimmureposvc service is a local account with root access for managing the immutability. It only works locally and does not listen on any network port, hence it cannot be accessed remotely.

Immutability of Linux files on the Veeam hardened Linux repository
The veeamimmureposvc service controls the .veeam.x.lock file. the x is a number has increments with every backup job you run.

Let’s look inside to see if we can read something there?

cat .veeam.9.lock
the lock file is an XML file containing all the date/time stamps for every file in that backup job.

When you open that file you will find it to be an XML file. Inside you’ll see the date and time stamp for every file in the backup chains for that job. That’s cool.

But there is more. When we run “getfattr *”to look for extend file attributes we find that every Veeam created file has a one called user.immutable.until.

The backup files all have an extended file attribute called user.immutable.until.

With that name, it is clear it can be of interest to us. If you look at what is in there, you’ll see it contains the date and time stamp for that file’s immutability period.

getfattr * -n user.immutable.until
Immutability of Linux files on the Veeam hardened Linux repository
The extended file attribute contains the timestamp until when that backup file is immutable!

That I find interesting. Veeam saves the information twice. Is that for redundancy or as some sort of checksum? Maybe it also has to do with the fact Veeam backup files are transportable and self-contained so that information is stored as an extended file attribute.

Conclusion

So there you have it. A small piece of information on where the immutability information is stored. The most surprising thing to me was that it is actual stored twice.

I hope you fund this interesting. Poking around to figure out the how and what of things always helps me tremendously to learn and understand the technologies I want to work with. That leads to better decisions in design and implementation. It leads to both trust and confidence, which helps me decide where and when to leverage it. Finally it also, almost without, it is invaluable when supporting the technology.

Extending a Veeam Repository XFS File System

Posted on by
4

Extending a Veeam Repository XFS File System

Since diving into the Veeam Backup & Replication v11 Linux hardened repository I have started to use XFS in bite-size deployments to gain experience with it. One of the things that will certainly come in handy is extending a Veeam Repository XFS File System. In this blog post, I show to do that.

Mind you that I am doing this with a virtual machine on Hyper-V (Windows Server 2019) in the lab. Not every permutation of hardware and storage controllers you can find. But still, the procedure here will not differ that much.

Determine the size of the current disk.

sudo slblk
Extending a Veeam Repository XFS File System
Ours is the 20 TB disk, sdd, a SCSI disk.

Now take note of the bytes and sectors

sudo fdisk -l
We just notice the size, bytes and sectors to compare after we extended disk.

Expand the disk

In the virtual machine settings I extend the virtual disk I want to grow with the required capacity.

Extending a Veeam Repository XFS File System
Let’s add 20 TB and make it 30 TB in total.

In real life that might be you growing a raid controllers’ virtual disk by adding physical disks to the raid controller, you expanding the volume on the storage array or simply adding disks to the local server and adding them to the software-based raid solution you use.

The virtual machine will pick up the extra capacity right away. For our UBUNTU 20.04.1 OS to see it up we’ll need to rescan the SCSI busses for change. In a virtual machine, this can be done via rescan-scsi-bus.sh, available scsitools that will need to be installed if not there.

Use the -s options as that will really show the resized disks.

sudo apt-get install scsitools
sudo rescan-scsi-bus.sh -s
Yup, that’s our disk on SCSI controller 1, location 0.

Now let’s check the disk size again

Yes, lsbsk shows 30 TB.
fdisk -l confirms. Note the new bytes and sector values. It has gone up.

Extend the xfs volume to use the unallocated space

Now we need our xfs volume to use the unallocated capacity in this disk. We use -d as this will grow the file system to the largest possible size, 30 TB in our case.

Note: If you run the below command with -n instead of -d, this gives you the current information on your xfs volume with extending the filesystem yet.

sudo xfs_growfs -d /mnt/veeamxfsrepo-03
Extending a Veeam Repository XFS File System
Voila. We are done.

See Ubuntu Manpage: xfs_growfs – expand an XFS filesystem for more options

Note: What I did find is that if you just expand the disk and than extend the xfs file system, it also works. It seems to just work without rescanning the disk after extending it! The disks size in df -h will show this space then as well.

Conclusion

That was it. Short and sweet. There is not much to it once you know how to do it. One thing to remember is that you cannot shrink an XFS file system. So, as always, start smaller and grow when needed. Always leave spare capacity to work with when needed. Yes, even in 2021 this is advice to live by in the storage world. For Veeam this means that multiple smaller repositories or extents give you more wiggle room than fewer very large ones. Leave capacity in reserve, either in a spare repository/extend or unallocated. This, especially combined with a scale-out backup repository in Veeam will allow you to work your self out of most capacity pickles you might find your self in.

Check/repair/defragment an XFS volume

Posted on by
2

Introduction

As I have started to use XFS in bite-size deployments to gain experience with it I wanted to write up some of the toolings I found to manage XFS file systems. Here’s how to check/repair/defragment an XFS volume.

My main use case for XFS volumes is on hardened Linux repositories with immutability to use with Veeam Backup & Replication v11 and higher. It’s handy to be able to find out if XFS needs repairing and if they do, repair them. Another consideration is fragmentation. You can also check that and defrag the volume.

Check XFS Volume and repair it

xfs_repair is the tool you need. You can both check if a volume needs repair and actually repair it with the same tool. Note that the use of xfs_check has been depreciated or is not even available (anymore).

To work with xfs_repair you have to unmount the filesystem, so there will be downtime. Plan for a maintenance window.

To check the file system use the -n switch

sudo xfs_repair -n /dev/sdc
Check, repair, and defragment an XFS volume
Check, repair and defragment an XFS volumea dry run with xfs_repair -n

There is nothing much to do but we’ll now let’s run the repair.

sudo xfs_repair /dev/sdc
Check, repair, and defragment an XFS volume
Repairing an XFS file system

The output is similar as for the check we did for anything to repair is basically a dry run of what will be done. In this case, nothing.

Now, don’t forget to mount the file system again!

sudo mount /dev/sdc /mnt/veeamsfxrepo01-02

Check a volume for fragmentation and defrag it

Want to check the fragmentation of an XFS volume? You can but again, with xfs_db. The file system has to be unmounted for that or you will get the error xfs_db: can’t determine device size. To check for fragmentation run the following command against the storage device /file system.

sudo xfs_db -c frag -r /dev/sdc
Check, repair, and defragment an XFS volume
A lab simulation of sudo xfs_db -c frag -r /dev/sdc – Yeah know it’s meaningless 😉

Cool, now we know that we can defrag it online. For that we use xfs_fsr.

xfs_fsr /devsdc /mnt/veeamxfsrepo01-02
Check, repair, and defragment an XFS volume
There is nothing to do in our example

xfs_scrub – the experimental tool

xfs_scrub is a more recent addition but the program is still experimental. The good news is it will check and repair a mounted XFS filesystem. At least it sounds promising, right? It does, but it doesn’t work (Ubuntu 20.04.1 LTS).

No joy – still a confirmed bug – not assigned yet, importance undecided. Not yet my friends.

Conclusion

That’s it. I hope this helps you when you decide to take XFS for a spin for your storage needs knowing a bit more about the tooling. As said, for me, the main use case is hardened Linux repositories with immutability to use with Veeam Backup & Replication v11. In a Hyper-V environment of course.