The Veeam hardened repository at the TechNine User group

Posted on March 9, 2021 by workinghardinit

The Veeam hardened repository

Just a quick blog post to let you all know I will be presenting on the Veeam hardened repository in Veeam Backup & Replication v11. I will do this at the TechNine user group meeting on March the 23rd. It is the world premiere of this presentation, so grab your seat and be able to tell your grandchildren you were there!

There is a tremendous amount of new and enhanced capabilities in v11, both small and big ones. But this one stands out because no matter who you are, a small business or a fortune 500 global enterprise, we are all at risk from ransomware and wipers. And guess what, the Veeam hardened repository gives you yet another option in your arsenal to make sure you can recover from such a disaster.

And end to end solution

In VBR v11 Veeam now offers complete end to end immutability for all your backup copies if you desire to do so. You can bring your own storage and so it leaves you the flexibility and freedom of choice you are used to with Veeam.

Like many of you, I like to stay up to date with new technology and implement it to solve real-world problems. Often coo tech is very forward-looking and foundational. This translates into “it is very early days yet”, “the code exists but you cannot use it yet”.

Well, this is not something that shows you the promises of tomorrow and then leaves you to go back empty-handed. That always feels like a bit of a letdown when you are dealing with your real-world, real-time issues. The Veeam hardened repository is available now to help address the challenges you are facing today! It doesn’t get any better than that. I will show you the cookie, you can have the cookie and you can eat the cookie! Want to play with it yourself? You can, right now with the free community edition of Veeam Backup & Replication v11!

Register!

What do you need to do? Register of cause! Do it right here. Make sure you show up in time and stay around for an interactive discussion afterward. We start at 20:00 on March 23rd, 2021. And yes it is all virtual as we plan to get rid of the coronavirus first and then get together in real life again.

I will explain why you need it, how to set it up, and I will dive under the hood to show how it gets the job done. You can now protect your hypervisor and supported backups end to end with immutability in the hardened repository, giving you the extra security and protection you might need to recovery from a ransomware event. There will be demos! You can read up on it here but I will go behind the scenes with you!

I am a Veeam Vanguard 2021

Posted on February 26, 2021 by workinghardinit

I am a Veeam Vanguard 2021

This afternoon I received an email notifying me that my 2021 Veeam Vanguard Renewal Nomination has been approved! That means I am a Veeam Vanguard 2021!

The Veeam Vanguard Program

You can read all about the Veeam Vanguard Program here. It is Veaam’s top-level influencer community. We share our insights, provide feedback, and help each other, as well as Veeam, succeed. The Veeam Vanguards are a varied group of people and combine a wide range of expertise in different verticals and in different disciplines. We all contribute to the community at large in different ways. What unites us is that we all share a passion for our industry and show technical thought leadership for the different technical communities in which Veeam exists.

Now I would not call myself a technical thought leader but I do know that via this program I get to stand on the shoulders of giants. Doing so gives me a majestic view over this industry segment, a view I would not have without it.

Thank you!

This is a very nice way to head into the weekend. I am both honored and proud to be a Veeam Vanguard. Veeam as a company has won my respect many years ago and it has only grown. They focus on customer value, deliver an excellent high-quality product, offer top-notch support, and finally offer great tools to the community. I can honestly say that I enjoy working with their products as well as the company and the individuals I know at Veeam.

I am looking forward to another year in this program and the opportunities this brings to us all for sharing, learning, and growing.

Thank you, we’ll make 2021 a good community year!

Immutability of Linux files on the Veeam hardened Linux repository

Posted on January 18, 2021 by workinghardinit

Introduction

Without any doubt, I find the hardened Linux repository Veeam introduces in Veeam Backup & Replication v11 one of the most fascinating new features to get my hands on. In the ever-escalating battle with ransomware and wipers, this is a very valuable option to have in your defensive arsenal. So, I grabbed the Beta 2 and got to work in the lab over the holidays to investigate and find out some details about the Immutability of Linux files on the Veeam hardened Linux repository.

Immutability of Linux files on the Veeam hardened Linux repository

It’s quite easy to find the file attribute “i” that marks a file as immutable.

lsattr -a

Immutability of Linux files on the Veeam hardened Linux repository — lsattr -a also shows the hidden files

lsattr -a -l

Where is the information about the immutability actually stored? I mean, that “i” attribute is one thing but how do the Linux host and Veeam know from what time period this immutability is valid. In the end, the service has to clear it and know when to do this. Or is this only stored in the Veeam database or both?

How does it now from when till when a file must be immutable?

Digging around in the files and folders of the Veeam repository, I soon found the lock file “.veeam.x.lock” (see the green arrow in the image above) that is created by the veeamimmureposvc service. The owner is root, hence it is not created by the Veeam transport service. The veeamimmureposvc service is a local account with root access for managing the immutability. It only works locally and does not listen on any network port, hence it cannot be accessed remotely.

Let’s look inside to see if we can read something there?

cat .veeam.9.lock

the lock file is an XML file containing all the date/time stamps for every file in that backup job.

When you open that file you will find it to be an XML file. Inside you’ll see the date and time stamp for every file in the backup chains for that job. That’s cool.

But there is more. When we run “getfattr *”to look for extend file attributes we find that every Veeam created file has a one called user.immutable.until.

The backup files all have an extended file attribute called user.immutable.until.

With that name, it is clear it can be of interest to us. If you look at what is in there, you’ll see it contains the date and time stamp for that file’s immutability period.

getfattr * -n user.immutable.until

That I find interesting. Veeam saves the information twice. Is that for redundancy or as some sort of checksum? Maybe it also has to do with the fact Veeam backup files are transportable and self-contained so that information is stored as an extended file attribute.

Conclusion

So there you have it. A small piece of information on where the immutability information is stored. The most surprising thing to me was that it is actual stored twice.

I hope you fund this interesting. Poking around to figure out the how and what of things always helps me tremendously to learn and understand the technologies I want to work with. That leads to better decisions in design and implementation. It leads to both trust and confidence, which helps me decide where and when to leverage it. Finally it also, almost without, it is invaluable when supporting the technology.

Extending a Veeam Repository XFS File System

Posted on January 11, 2021 by workinghardinit

Extending a Veeam Repository XFS File System

Since diving into the Veeam Backup & Replication v11 Linux hardened repository I have started to use XFS in bite-size deployments to gain experience with it. One of the things that will certainly come in handy is extending a Veeam Repository XFS File System. In this blog post, I show to do that.

Mind you that I am doing this with a virtual machine on Hyper-V (Windows Server 2019) in the lab. Not every permutation of hardware and storage controllers you can find. But still, the procedure here will not differ that much.

Determine the size of the current disk.

sudo slblk

Extending a Veeam Repository XFS File System — Ours is the 20 TB disk, sdd, a SCSI disk.

Now take note of the bytes and sectors

sudo fdisk -l

We just notice the size, bytes and sectors to compare after we extended disk.

Expand the disk

In the virtual machine settings I extend the virtual disk I want to grow with the required capacity.

In real life that might be you growing a raid controllers’ virtual disk by adding physical disks to the raid controller, you expanding the volume on the storage array or simply adding disks to the local server and adding them to the software-based raid solution you use.

The virtual machine will pick up the extra capacity right away. For our UBUNTU 20.04.1 OS to see it up we’ll need to rescan the SCSI busses for change. In a virtual machine, this can be done via rescan-scsi-bus.sh, available scsitools that will need to be installed if not there.

Use the -s options as that will really show the resized disks.

sudo apt-get install scsitools
sudo rescan-scsi-bus.sh -s

Yup, that’s our disk on SCSI controller 1, location 0.

Now let’s check the disk size again

fdisk -l confirms. Note the new bytes and sector values. It has gone up.

Extend the xfs volume to use the unallocated space

Now we need our xfs volume to use the unallocated capacity in this disk. We use -d as this will grow the file system to the largest possible size, 30 TB in our case.

Note: If you run the below command with -n instead of -d, this gives you the current information on your xfs volume with extending the filesystem yet.

sudo xfs_growfs -d /mnt/veeamxfsrepo-03

See Ubuntu Manpage: xfs_growfs – expand an XFS filesystem for more options

Note: What I did find is that if you just expand the disk and than extend the xfs file system, it also works. It seems to just work without rescanning the disk after extending it! The disks size in df -h will show this space then as well.

Conclusion

That was it. Short and sweet. There is not much to it once you know how to do it. One thing to remember is that you cannot shrink an XFS file system. So, as always, start smaller and grow when needed. Always leave spare capacity to work with when needed. Yes, even in 2021 this is advice to live by in the storage world. For Veeam this means that multiple smaller repositories or extents give you more wiggle room than fewer very large ones. Leave capacity in reserve, either in a spare repository/extend or unallocated. This, especially combined with a scale-out backup repository in Veeam will allow you to work your self out of most capacity pickles you might find your self in.

Working Hard In IT

My view on IT from the trenches

Tag Archives: VEEAM

The Veeam hardened repository at the TechNine User group