KeePass SecretManagement extension vault

KeePass SecretManagement extension vault

The SecretManagement and SecretStore can work with SecretManagement extension vault modules. These can be found in the PowerShell Gallery using the “SecretManagement” search tag. Some example are:

I use KeePass and as such, the KeePass SecretManagement extension vault is the one I will demonstrate. First of all, install the module. Note that I chose to use the most recent beta version, which is 0.9.2-beta0008 at the time of writing this blog post.

Install-Module -Name SecretManagement.KeePass -AllowPrerelease

Naturally, if you haven’t installed SecretManagement and SecretStore modules yet, you must now really do that to be able to play with them.

Install-Module Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore

Now that has been taken care of we can start testing the KeePass SecretManagement extension vault.

Using the KeePass SecretManagement extension vault

I created a demo KeePass .kdbx file in which I stored some example user names with their passwords. This file has a master password. You can also use a key or the Windows user account if you want to do so.

Our demo .kdbx file

Now I will register the KeePass file as a Vault

Register-KeePassSecretVault -Name 'WorkingHardInITKeePassVault' -Path 'C:\SysAdmin\Authentication\workinghardinit.kdbx' -UseMasterPassword
KeePass SecretManagement extension vaultRegister the KeePass Vault

As you can see this prompts you for the KeePass Master Pasword.

Keepass Master Password
Enter the Keepass Master password for: C:\SysAdmin\Authentication\workinghardinit.kdbx
Password for user Keepass Master Password:

Now that is done, I will unlock the KeePass secret vault so I can use it in automation without being prompted for it. By default, it remains unlocked for 900 seconds (15 minutes). This is configurable.

Unlock-KeePassSecretVault -Name 'WorkingHardInITKeePassVault'
Unlock the KeePass Vault, by entering the store password and, if not opened yes the KeePass master password
$FCcreds = Get-Secret -Name 'FC Switch 01'  -Vault 'WorkingHardInITKeePassVault'
$FCSwitchUser = $FCcreds.GetNetworkCredential().UserName
$FCSwitchPwd  =$FCcreds.GetNetworkCredential().Password
write-Host -foregroundcolor Green "FC Swicth 01 username $FCSwitchUser has $FCSwitchPwd for its password"
KeePass SecretManagement extension vault
We grab the username for the FC Switch 01 entry in the KeePass secret Vault.

Note that the entry for the secret is a network credential. As result, we can use the properties of the credential object to obtain the username and password in plain text. That is to say, we can (and should) use the credentials directly. You do not need to show or use the password in plain text. I did this here to show you that we got the correct values back.

Credentials ready to use.

Updating and adding secrets

Currently, updating the secrets with is not supported.

Let’s hope that theu allow updating and document using the hash table to enter metadata better in the future.

We need to first remove the existing one for now and re-enter the information. We’ll see how this evolves

Remove-Secret -Name 'FC Switch 01' -Vault 'WorkingHardInITKeePassVault'
$FCcreds = Get-Credential -UserName 'fcadmin'
Set-Secret -Name 'FC Switch 01' -Secret $FCcreds -Vault 'WorkingHardInITKeePassVault'

Finally, the good news is that there is also a PowerShell KeePass module that you can use for that sort of work. So you have the means in PowerShell to do so. See Getting Started · PSKeePass/PoShKeePass Wiki (github.com).

Conclusion

That was fun, was it not? The SecretManagement and SecretStore modules are going places. I hope this helps and happy scripting!

Veeam Backup & Replication Preferred Subnet & SMB Multichannel

Introduction

In a previous blog post Veeam Backup & Replication leverages SMB Multichannel post we showed that Veeam backup & Replication leverages SMB multichannel when possible.

But what about Veeam Backup & Replication Preferred Subnet & SMB Multichannel, does that work? We mentioned that we wanted to answer the question what happens if we configure a preferred back-up network in Veeam Backup & Replication. Would this affect the operation of SMB multichannel at all? By that I means, would enabling a preferred network in Veeam prevent multichannel from using more than one NIC?

In this blog post we dive in to that question and some scenarios. We actually need to be able to deal with multiple scenarios. When you have equally capable NICs that are on different subnets you might want to make sure it uses only one. Likewise, you want both to be used whether they are or are not on the same subnet even if you set a preferred subnet in Veeam. The good news is that the nature of SMB Multichannel and how Veaam preferred networks work do allow for flexibility to achieve this. But it might not work like you would expect, unless you understand SMB Multichannel.

Veeam Backup & Replication Preferred Subnet & SMB Multichannel

For this blog post we adapt our lab networking a bit so that our non-management 10Gbps rNICs are on different subnets. We have subnet 10.10.110.0/24 for one set of NICs and 10.10.120.0/24 for the second set of NICs. This is shown in the figure below.

image

These networks can live in a separate VLAN or not, that doesn’t really matter. It does matter if to have a tagged VLAN or VLANS if you want to use RDMA because you need it to have the priority set.

We now need to configure our preferred network in Veeam Backup & Replication. We go to the main menu and select Network Traffic Rules

clip_image004

In the Global Network Traffic Rules window, click Networks.

clip_image006

In the Preferred Networks window, select the Prefer the following networks for backup and replication traffic check box.

clip_image008

Click Add. We use the CIDR notation to fill out our preferred network or you can use the network mask and click OK.

To prove a point in regards to how Multichannel works isn’t affected by what you fill out here we add only one of our two subnets here. SMB will see where it can leverage SMB multichannel and it will kick in. Veeam isn’t blocking any of its logic.

So now we kick of a backup of our Hyper-V host to our SMB hare target backup repository. We can see multichannel work just fine.

clip_image010

Below is a screenshot on the backup target of the backup running over SMB multichannel, leveraging 2 subnets, while having set only one of those as the preferred network in Veeam Backup & Replication

clip_image012

Look at my backup fly … and this is only one host that’s being backup (4 VMs actually). Have I told you how much I love flash storage? And why I’m so interested in getting ReFS hybrid volumes with SSD/SATA disks to work as backup target? I bet you do!

Looking good and it’s easy, right? Well not so fast!

Veeam does not control SMB Multichannel

Before you think you’re golden here and in control via Veeam lets do another demo. In the preferred network, we enter a subnet available to both the source and the target server but that is an LBFO (teamed) NIC with to 1Gbps members (RSS is enabled).

clip_image014

No let’s see what happens when we kick of a backup.

clip_image016

Well SMB multichannel just goes through its rules and decided to take the two best, equally capable NICs. These are still our two 10Gbps rNICs. Whatever you put in the preferred network is ignored.

This is neither good or bad but you need to be aware of this in order to arrange for backups to leverage the network path(s) you had in mind. This is to avoid surprises. The way to do that the same as you plan and design for all SMB multichannel traffic.

As stated in the previous blog post you can control what NICs SMB multichannel will use by designing around the NIC capabilities or if needed disabling or enabling some of these or by disabling SMB multichannel on a NIC. This isn’t always possible or can lead to issues for other workloads so the easiest way to go is using SMB Multichannel Constraints. Do note however that you need to take into consideration what other workloads on your server leverage SMB Multichannel when you go that route to avoid possible issues.

As an example, I disabled multichannel on my hosts. Awful idea but it’s to prove point. And still with our 10.10.0.0/16 subnet set as preferred subnet I ran a backup again.

clip_image018

As you can see the 2*1Gbps LBFO NIC is doing all the lifting on both hosts as it’s switch independent and not LACP load balancing mode we’re limited to 1Gbps.

So how do we control the NICs used with SMB multichannel?

Well SMB Multichannel rules apply. You use your physical design, the capabilities of the NICs and SMB constraints. In reality you’re better off using your design and if needed SMB multichannel constraints to limit SMB to the NICs you want it to use. Do not that disabling SMB Multichannel (client and or server) is a global for the host. Consider this as it affects all NICs on the host, not just the ones you have in mind for your backups. In most cases these NICs will be the same. Messing around with disabling multichannel or NIC capabilities (RSS, RDMA) isn’t a great solution. But it’s good to know the options and behavior.

Some things to note

Realize you don’t even have to set both subnets in the preferred subnets if they are different. SMB kicks of over one, sees it can leverage both and just does so. The only thing you manipulated here SMB multichannel wise is which subnet is used first.

If both of our rNICs would have been on the same subnet you would not even have manipulated this.

Another thing that’s worth pointing out that this doesn’t require your Veeam Backup & Replication VM to have an IP address in any of the SMB multichannel subnets. So as long as the source Hyper-V hosts and the backup target are connected you’re good to go.

Last but not least, and already mentioned in the previous blog post, this also leverages RDMA capabilities when available to help you get the best throughput, lowest latency and leave those CPU cycles for other needs. Scalability baby! No I realize that you might think that the CPU offload benefit is not a huge deal on your Hyper-V host but consider the backup target being hammered by several simultaneous backups. And also consider that some people their virtual machines look like below in regards to CPU usage, in ever more need of more vCPU and CPU time slices.

clip_image020

And that’s what the Hyper-V host looks like during a backup without SMB Direct (with idle VMs mind you).

clip_image022

All I’m saying here is don’t dismiss RDMA too fast, everything you can leverage to help out and that is available for free in the box is worth considering.

Note: I have gotten the feedback that Veeam doesn’t support SMB Direct and that this was confirmed by Veeam Support. Well, Veeam Backup & Replication leverages SMB 3 but that’s an OS feature. Veeam Backup & Replication will work with SMB Multichannel, Direct, Signing, Transparent Failover … It’s out of the Veeam Backup & Replication scope of responsibilities as we have seen here. You feel free to leverage SMB Direct whether that is using iWarp/Roce or Infiniband. This information was confirmed by Veeam and bears the “Anton Gostev seal of approval”. So if SMB Direct cause issues you have a configuration problem with that feature, it’s not Veeam not being able to support it, it doesn’t know or care actually.

Conclusion

The elegance and simplicity of the Veeam Backup & Replication GUI are deceiving. Veeam is extremely powerful and is surprisingly flexible in how you can leverage and configure it. I hope both my previous blog post and this one have given you some food for thought and ideas. There’s more Veeam goodness to come in the coming months when times allows. Many years ago, when SMB 3 was introduced I demonstrated the high availability capabilities this offered for Veeam backups. I’ll be writing about that in another blog post.

Kick start ADFS when your self- signed certificates have expired already

I recently had to do some lab work on a Windows Server 2012 R2 ADFS farm to prep for a migration to Windows Server 2016.  Due to some storage shortage and some upgrades and migrations (all hardware in the lab runs Windows Server 2016) I had parked my Windows Server 2012 R2 ADFS farm offline.

So when I copied them back to my cluster and imported them I knew I had to make sure the domain was OK. This is easy enough, just run:

Reset-ComputerMachinePassword [-Credential Mydomain\bigadmin -Server MyDC01

That worked like a charm and soon enough my 2 VMs where up an running happily in the domain. I did have some issues however. My AFDS servers had been of line long enough before the expiration of the token-decrypting and the token-signing certificates to not yet have generate the new certificates for auto renewal and long enough to have them expire already. Darn!

image

The result was a bunch of errors in the event log as you might expect and appreciate.

image

An error occurred during an attempt to build the certificate chain for configuration certificate identified by thumbprint ’26AFDC4A226D2605955BF6F844F0866C14B1E82B’. Possible causes are that the certificate has been revoked or certificate is not within its validity period.
The following errors occurred while building the certificate chain: 
MSIS2013: A required certificate is not within its validity period when verifying against the current system clock.

But this also raised the question on how to get the ADFS servers back in a working condition. Normally these are generated automatically close to the expiration date of your existing certs (or at the critical threshold you configured). So I disabled / re-enabled auto certificate rollover but does actually does it even kick in if you have already expired? That I don’t know and I really had no time to wait hours or days to see what happens.

Luckily there is a command you can issue to renew the certificates immediately. This is the same command you can use when you have disabled auto rollover and need it re-enabled. That works normally after some patience.

Update-AdfsCertificate -Urgent

The result was immediate, the self signing certs were renewed.

image

And we can see this in the various entries in the event log

image

Do note that this command will cause a disruption of the service with your partners until they have refreshed the information from your federation metadata – or in the case this isn’t or can’t be leveraged, manually updated. In my case I had a “service down” situation anyway, but in normal conditions you’d plan this and follow the normal procedure you have in place with any partner that need your ADFS Services.

Disk2VHD on a Generation 2 VM results in an unbootable VHDX

Most people who have been in IT for a while will know the Windows  Sysinternals tools and most certainly the small but brilliant Disk2VHD tool we can use for Physical To Virtual (P2V) and Virtual to Virtual (V2V) conversions. It’s free, it’s good and it’s trustworthy as it’s made available by Microsoft.

For legacy systems, whether they are physical  with IDE/SATA/SAS controllers or virtual with an IDE generation 1 VMS thing normally go smooth.

image

But sometimes you have hiccups. One of those is when you do a V2V of a generation 2 virtual machine using Disk2VHD. It’s a small issue, when you create a new generation 2 VM and point it to the OS vhdx it just won’t boot. That’s pretty annoying.

image

Why do a V2V in such a case you might ask. Well, sometimes is the only or fasted way to get out of pickle with a ton of phantom, non-removable checkpoints you’ve gotten yourself into.

But back to the real subject, how to fix this. What we need to do is repair the boot partition. Well recreate it actually as when you look at it after the conversion you’ll notice is RAW. That’s no good. So let’s walk through how to fix a vdhx that your created from a source generation 2 Hyper-V vm via Disk2VHD.

First of all create a new generation 2 VM that we’ll use with our new VHDX we created using Disk2VHD. Don’t create a new vdhx but select to use an existing one and point it to the one we just created with Disk2VHD. Rename it if needed to something more suitable.

Don’t boot the VM but add a DVD and attach the Windows Server ISO of the version your vhdx contains to the DVD.

image

Move the DVD to the top of the boot order I firmware.

image

The VM will boot to the DVD when you hit a key.

Select your language and keyboard layoout when asked and the don’t install or upgrade the OS but boot

image

image

 

image

 

 

image

 

image

Type diskpart and  list the disks. Select the disk we need (the OS disk, the only one here) and list the volumes. You can see that volume 3 off 99MB is RAW. That’s not supposed to be that way. So let’s fix this by creating boot loader directory structure, repair the boot record by creating the boot sector & copy the needed boot files into it.

Type:

select volume 3

assign drive letter L:

FORMAT FS=FAT32 LABEL=”BOOT”

That’s it we can now us that 99MB volume to make our disk bootable to windows again.  Type Exit to leave diskpart.

image

So now we have a formatted boot partition we can create the need folder structure and fix the boot record and configure our UEFI bootloader

Switch to the L: volume

create efi\microsoft\boot folder structure for the bootloader as show below with the md command(make directory)

Type: bootrec /fixboot to create the bootrecord

Type: bcdboot C:\Windows  /l en-us /s l: /f ALL

This creates the BCD store & copies the boot files from the windows system directory

 image

Just click Continue to exit and continue to Windows Server 2012R2

image

.. and voila, your new VM has now booted.

image

Now it’s a matter of cleaning up the remnants of the original VMs hardware such as the NIC and maybe some other devices. The NIC is very important as it will have any static TCP/IP configuration you might want to assign tied to it which mean you can’t reuse it for your new VM. So, the 1st thing to do is uninstall the old network adapters from device managers, you’ll see them when you select “show hidden devices” in the view menu.

Good luck!