Do Azure VPN Gateways that leverage BGP support BFD?

Introduction

Anyone doing redundant, high-available VPN gateways leveraging BGP (Border Gateway Protocol) has encountered BFD (Bi-Directional Forwarding Detection). That said, BFD is not limited to BGP but also works with OSPF and OSPF6. But before we answer whether Azure VPN Gateways that leverage BGP support BFD, l briefly discuss what BFD does.

Bi-Directional Forwarding Detection

The BFD (Bi-Directional Forwarding Detection) protocol provides high-speed and efficient detection for link failures. It works even when the physical link has no failure detection support itself. As such, it helps routing protocols, such as BGP, failover much quicker than they could achieve if left to their own devices.

BFD control packets are transmitted via UDP from source ports between 49152-65535 to destination port 3784 (single-hop, RFC 5880, RFC 5881, and RFC 5882) or 4784 (multi-hop, RFC 5883). It can be IPv4 as well as IPv6. See Bidirectional Forwarding Detection on Wikipedia for more information. Note that this works between directly connected routers (single-hop) or (multi-hop).

Azure VPN Gateways that leverage BGP support BFD

Currently, OPNsense and pfSense, with the FRR (Flexible Rigid Routing) plugin, support BFD integration with BGP, Open Shortest Path First (OSPF), and Open Shortest Path First version 6 (OSPF6). Naturally, most vendors support this, but I mention OPNsense and pfSense because they offer free, fully functional products that are very handy for demos and lab testing.

Do Azure VPN Gateways that leverage BGP support BFD?

TL;DR: No.

You do not find much information when you search for BFD information about Microsoft Azure networking. Only for Azure ExpressRoute does Microsoft clearly state that it is supported and provides information.

But what about Azure VPN gateways with BFD? Well, no, that is not supported at all. You can try to set it up, but your VPN Gateways on-premises is shouting into a void. The session status with the peers will always be “down.” It just won’t work.

Visualize an Always On VPN device tunnel connection while disabling the disconnect button

Visualize an Always On VPN device tunnel connection while disabling the disconnect button

The need to visualize an Always On VPN device tunnel connection while disabling the disconnect button arises when the user experiences connectivity issues. End users should be able to communicate quickly to their support desk whether or not they have a connected Always On VPN device tunnel. They usually do not see the device VPN tunnel in the modern UI. Only user VPN tunnels show up. Naturally, we don’t want them to disconnect the device VPN or change its properties, so we want to disable the “disconnect” and the “advanced setting buttons. Since a device VPN tunnel runs as a “SYSTEM,” they cannot do this anyway. The GUI shows “Disconnecting” but never complete.

Refreshing the GUI correctly shows “Connected” again. However, it makes sense to disable the buttons to indicate this. So how to we set all of this up?

Visualize an Always On VPN device tunnel connection

Visualizing the Always On VPN device tunnel in the modern GUI is something we achieve via the registry. Scripting deploying these registry settings via GPO or Intune is the way to go.

New-Item -Path ‘HKLM:\SOFTWARE\Microsoft\Flyout\VPN’ -Force
New-ItemProperty -Path ‘HKLM:\Software\Microsoft\Flyout\VPN\’ -Name ‘ShowDeviceTunnelInUI’ -PropertyType DWORD -Value 1 -Force

Disable the disconnect button and the advanced options buttons

Now that the Always On VPN device tunnel is visible in the GUI, we want to disable the disconnect button and the advanced options buttons. How? Well, we can do this in Windows 11 22H2 or more recent versions. For this, we add the following to the VPN configuration file.

<!-- The below 2 GUI settings are only available in Windows 11 22H2 or higher. --><DisableAdvancedOptionsEditButton>true</DisableAdvancedOptionsEditButton><DisableDisconnectButton>true</DisableDisconnectButton>

  <!– These GUI settings below are only available in Windows 11 22H2 or higher. –>    <DisableAdvancedOptionsEditButton>true</DisableAdvancedOptionsEditButton>    <DisableDisconnectButton>true</DisableDisconnectButton>

Visualize a device VPN tunnel connection while disabling the disconnect button

Results

For an administrative account, the Always On VPN device tunnel is visible, but the buttons are dimmed (greyed out).

As before, the administrator can still use the rasphone GUI to hang up the Always On VPN device tunnel or edit the properties like before. Usually, you’ll configure the setting with Intune or via GPO with Powershell and custom XML. There is also a 3rd party option for configuring Always On VPNs via GPO (AOVPN Dynamic Profile Configurator).

For a non-administrator user account, the GUI looks precisely the same. The big difference is that when such a user launches the rasphone GUI, they cannot “Hang Up” the connection. The error message may not be the clearest, but in the end, a user with non-administrative rights cannot disconnect the Always On VPN device tunnel.

So now we have the best of both worlds. An administrator and a standard user can see that the Always On VPN device tunnel is connected. Remember that disabling the buttons requires Windows 11 22H2 or more recent. This blog was written using 23H2. The administrator can use the rasphone GUI or rasdial CLI to access the Always On VPN device tunnel like before.

Conclusion

Device VPN tunnels are supposed to be connected at all times, whether a user is logged on or not. It is also something that users are not supposed to be concerned about in contrast to a user VPN tunnel. However, it can be beneficial to see whether the Always On VPN device tunnel is connected. That is most certainly so when talking to support about an issue. We showed you how to achieve this, combined with disabling the “disconnect” and “advanced” options buttons), in this blog post.

What is AzureArcSysTray.exe doing on my Windows Server?

Introduction to AzureArcSysTray.exe

After installing the October 2023 updates for Windows Server 2022, I noticed a new systray icon, AzureArcSysTray.exe.

What is AzureArcSysTray.exe doing on my Windows Server?

It encourages me to launch Azure Arc Setup.

AzureArcSysTray.exe

Which I hope takes a bit more planning than following a systray link. But that’s just me, an old-school IT Pro.

Get rid of the systray entry

Delete the AzureArcSysTray.exe value from the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run registry key. Well, use GPO or another form of automation to get this done whole sales. I used Computer Configuration GPO Preferences in the lab. See the screenshot below from my home lab. It is self-explanatory.

The added benefit of a GPO is that it will deal with it again if Microsoft pushes it again in the next update cycle.

Uninstall the feature

Using DISM or Server Manager, you can uninstall the feature altogether. Do note that this requires a reboot!

Disable-WindowsOptionalFeature -Online -FeatureName AzureArcSetup

Or

Note: This removes the systray exe and Azure Arc Setup. If someone already set it up and configured it, that is still there and needs more attention as the Azure Connect Machine Agent is up and running. Does anyone really onboard servers like this in Azure Arc?

Bad Timing

Well, this was one thing I could have done without on the day I was deploying the October 2023 updates expeditiously. Why expedited? Well, it was about 104 CVEs, of which 12 are critical Remote Code Execution issues, and 3 are ZeroDays) and a Hyper-V RCT fix that we have been waiting for (for the past five – yes, 5 – years). Needless to say, testing + rollout was swift. That AzureArcSysTray.exe delayed us, as we had to explain and mitigate it.

Is it documented?

Yes, it is, right here: https://learn.microsoft.com/en-us/azure/azure-arc/servers/onboard-windows-server. It was incomplete on Tuesday night, but they added to it quickly.

Judging from some social media, Reddit. Slack channels, not too many people were amused with all this. See https://www.reddit.com/r/sysadmin/comments/174ncwy/patch_tuesday_megathread_20231010/k4cdqsj/

https://www.reddit.com/r/sysadmin/comments/1763a7o/azure_arc/https://www.reddit.com/r/sysadmin/comments/1763a7o/azure_arc/

Conclusion

I had to explain what it was and our options to eliminate it, all while we were asked to deploy the updates as soon as possible. Finding AzureArcSysTray.exe Azure Arc Setup installed was not part of the plan late Tuesday night in the lab.

Please, Microsoft, don’t do this. We all know Azure Arc is high on all of Microsoft’s agenda. It is all the local Microsoft employees have been talking about for weeks now. We get it. But nagging us with systray icons is cheesy at best, very annoying, and, for many customers, nearly unacceptable.

Active Directory Replication Status Tool’s rise, fall, and rebirth

Active Directory Replication Status Tool’s rise, fall, and rebirth

For many years the Active Directory Replication Status Tool has been a trusty companion for many IT Pro. That includes seasoned systems engineers as well as accidental Active Directory administrators. It was an easy way to get a quick and good idea of the replication health of your Active Directory forest or domain.

Sure, repladm is our friend and keeps doing its job with us in the trenches. But I would say that the Active Directory Replication Status Tool is a lot less scary for people. Especially those who incidentally need to find out what issues to address where. It is a non-scarry, kind wrapper to visualize the results and see where we need to pay attention.

It had some quirks, like the ridiculous need to download it again when the license (certificate) expired (I never liked the Sirona DLL hack in a production environment). If other issues arose, I blogged on how to work around them, like in Microsoft Active Directory Replication Status Tool won’t upgrade – Working Hard In ITWorking Hard In IT. But that was minor compared to the sage of the last 18 months. Let’s quickly look at the Active Directory Replication Status Tool’s rise, fall, and rebirth.

The fall

Once in a while, an update to Windows or .NET broke the application. Normally, a fix would follow soon, and everyone was happy again. Last year in the spring or early summer of 2022, the tool was broken for months. Finally, somewhere in August 2022, we got a new version that worked, as far as my experience goes, for about 4 to 6 weeks, It then broke again, and all it ever did after was crash. The cert also expired again but who cared? It was no longer functional anyway,

Well, I cared, and I gave a lot of feedback via Twitter and to Microsoft via e-mail.

Active Directory Replication Status Tool's rise, fall, and rebirth

That went on for about 10 months without any progress! Then suddenly, I notice this: How to get and use the Active Directory Replication Status Tool – Windows Server | Microsoft Learn.

Active Directory Replication Status Tool's rise, fall, and rebirth

What!!??? Is this how it ends? Abandoned in a ditch somewhere?

Important

As of June 2nd, 2023, the Active Directory Replication Status Tool is no longer available for download. The following article is provided for historical purposes only.

Are you kidding me? There went another e-mail right to Redmond! Was this the sad end? I got a reply that not all was lost with a link.

The rebirth

The link I got is this one GitHub – ryanries/ADReplStatus: AD Replication Status Tool. People meet Ryan Ries, an Escalation Engineer at Microsoft who has some very useful and handy private projects to share with the world. ADReplStatus is one of the more recent ones.

I downloaded it and started testing it in the lab. That went well and has already found its way to two production environments. Here is a screenshot from my lab environment!

Active Directory Replication Status Tool's rise, fall, and rebirth

Thank you, Ryan Ries, for helping your customers in your free time with your private projects. You have made my IT Pro existence a bit easier again instead of more difficult. It is appreciated! Thank you, Ned Pyle, for bringing this GitHub repo to my attention. Download your copy here Release v1.3.1 · ryanries/ADReplStatus · GitHub.

What I did find interesting was that the cert expiration time bomb was also an internal issue. That and the fact that the application had no maintainers. But that was obvious to us all.

The old version of the tool had a time bomb in it – an expiring SSL certificate – that rendered the app unusable sometime around September 2022. Only through great effort internally were we able to periodically renew this certificate and republish the app, and the app was architected in such a way that excising the signing certificate check was more work than just rewriting the whole thing. (I know about the Sirona DLL hack, but still.)
The old tool had no active maintainers and no one who was still around was familiar with the app’s internals or source code.

Things are tough all over, it seems.