This blog post will address a troubleshooting exercise with a VM (virtual machine) that would not route. As it turned out it had the default gateway set to 0.0.0.0 next to the actual gateway IP address. The VM did its job as the workload it serves is in the same subnet as the client, as it happens in the same subnet of the DC and DNS. This meant it did not lose its trust with Active Directory.
But the admins could not RDP into that VM, nor would it update, But as it did its job, many months went by until it fell too far behind in updates so they could not ignore it anymore. That’s how things go goes in real life.
Finding & fixing the issue
Superficially the configuration of the VM was totally OK. The gateway for the NIC is correct.
Under Advanced we see no other entries that would cause any issues.
But we could not deny that we had a VM that would not route at hand. Let’s figure this out and fix it.
So what does one do? If you don’t trust the CLI, check the GUI, and if you don’t trust the GUI check the CLI. As in the GUI, everything seemed fine we checked via the CLI. Name resolution worked fine, both internally and externally when checking this with nslookup. But actually getting anywhere not on the same subnet was not successful. Naturally, I did check if any forward proxy was in play but that was not the case and, this was an issue for more use cases than HTTP(S).
When I ran ipconfig /all I quickly saw the culprit. We have a Default Gateway entry pointing to 0.0.0.0 next to one for the actual gateway!
So where does that come from? Not from the GUI settings, that we can see. So I ran route print and that show us the root cause
So we needed to drop the route sending traffic for 0.0.0.0 to its own IP address as the gateway. They missed this as it does not show up in the GUI at all.
I dropped all persistent routes for 0.0.0.0 via route delete 0.0.0.0.0 mask 0.0.0.0. I check if this deleted all persistent routes via route print.
At that moment routing won’t work and we need to add the gateway back to the NIC. YOu can use the GUI or route add -p 0.0.0.0 MASK 0.0.0.0 192.168.2.7 IF 9 Once I did that things lit up. We could download and install updates from the WSUS server, they had remote desktop access again. Routing worked again in other words.
How did it happen? Ah, somewhere, somehow, someone added that route. I am not paid to do archeology or forensics in this case so, I did not try to find out the what, when, and why. But my guess is that VM had another NIC at a given time with those setting and they removed it from the Hyper-V setting without cleaning up, leading to that setting being left behind in the routing table leaving a gateway 0.0.0.0 on the NIC that is only visible in via ipconfig /all. Or they have tried to add a gateway manually to address this or another issue.
A final note
When faced with this issue, some folks on the internet will tell you to reset the TCP/IP stack and Winsock with netsh, or add a new NIC with a new IP (dynamically or via DHCP) and dump the old one. But this is all bit drastic. Check the root cause and try to fix that first. You can try drastic measures when all else fails.
IIS and HTTP/3, QUIC, TLS 1.3 in Windows Server 2022
In this blog post, we will show you how to test IIS and HTTP/3, QUIC, TLS 1.3 in Windows Server 2022. As most of you know by now, Microsoft has released Windows Server 2022 on August 18th, 2021. There are a lot of new and interesting capabilities and features. Some of them are only available in Windows Server 2022 Azure edition. The good news is that in contrast to SMB over QUIC, QUIC for IIS is available in any version of Windows Server 2022.
This will not work out of the box, but I will demonstrate how I got it to work.
HTTP/3 avoids “head of line” (HOL) blocking at the transport layer even for multiple streams. This is an improvement over HTTP/2 that still suffered from HOL despite heaving multiple streams in a single connection versus multiple connections in HTTP/1.1. As HTTP/3 leverages TLS 1.3 it also benefits from the benefits it offers.
However, you need to opt-in for TLS 1.3 to work. We do that via a registry key.
Without TLS 1.3 you cannot have QUIC and HTTP/3 used QUIC for its transport. You will need to restart http.sys or restart the server.
Below you see HTTP/2 traffic and it is leveraging TLS 1.3
When you check the certificate in the browser you can see that TLS 1.3 is used.
You can also see TLS 1.3 and TCP in WireShark.
Getting QUIC to work
Now we are not done yet, your while you now will see HTTP/2 traffic use TLS 1.3 you won’t see QUIC yet. For that, we need to add another registry key.
The web service or site will need to advertise it is available over HTTP/3. For this, we can use “Alt-Svc” headers in HTTP/2 responses or via HTTP/2 ALTSVC frames. Clients who connect over HTTP/2 can now learn the service’s HTTP/3 endpoint and, if successful, use HTTP/3.
This process happens by sending an HTTP/3 ALPN (“Application-layer Protocol Negotiation”) identifier along with the HTTP/2 responses. the HTTP3/ALPN advertises a specific version of HTTP/3 that the client can use for future requests.
The HTTP/2 ALTSVC frames can be sent via http.sys. This needs to be enabled via a registry key “EnableAltSvc”.
Again, you will need to restart http.sys or restart the server.
Start testing HTTP/3
Your IIS server via the http.sys service is now capable of serving content over HTTP/3. To check whether it is working you can use WireShark on both the client and the server to verify the web traffic is using QUIC.
Below you can see QUIC traffic to my IIS server being captured.
You can also check this via your browser’s dev tools. The way to do this differs a bit from browser to browser. Below you find a screenshot from Firefox, this has proven the most reliable browser when it comes to effectively negotiating QUIC. Hit F12, select “Network” and add the protocol column to the view. Watch out for HTTP/2 and HTTP/3.
It will help to hit refresh to make sure HTTP/3 is advertised to the client, which can then leverage it. Sometimes hitting refresh too much seems to break QUIC and then you will fall back to HTTP/2, all be it with TLS 1.3.
Any way that’s it for IIS and HTTP/3, QUIC, TLS 1.3 in Windows Server 2022 for now. I hope to come back to this later.
Allow or block specific FIDO2 security keys in Azure
There might be situations where you want to allow or block specific FIDO2 security keys in Azure. A policy mandating biometric FIDO2 keys will enforce the specific biometric capable FIDO2 security keys. This blog post provides an example of how to achieve this in Azure.
Allowing only a specific type of security key in Azure
In my example, I enforce the use of one particular biometric key, meaning that other, non-biometric FIDO2 security keys are blocked. In the lab, I only have a biometric key and a non-biometric key. I want to allow only my FEITIAN BioPass K26 security key and block the use of any other type.
We can achieve this surprisingly quickly in Azure. The capability to do so leverages the Authenticator Attestation GUID (AAGUID). During attestation of the security key, the AAGUID comes into play for looking up the device’s metadata in the FIDO Alliance Metadata Service – FIDO Alliance. As the AAGUID uniquely identifies a type of key from a specific vendor, we can use it to allow or block particular types of keys.
Note that a “type” of keys does not mean unique keys form factors by default. Keys from a vendor with the same capabilities and functionality but with different interfaces can have the same AAGUID. For example, the FEITIAN BioPass security keys come in multiple interface variants (USB-A, USB-C, Bluetooth, NFC). The K26 has a USB-C interface, and the K27 has a USB-A interface. Yet, both have the same AAGUID. So, when I allow a security key with this AAGUID in Azure, both models of the same type will be allowed. The eiPass, a touch-only device with a USB-C and a Lightning interface, will be blocked as we did not put it in our allow list.
How do you find out the AAGUID?
Perhaps the easiest way of finding out the AAGUID of your security key is to look it up in Azure if you have registered the key there. That is feasible because you will have been testing the security key or keys you want to allow. Now, when you want to block specific keys, you might not have added them. You might not even have them. Then you will need to find the AAGUID online or from the vendor.
There is also a Python script (in the Python-FIDO2 library provided by Yubico) you can use to find out the AAGUID. But, again, you need to have the device to do this.
Now, some vendors publish a list of AAGUID values for their devices. Here is the AAGUID list from Ubico and TrustKey. Of course, you can always reach out to your vendor to get them.
Setting FIDO2 security key restrictions
First of all, make sure that you have enabled the FIDO2 Security Key authentication method. You do this in the Azure portal by navigating to Azure Active Directory > Security > Authentication methods
Secondly, under Policies, click on FIDO2 Security Key to enter its settings. Under Basics, set ENABLE to Yes and set TARGET to All users or a selection of users. If you choose the latter, add users or a group of users.
In the FIDO2 Security Key settings under Configure, you find two sections GENERAL and KEY RESTRICTION POLICY.
Under GENERAL
You will generally have Allow self-service setup enabled and Enforce attestation set to Yes
Under KEY RESTRICTION POLICY
Set Enforce key restrictions to Yes
Set Restrict specific keys to Allow
Add the AAGUID of the K26 FEITIAN BioPass FIDO2 security key: 77010bd7-212a-4fc9-b236-d2ca5e9d4084
Click Save to activate the policy.
Here, I work with an allow list, so only security keys with their AAGUID in that list will be allowed to register and will work. If we used a blocklist, you allow all keys except those we explicitly put in the block list.
The effects of FIDO2 security key restrictions
So, let’s look at what happens when an end-user has a security key that is not explicitly allowed or is explicitly blocked and tries to register it. First, we allowed self-service so that the user could register their keys by themselves. They do this in the security info section under My Profile or My Sign-Ins. The process seems to work well with the FEITIAN eiPass USB-C/Lightning FIDO2 Security key, which has no biometrics. Hence we don’t allow it.
The user can complete the workflow right up to naming their security key, but when they want to apply the settings, it throws the below error.
That’s cool. What happens to users that have already registered a security key type we now block or don’t allow? Does that still work or not? Let’s find out! I tried to log on with a security key that was previously allowed, but we now blocked it. All goes well up to when I swipe my fingerprint. Then, it informs me, I cannot log in using the method and advises me to sign in via a different method and remove this security key. That is what we expect.
Finally, what happens when someone changes the policy while a user is still logged in? It either throws the same message as above or while navigating, or it throws a “something went wrong” message in your browser. When you click “View more,” it becomes evident a policy is blocking your FIDO security key.
All in all, Azure offers straightforward, effective, and efficient ways of managing what keys to allow or block. Going passwordless when you have played with the FIDO2 security keys seems a lot less complicated and scary than you might think. So please test it out and go for it. A better, safer, and easier authentication method is within grasp for everyone!
Your online credentials do not only deserve but need adequate protection against abuse. That means protecting your credentials. Unfortunately, the reality is that the most used way of doing so, the user password combo, while ubiquitous, has been insufficient for a truckload of reasons for a long time. I will not address this here, but you can read about it all day long online.
Multifactor authentication comes to the rescue, and there are two prevalent and secure forms of this. First of all, we have the software-based solution in the form of an “authenticator” app that lives on a (hopefully) secured smartphone. The second is hardware-based security keys that protect your credentials in a secure, tamper-proof vault. These come in many interface forms, but USB-A or USB-C is very popular, and the most future-proof ones are FIDO2 security keys. I prefer FIDO2 keys with biometric capabilities. These allow us to go fully passwordless in use cases that support this.
Lab setup with the FEITIAN BiosPass K26 security key
We set up a FEITIAN BioPass FIDO2 Security key for personal identity protection in our earlier blog post. Now we will use that security key to protect some of our online credentials for personal use.
FEITIAN BioPass FIDO2 Security key for personal identity protection
Microsoft personal account
Many of us have a Microsoft personal account. Think Outlook, OneDrive, Teams, etc. These are not just for O365 personal or business subscriptions but also work with Microsoft’s free offerings.
What I will do now is configure an account to leverage a security key. In this case, this is the FEITIAN BioPass FIDO2 K26 USB-C model I configured with my fingerprint in
If you already have MFA set up, you might need to approve the challenge as I do below. In that case, your security key will become the second option for MFA.
Once you log in, click on the security panel.
There you will find the” Advanced security options panel,” which you click.
Now you can “Add a new way to sign in,” which is what we want.
Select “Use a security key.”
Next, you find the instructions on what you will need to do to onboard your security key, both in writing and via images.
Windows Security notifies you that you are setting up your security key and tells you what application makes this request. In our example, it is the Chrome web browser.
Click “OK” and follow the instructions
Windows Security informs you that a credential will be stored on your security key so you can log in without having to type a username. Click “OK” to continue.
Touch the security key when requested. Remember we already stored your on the security key, but you need to provide your pin code to verify that this key belongs to you.
Enter your PIN-code and click “OK.”
Touch your security key again, and this time it verifies a fingerprint of yours. When that is successful, you will see that you need to name your security name to identify which one it is quickly. Do so and click “Next.”
You have now successfully onboarded the security key and are ready to use it instead of a password.
That is it. Tell me now, that wasn’t to be right? You can see the security key you added to your sign-in options.
Now that you have added the FEITIAN BioPass FIDO2 security key to your personal Microsoft account, you can now use it to log in to Outlook, OneDrive, and Teams. Of course, teams can be a bit picky when dealing with a security key and if your account belongs to multiple tenants as a guest user and you do not always use the tenant in which your account lives. But you can work through that.
This time you select “Sign In with Windows Hello or a security key” or use the “Sign in options” button.
The security key is the option that you are after, so select that.
The form prompts you to touch your security key, which, as it is a FEITIAN BioPass FIDO2 device, will read your fingerprint and validate that.
That will log you on. The following prompt asks if you want to stay signed in or not.
Realize that you no longer need to type in your username or password. Having the key is one factor. Knowing the PIN or presenting the fingerprint while touching the key provides for a secure second factor. Note that you might opt not to stay signed in for maximum security for the authentication always to be required.
Also, note that if you navigate to onedrive.com versus login.live.com, the experience is similar but different. When writing this article, in onedrive.com, you need to enter your account name before continuing and selecting another option for authentication than a password. With login.live.com, you get that option directly.
Finally, don’t forget that you can go genuinely passwordless and remove your password when you are ready to take that step.
Google account
Adding an MFA option to your google account is pretty straightforward. You have the option to add a security key next to push notifications to an authenticator app, a voice or an SMS message, or a Google prompt to a phone where you signed in with your Google account. Note that I gave the security key a sensible name for identification. See Use a security key for 2-Step Verification – Computer – Google Account Help for more information and guidance.
The above screenshot is of the FETIAN BioPass FIDO2 security key added to a Google account. Note that for now, Google does not let you go completely passwordless. The fingerprint on your security key is the second factor after entering your password. We’ll see how this evolves in the future.
Twitter
Recently Twitter also announced support for a security key in Stronger security for your Twitter account. Once you have done that, you will have a login process as described below.
First, navigate to twitter.com and select Sign in, where you opt for username, e-mail, or phone.
Fill out your username when requested. Click “Next” to continue.
Enter your password en click “Log in” to carry on.
Now MFA kicks in, and you will need to touch your security key to respond to the MFA challenge. Note that you still have to enter your password here. You cannot (yet) go passwordless here.
The security key will read and verify (one of) your registered fingers prints and if that matches, allow your login. That’s it, and I am now reading my Twitter stream!
Call to action
First of all, start using MFA for your personal computing needs. Authenticator apps on smartphones are the usual first choice. Secondly, give FIDO 2 security keys a go either as a secondary or primary MFA device. While not all applications support a FIDO2 security key, many do, and I encourage anyone to try it out and see where it fits in. On that note, you can even use it for access to your Windows laptop or PC and even servers with the right (free) MFA provider and software. That means you have the best possible protection even when Windows Hello is unavailable to you due to a non-compatible camera on your client or because it does not work for your use case.
When and where supported, you are now ready to go passwordless, and you have at least two forms of MFA set up. Last but not least, you have your recovery keys and know the recovery process for your services when you lose access to a service. Well done!
