November 2019 updates caused Windows Server 2012 reboot loop

Introduction

Some laggards that still have some Windows Server 2012 virtual machines running got a bit of a nasty surprise last weekend. A number of them went into a boot loop. Apparently the November 2019 updates caused Windows Server 2012 reboot loop. Well, we have dealt with update issues before like here in Quick Fix Publish : VM won’t boot after October 2017 Updates for Windows Server 2016 and Windows 10 (KB4041691). No need to panic.

Symptoms

Not all Windows Server 2012 virtual machines were affected. We did not see any issues with Windows Server 2012 R2, 2016 or 2019. Well the symptom is a reboot loop and below I have a sequence of what it looked like visually.

November 2019 updates caused Windows Server 2012 reboot loop
Restarting
Looking good so far …
OK, stage 2 of 4 … stlil seems OK.
November 2019 updates caused Windows Server 2012 reboot loop
Could still be OK … but no, its starts again in an endless loop.

So what caused this Windows Server 2012 reboot loop?

Fix

We turned of the virtual machines on which the November 2019 updates caused Windows Server 2012 reboot loop. We started them up again in “Safe mode” which completed successfully. Finally, we then did a normal reboot and that completed as well. All updates had been applied bar one. That was the 2019-11 Servicing Stack Update for Windows Server 2012 (KB4523208).

We manually installed it via Windows update and that succeeded.

When reading the information about this update https://www.catalog.update.microsoft.com/ScopedViewInline.aspx?updateid=5fa2a68f-e7cd-43c7-a48a-5e080472cb77 its states it need to be installed exclusively.

Maybe that was the root cause. It got deployed via WSUS with the other updates for November 2019.

Anyway, all is well now. I remind you that we have at least 2 ways of restoring those virtual machines. In case we had not been able to fix them. Have known good backups and a way to restore them people.

Conclusion

We fixed the issue and patched those servers completely. So all is well now. Except for the fact that they now , once more, have been urged to get of this operating system asap. You don’t go more than N-2 behind. It incurs operational overhead and risks. They did not test updates against this old server VM and got bitten. Technology debt without a plan is never worth it.

Quick Assist: CredSSP encryption oracle remediation Error

In the past 12 hours I’ve seen the first mentions of people no longer being able to connect over RDP via a RD Gateway to their clients or servers. I also got a call to ask for help with such an issue. The moment I saw the error message it rang home that this was a known and documented issue with CredSSP encryption oracle remediation, which is both preventable and fixable.

The person trying to connect over RD Gateway get the following message:
[Window Title]
Remote Desktop Connection
[Content]
An authentication error has occurred.
The function requested is not supported
Remote computer: target.domain.com
This could be due to CredSSP encryption oracle remediation.
For more information, see
https://go.microsoft.com/fwlink/?linkid=866660
[OK]

image

Follow that link and it will tell you all you need to know to fix it and how to avoid it.
A remote code execution vulnerability (CVE-2018-0886) exists in unpatched versions of CredSSP. This issue was addressed by correcting how CredSSP validates requests during the authentication process.

The initial March 13, 2018, release updates the CredSSP authentication protocol and the Remote Desktop clients for all affected platforms.
Mitigation consists of installing the update on all eligible client and server operating systems and then using included Group Policy settings or registry-based equivalents to manage the setting options on the client and server computers. We recommend that administrators apply the policy and set it to  “Force updated clients” or “Mitigated” on client and server computers as soon as possible.  These changes will require a reboot of the affected systems. Pay close attention to Group Policy or registry settings pairs that result in “Blocked” interactions between clients and servers in the compatibility table later in this article.

April 17, 2018:
The Remote Desktop Client (RDP) update update in KB 4093120 will enhance the error message that is presented when an updated client fails to connect to a server that has not been updated.

May 8, 2018:
An update to change the default setting from Vulnerable to Mitigated (KB4103723 for W2K16 servers) and KB4103727 for Windows 10 clients. Don’t forget the vulnerability also exists for W2K12(R2) and lower as well as equivalent clients.

The key here is that with the May updates change the default for the new policy setting changes the default setting from to mitigated.

Microsoft is releasing new Windows security updates to address this CVE on May 8, 2018. The updates released in March did not enforce the new version of the Credential Security Support Provider protocol. These security updates do make the new version mandatory. For more information see “CredSSP updates for CVE-2018-0886” located at https://support.microsoft.com/en-us/help/4093492.

This can result in mismatches between systems at different patch levels. Which is why it’s now more of a wide spread issue. Looking at the table in the article and the documented errors it’s clear enough there was a mismatch. It was also clear how to fix it. Patch all systems and make sure the settings are consistent. Use GPO or edit the registry settings to do so. Automation is key here. Uninstalling the patch works but is not a good idea. This vulnerability is serious.

image

Now Microsoft did warn about this change. You can even read about it on the PFE blog https://blogs.technet.microsoft.com/askpfeplat/tag/encryption-oracle-remediation/. Nevertheless, many people seem to have been bitten by this one. I know it’s hard to keep up with everything that is moving at the speed of light in IT but this is one I was on top of. This is due to the fact that the fix is for a remote vulnerability in RDS. That’s a big deal and not one I was willing let slide. You need to roll out the updates and you need to configure your policy and make sure you’re secured. The alternative (rolling back the updates, allowing vulnerable connections) is not acceptable, be vulnerable to a known and fixable exploit. TAKE YOUR MEDICIN!  Read the links above for detailed guidance on how to do this. Set your policy on both sides to mitigated. You don’t need to force updated clients to fix the issue this way and you can patch your servers 1st followed by your clients. Do note the tips given on doing this in the PFE blog:

Note: Ensure that you update the Group Policy Central Store (Or if not using a Central Store, use a device with the patch applied when editing Group Policy) with the latest CredSSP.admx and CredSSP.adml. These files will contain the latest copy of the edit configuration settings for these settings, as seen below.

Registry
Path: HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters
Value: AllowEncryptionOracle
Date type: DWORD
Reboot required: Yes

Here’s are the registry settings you need to make sure connectivity is restored

Everything patched: 0 => when all is patched including 3rd party CredSSP clients you can use “Force updated clients”
server patched but not all clients: 1 =>use “mitigated”, you’ll be as secure as possible without blocking people. Alternatively you can use 2 (“vulnerable”) but avoid that if possible  as it is more risky, so I would avoid that.

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP][HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters]
“AllowEncryptionOracle”=dword:00000001

So, check your clients and servers, both on-premises and in the cloud to make sure you’re protected and have as little RDS connectivity issues as possible. Don’t forget about 3rd party clients that need updates to if you have those! Don’t panic and carry on.

PowerShell Script to Load Balance DNS Server Search Order

Load Balance DNS Server Search Order

DNS servers need to be configured correctly, operate perfectly and respond as fast as possible to their clients. For some applications this is critical, but many have a more relaxed attitude. Hence a DNS Server has a full second to respond to a query. That means that even when you have 2 DNS servers configured on the clients the second will only be used when the first is not available or doesn’t respond quickly enough. This has a side effect which is that moving traffic away from an overloaded DNS servers isn’t that easy or optimal. We’ll look at when to use a PowerShell script to Load balance DNS server search order.

DHCP now and then

The trick here is to balance the possible DNS servers search order amongst the clients. We used to do this via split scopes and use different DNS servers search orders in each scope. When we got Windows server 2012(R2) we not only gained policies to take care of this but also DHCP failover with replica. That’s awesome as it relieves us of much of the tedious work of keeping track of maintaining split scopes and different options on all DCHP servers involved. For more information in using the MAC addresses and DCHP policies to load balance the use of your DNS servers read this TechNet article Load balancing DNS servers using DHCP Server Policies.

Fixed IP configurations

But what about servers with fixed IP addresses? Indeed, the dream world where we’ll see dynamically assigned IP configuration everywhere is a good one but perfection is not of this world. Fixed IP configurations are still very common and often for good reasons. Some turn to DCHP reservations to achieve this but many go for static IP configuration on the servers.

image

When that’s the case, our sys admins are told the DNS servers to use. Most of the time they’ll enter those in the same order over and over again, whether they do this manual or automated. So that means that the first and second DNS server in the search order are the same everywhere. No load balancing to be found. So potentially one DNS server is doing all the work and getting slower at it while the second or third DNS servers in the search order only help out when the first one is down or doesn’t respond quickly enough anymore. Not good. When you consider many (most?) used AD integrated DNS for their MSFT environments that’s even less good.

PowerShell Script to Load Balance DNS Server Search Order

That’s why when replacing DNS Servers or seeing response time issues on AD/DNS servers I balance the DNS server search order list. I do this based on their IP address its last octet. If that’s even, DNS Server A is the first in the search order and if not it’s DNS Server B that goes in first. That mixes them up pseudo random enough.

I use a PowerShell script for that nowadays instead of my age-old VBScript one. But recently I wanted to update it to no longer use WMI calls to get the job done. That’s the script I’m sharing here, or at least the core cons pet part of it, you’ll need to turn it into a module and parameterize if further to suit your needs. The main idea is here offering an alternative to WMI calls. Do note you’ll need PowerShell remoting enabled and configured and have the more recent Windows OS versions (Windows Server 2012 and up).

cls
#The transcipt provides a log to check what was found and what changed.
Start-Transcript -Path C:\SysAdmin\MyDNSUpdateLog.txt #
$VMsOnHost = (Get-VM -ComputerName MyHyperVHostorClusterName).Name

foreach ($VM in $VMsOnHost)
{
    Invoke-Command -ComputerName $VM -ScriptBlock {

    #This function checks if the last octet of an IP address is even or not
    Function IsLastOctetEven ($IPAddress)
        {
             #$FirstIP
             $Octets = $IPAddress.Split(".")
             #$Octets[3] #0 based array, grab 4th octet

             #See if 4th octect is even
             $Boolean = [bool]!($Octets[3]%2)
             if ($Boolean)
             {
                 Return $Boolean
                 #write-host "even"
             }
             else
             {
                 Return $Boolean
                 #write-host "odd"
             }
        }

        $OldDns1 = "10.15.200.10"
        $OldDns2 = "10.15.200.11"
        $NewDns1 = "10.18.50.110"
        $NewDns2 = "10.18.50.120"

        $NicInterfaces = Get-DnsClientServerAddress

        foreach ($NICinterface in $NicInterfaces)
        {
                #Here we filter out all interfaces that are not used for client/server connectivity.
                #Cluster Interfaces, HeartBeats, Loop back adapters, ...
                #We also filter out IPv6 here as this is for a IVp4 environment.
             if($NicInterface.InterfaceAlias -notmatch "isatap" -and $NicInterface.InterfaceAlias -notmatch "Pseudo" `
                -and $NicInterface.InterfaceAlias.Contains("Local Area Connection*") -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("KEMP-DSR-LOOPBACK") -ne $True `
                -and $NicInterface.InterfaceAlias.ToLower().Contains("Heartbeat".Tolower()) -ne $True `
                -and $NicInterface.InterfaceAlias.Contains("NLB-PRIVATE") -ne $True-and $NicInterface.AddressFamily -ne "23")
             {

                $Output = "Hello from  $env:computername" + $NICinterface.InterfaceAlias
                write-Output $Output            
           
                $Output = $NicInterface.InterfaceAlias +": DNS1=" + $NicInterface.ServerAddresses.GetValue(0) + " & DNS2=" +  $NicInterface.ServerAddresses.GetValue(1)
                write-Output $Output

                If (($NicInterface.ServerAddresses.GetValue(0) -like $OldDns1 -or $NicInterface.ServerAddresses.getvalue(0) -like $OldDns2) -and ($NicInterface.ServerAddresses.getvalue(1) -like $oldDns1 -or $NicInterface.ServerAddresses.getvalue(1) -like $OldDns2))
                {
                    #If the IP address is DHCP assignd, leave it alone,
                    #that's handled via DHCP policies on the MAC address
                    $GetNetIPInfo = Get-NetIpAddress -InterfaceIndex  $NicInterface.InterfaceIndex
                     if ($GetNetIPInfo.PrefixOrigin -like "DHCP")
                     {
                        $VM                   
                        write-output "DHCP address - leave it alone"
                     }
                     Else
                     {
                         $IPAddresses = $GetNetIPInfo.IPv4Address
                         $FirstIP = $IPAddresses[1] #1 based array
                 
                         if (IsLastOctetEven($FirstIP)){
                            $VM
                            write-output "EVEN 4th IP octet => so DNS search order becomes $NewDns1 , $NewDns2"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns1,$NewDns2)
                         }
                         else
                         {   
                            $VM
                            write-Output "ODD 4th IP octet => so DNS search order becomes $NewDns2 , $NewDns1"
                            Set-DnsClientServerAddress -InterfaceIndex $NicInterface.InterfaceIndex -ServerAddresses ($NewDns2, $NewDns1)
                         } 
                         $NicInterface |  Select-Object -ExpandProperty ServerAddresses    
                     }
                }
                else
                {
                    $VM
                    write-Output "Existing DNS values not like expected old values. They are propably already changed"
                }        
            }
        }
    }
}
Stop-Transcript

 

Microsoft Security Bulletin MS16-045

Just a quick post to make sure you all know there’s an important security update for Hyper-V in the April 2016 batch of updates.

Please review Microsoft Knowledge Base Article 3143118  and Microsoft Security Bulletin MS16-045 – Important for details. Realize thatthis ios one you’d better test en deploy asap. In my deployments I have not seen or heard o any issues with the update so far.

Why this little shout out? Well it’s a remote code execution vulnerability that can leverage the guest to run code on the host.

This security update resolves vulnerabilities in Microsoft Windows. The most severe of the vulnerabilities could allow remote code execution if an authenticated attacker on a guest operating system runs a specially crafted application that causes the Hyper-V host operating system to execute arbitrary code. Customers who have not enabled the Hyper-V role are not affected.

It affect Windows 8.1 (x64), Windows Server 2012, Windows Server 2012 R2, and Windows 10 (x64). Test and patch a.s.a.p. When you’re a hosting provider, I hope you’re already on top of this one.