Active Directory Replication Status Tool’s rise, fall, and rebirth
For many years the Active Directory Replication Status Tool has been a trusty companion for many IT Pro. That includes seasoned systems engineers as well as accidental Active Directory administrators. It was an easy way to get a quick and good idea of the replication health of your Active Directory forest or domain.
Sure, repladm is our friend and keeps doing its job with us in the trenches. But I would say that the Active Directory Replication Status Tool is a lot less scary for people. Especially those who incidentally need to find out what issues to address where. It is a non-scarry, kind wrapper to visualize the results and see where we need to pay attention.
It had some quirks, like the ridiculous need to download it again when the license (certificate) expired (I never liked the Sirona DLL hack in a production environment). If other issues arose, I blogged on how to work around them, like in Microsoft Active Directory Replication Status Tool won’t upgrade – Working Hard In ITWorking Hard In IT. But that was minor compared to the sage of the last 18 months. Let’s quickly look at the Active Directory Replication Status Tool’s rise, fall, and rebirth.
The fall
Once in a while, an update to Windows or .NET broke the application. Normally, a fix would follow soon, and everyone was happy again. Last year in the spring or early summer of 2022, the tool was broken for months. Finally, somewhere in August 2022, we got a new version that worked, as far as my experience goes, for about 4 to 6 weeks, It then broke again, and all it ever did after was crash. The cert also expired again but who cared? It was no longer functional anyway,
Well, I cared, and I gave a lot of feedback via Twitter and to Microsoft via e-mail.
What!!??? Is this how it ends? Abandoned in a ditch somewhere?
Important
As of June 2nd, 2023, the Active Directory Replication Status Tool is no longer available for download. The following article is provided for historical purposes only.
Are you kidding me? There went another e-mail right to Redmond! Was this the sad end? I got a reply that not all was lost with a link.
The rebirth
The link I got is this one GitHub – ryanries/ADReplStatus: AD Replication Status Tool. People meet Ryan Ries, an Escalation Engineer at Microsoft who has some very useful and handy private projects to share with the world. ADReplStatus is one of the more recent ones.
I downloaded it and started testing it in the lab. That went well and has already found its way to two production environments. Here is a screenshot from my lab environment!
Thank you, Ryan Ries, for helping your customers in your free time with your private projects. You have made my IT Pro existence a bit easier again instead of more difficult. It is appreciated! Thank you, Ned Pyle, for bringing this GitHub repo to my attention. Download your copy here Release v1.3.1 · ryanries/ADReplStatus · GitHub.
What I did find interesting was that the cert expiration time bomb was also an internal issue. That and the fact that the application had no maintainers. But that was obvious to us all.
The old version of the tool had a time bomb in it – an expiring SSL certificate – that rendered the app unusable sometime around September 2022. Only through great effort internally were we able to periodically renew this certificate and republish the app, and the app was architected in such a way that excising the signing certificate check was more work than just rewriting the whole thing. (I know about the Sirona DLL hack, but still.) The old tool had no active maintainers and no one who was still around was familiar with the app’s internals or source code.
Since Azure does not offer full-blown FTPS as a service, I usually migrate existing Windows Server-based FTPS solutions to IAAS in Azure. That means a virtual machine in Azure. Since many such implementations or lift and shift projects leverage Active Directory integrated authentication. As a result, they fit well into many projects where a small percentage of on-premises workloads have to live on in an IAAS ADDS Azure environment anyway.
The reasons for this vary, but most often, we see that these cannot be refactored or replaced by cloud-native solutions. This is due to the technology required or because of budget constraints. Ultimately, one cannot expect to swap all currently used technology with newer technology all of the time. Such is the reality of real-world IT unless you cherry-pick projects to only deal with cloud-native solutions. Since Azure landing zones often leverage Azure Firewall, we have arrived at the topic of this blog post. We need to do an FTP configuration for passive FTP over SSL support in Azure Firewall.
Passive FTP over SSL support in Azure Firewall
A quick search on the internet leads to results quickly confirming that Azure Firewall supports passive FTP. As expected, this requires DNAT rules for port 21 and the chosen port range for the FTP data channel. When looking at the official Microsoft documentation, this is confirmed. They mention that you must “configure FTP server to accept data and control channels from different source IP addresses.” I glanced over that one, and I should not have done that. On the other hand, that line is about accessing an FTP server on the Internet with the FTP client behind the Azure Firewall. That is the Outbound VNet – Internet (FTP client in VNet, server on Internet) section in the Microsoft docs.
Here I run the FTP Server behind Azure Firewall (a Secured Hub in Azure VWAN) with the client on the Internet. That is covered in the Inbound DNAT (FTP client on Internet, server in VNet) section which does not mention to configure FTP server to accept data and control channels from different source IP addresses.
The FTP Server
I stored the original FTPS virtual Machine (FTP via IIS on Windows Server 2022) to Azure using Veeam Backup and Recovery V12. That went smoothly, After fixing the IP configuration in the FTP sites, I was ready to configure the NSGs and Azure Firewall with all the needed rules.
Initial testing and troubleshooting
Using the Filezilla FTP client, I started testing the FTPS connectivity over Explicit FTPS, which initially seemed to work well. But browsing through the folders, I soon noticed that I had directory listing issues in about 50% of cases.I quickly tried this with WinSCP as an FTPS client to see if that worked as expeted, but I had the same issue.
I checked the Azure Firewall DNAT rules, and these were all in place and correct. Next, I checked the rules in the NSG on both the subnet and the NIC of the FTPS virtual machine to see if I missed something. Those were all in order as well. Next up was the Windows Firewall, but again, all was well. Finally, I tested FTP from a virtual machine in the same subnet, bypassing the Azure Firewall, and everything worked as expected. Comparing the FTP logs, I could see some differences in data channel logging but no errors.
So I tried Passive FTP without SSL. But again, I had the same issue. The error message is a bit different, as we don’t have TLS failures in this case.
OK, so what’s happening here?
Wireshark
As often, I resorted to using Wireshark in the FTPS virtual machine to get more insights into what was happening. Things got interesting very quickly.
Passive FTP without TLS
I am using passive FTP without TLS in the screenshots, which shows the FTP commands better.
Let’s look at the Wireshark capture we made when connecting and navigating the folders.
In the green rectangle, you see us connect and navigate to /Public/A-A-Test03. That works well. Observe that all traffic for control and data channel traffic (green fluorescent is data channel activity)comes from 10.250.250.69. Next, look at the red rectangle. Control (10.250.250.69) and data channel (red fluorescent, 10.250.250.70) traffic comes in from different IP addresses. We have found why the directory listing fails with IIS’s default and most secure data channel security configuration. We now also realize what “configure FTP server to accept data and control channels from different source IP addresses” means in the Azure Firewall documentation for Passive FTP.
Passive FTP with TLS
The same is true for passive FTP over TLS; the capture is slightly less readable and more verbose, but the directory listing fails again. Note that in FileZilla, we also see an extra error: “The data connection could not be established: ECONNABORTED – Connection aborted” as the TLS data connection also fails.
Note the red rectangle again where, at a given moment, while navigating directories, the data channel comes in from a different IP (red fluorescent), and the directory listing fails.
Searching for answers
The good news is that I know clearly when and why directory listing fails. That is possibly due to how the Azure Firewall redundant nodes behind the internal load balancers deal with Passive FTP. That i me speculating here, not an oficial statement ;-). The next step is figuring out how to solve this issue. I have no control over how Azure Firewall handles this. Replacing Azure Firewall with a 3rd party solution where we know this to work night not work in Azure. Also this should not be the first and final answer.
Some internet searching led me to a solution I was eager to try. Again, the information in the Azure Firewall docs should link to the IIS/FTP documentation on how to do this. The fact remains however that the Microsoft docsmention this for outbound access from a client on a VNET to an FTP server on the Internet. We do the opposite. We have our FTP server in a VNET behind Azure Firewall and the FTP client is on the internet. In that case, no mention is made of onfigure FTP server to accept data and control channels from different source IP addresses. Also note that in our case the source addresses are private, not public.
Read this article and note the following.
Implementing the workaround
Never the less, that does indeed ring a bell with what we see in Wireshark and whith what that line in the Azure Firewall documentation we glanced over so cavalierly. But it does not seem like a good idea for security reasons. But let’s try it! You can find where this setting lives at https://learn.microsoft.com/en-us/previous-versions/iis/settings-schema/ff713742(v=vs.90).
Navigate to the IIS Server and, under “Management” in the middle pane, open the configuration editor.
Once there, open the systems.applicationHost/Sites node and navigate to siteDefaults/ftpServer/security/dataChannelSecurity. Find the matchClientAddressForPasv element and set it to false.
Apply your configuration change, and your passive FTP(S) experience will consistently work when accessed over an Azure Firewall!
Now let’s try passive FTP again. With or without TLS, it now works like a charm. No more failures to retrieve directory listings at all! In Wireshark, you will still see that the data channel and control channel can come from a different IP (Redundant Azure Firewall with internal load balancers), but it no longer breaks FTP. Hoorah!
Musings on this setting
I have set up many FTPS solutions with Windows Server IIS FTP, and I have never needed to make this configuration change before to get passive FTP to work. Somehow, Checkpoint, ASA, Baracuda. My explanation is that these hide the two separate internal Ips of the redundant firewall by using a VIP. Fair enough. Ultimately, I was lucky to have always used FTP with IIS. That solution allowed us to solve the issue with a configuration change.
That used to be possible in other popular FTP server software like older versions of FileZilla Server,. Today this it is no longer available in recent versions. The reason is that this poses a security risk.
In the IIS documentation, they also mention that this is and are clear that it is preferable not to change this.
Ultimately, with Azure Firewall, in contrast to the MSFT docs we must set the matchClientAddressForPort to true under data channel security in IIS/FTP configuration. If not, client connections form the Internet to our passive FTP server behind an Azure Firewall will not work reliably. Maybe this is due to the fact this is a VWAN setup with a secured hub and not a tradditional hub/spoke network?
Finally, I am not looking forward to discussing this matter with the CISO office. The implication for the future of the Azure landing zone’s firewall of choice (i.e., Azure Firewall) is that it will be a point of discussion again. Now, I’ll leave it to security experts to decide how dangerous this setting is when the two servers involved are both nodes of a redundant firewall.
Conclusion
First of all, RTFM. That’s it. Read The Fucking Manual. Now that is all good and well, but you need to understand what you read. You also need to know what that means for all parts in your configuration. Combined with the fact that what I read was for the scenario where the FTP server lives on the internet and the FTP client sits behind Azure FIrewall. We have the opposite, but Wireshark does not lie and the “fix”makes it work. However, troubleshooting this made me comprehend it very well as I concluded what to try next before revisiting the documentation, The docs initially confused me as I was not supposed to apply this workaround in this scenario, but the hint about what to try was in there.
Another conclusion is that this gives ammo to the case for 3rd party firewalls with the Azure Firewall haters. They now have a less secure passive FTP server configuration which is always an excellent drum to beat on. Especially when other FTP servers no longer allow this configuration. To add insult to injury, Microsoft warns against making this FTP data channel security change.
You can say that this is a niche use case, perfection is not of this world, and that this setting’s security risk is benign. Maybe so, but there is always a but! First, FTPS is more common than you might want to hear or think for valid use cases. Second, it all adds up, and death by a thousand little cuts does occur.
I was re-awarded as a Microsoft MVP – Cloud and Datacenter Management 2023-2024
While I was on vacation I received an e-mail that I was re-awarded as a Microsoft MVP – Cloud and Datacenter Management 2023-2024. Needless to say that this put a big smile on my face, despite my vacation coming to an end. Yes it is and remains an honor to be awarded and recognized as a global expert in the community by Microsoft. We all like a compliment and some confirmation that what we do is valuable. As always, it is also a humbling experience, as you meet quite a lot of very talented people which can make that old imposter syndrome flare up.
For those not in the know, MVP in this case stands for Most Valuable Professional, not Player 😊.
This is my 12th Microsoft MVP award and it represents a long journey through technology in the Microsoft ecosystem. In tech things seems to change fast and tend not to last long. But it is the learning you do along the way that last through time. Nothing I have learned has ever become completely obsolete and the experience and inights, even from learning VBA and Visual Basic in the early days still help me today as I talk to developers whose Azure infrastructure I help optimize and secure.
I am grateful for the recognition and support of both the community and Microsoft. Especially to the people who originally saw my potential and supported my first-ever nomination as well as those that support me today. The award itself is not the goal, it is a by-product of being active in the global community. Engaging in and with the community helps us all learn, progress, and succeed in our jobs by sharing experiences, knowledge, and insights.
It is all about community
Without any doubt, no matter how good and smart you are or how hard you work, we need others to grow and flourish. You do not contribute to the community in isolation. Sure, IT pros do a lot of technology work individually while testing and learning in the lab, troubleshooting, and delivering projects. Sharing and learning in the community happens with and thanks to others. People who give us opportunities, support, and help us. That means fellow technologists, employers, colleagues, vendors, and our audience asking questions that make us learn even more as well.
Aditionally, let’s not forget about the people who are not in the spotlight. We get a lot of support from those providing locations, material, and logistic support. Many “smaller” businesses are often very generous in this regard. We are all “kids” getting raised in and by a village and in one form or another we all stand on the shoulder of giants. That’s why we all owe at least a little gratitude to all the people and organizations that help us progress. So to all those who do, thank you!
Some tips for businesses that employ MVPs
In the end, here is to another year of community activity where we all share, learn, and grow together. You can always start today. There is no age limit or right time. I was a long time lurker before I became publicly active voicing my opinions and sharing information. You can be an introvert or extrovert, young or old, and anything in between. All you need is the willingness to share and learn. Furthermore, you don’t need 50,000 followers on social media or whatever, you just need to make a difference.
Finally, I can only advise employers to support their community-minded employees. These are the people that learn, grow, adapt, and change with the needs and technology. Also, the incidental cost of travel, registration fees, and hotel nights are nothing compared to the return on investment. So let them attend some conferences and the MVP Summit
Well, to be honest, not supporting this would be penny-wise and pound-foolish. In the end that would mean you have no idea of the value of both developing and retaining talent. An MVP on staff will very easily safe you at least 30-40 days consulting per year. Think about that. Without any doubt, even at a medium rate of 125.00- $/hour means a serious ROI. Now add the cost of replacing such talent … it is a no brainer! You need to invest in on-board talent development anyhow and these are profiles that love to learn and share. That is a near to perfect as it gets people.
Presenting at and attending Experts Live Europe 2023
I am happy to share that I am both presenting at and attending Experts Live Europe 2023. It runs September 18-20 2023 in Prague.
Isidora Katanic (@IsidoraKatanic) is the lead organizer and driving force behind Experts Live Europe. She’s dedicated to making this one of the best Microsoft technology-focused conferences in Europe. When you look at the pre-conference and session calendar you can already see this. She and her team lined up everything to make the 2023 edition a great professional and community experience. Experts Live Europe is a two-day conference (three days with the preconference workshops) and is scheduled in Prague, September 18-20, 2023. It is the first edition since 2019 due to the Corona/Covid pandemic. I can share that I personally, and many others, are happy that this is possible again. Next to that, I am thrilled to share my research and expertise at this conference once more.
In my session “Azure Storage – The SMB over QUIC protocol is here!” I will be diving deeper into the why and how of SMB over QUIC.
This is a very powerful and promising, relatively recent addition to the SMB 3 stack. Once again it shows that file sharing is far from an obsolete protocol in the era of anything “cloud”.
Meet the experts and ask me anything galore
This conference is about you and me, about us, sharing insights, experiences, knowledge, and expertise. Both the concept and the setup of the conference facilitate this by design.
While I’m there, come say hi, and talk shop about networking, storage, clustering, Hyper-V, DevOps, Bicep, and Veeam data protection in on-premises, hybrid, and Azure scenarios. I’ll be around during the breaks for the “Ask The Expert” and at the dedicated speaker’s booth in the expo area. Now, next to a Microsoft MVP I am also a Veeam Vanguard. Veeam is a gold sponsor and I’ll be around their boot as well. So come find me if you want to talk about Veeam Backup & Replication, Hardened (immutable) repositories, and other related subjects.
I am there to learn as well
Finally, I also look forward to the sessions other speakers are giving. One of those sessions, “Azure Firewall: The Legacy Firewall Killer”, is presented by Aidan Finn (@joe_elway). That subject is both very interesting and a bit controversial. Many people know and master 3rd party firewall interfaces with their specific tooling and capabilities. While there is nothing wrong with that, many people scoff at Azure Firewall. But you should not write off Azure firewall with different products. This is especially true when you start delivering Azure Firewall via Infrastructure as Code (IaC).
Call to action
Do not delay! Register to attend Experts Live Europe and do not miss out on a ton of great sessions by expert speakers, networking with knowledgeable attendees, and talking shop with your fellow IT professionals, who are as passionate about technology as you. I look forward to seeing you there.
