Exchange 2007-2010 Public Folders Issues “The Active Directory user wasn’t found.”

I was working on an Exchange 2007 to Exchange 2010 project when we ran into trouble creating our first public folder database on an Exchange 2010 server. Mind you, this was just creating the database. We did not even set up replication for this database yet. All mailboxes still resided in Exchange 2007 databases pointing to an Exchange 2007 public folder. Very soon after creating the database we got notified users could no longer send mails to mail enabled public folders. The exact error was this:

554 5.6.0 STOREDRV.Deliver.Exception:ObjectNotFoundException; Failed to process message due to a permanent exception with message The Active Directory user wasn’t found.

Also browsing of the public folders in Outlook was slow and the application froze/hung. These issues where fixed very fast by getting rid of the still unused public folder database all together. Now we could commence our search for the root cause. The error seemed related to the issue described in Public Folder Replication Fails Due To Empty Legacy Administrative Group which can be found @ http://msexchangeteam.com/archive/2010/05/05/454821.aspx  The blog describes this error during replication:

Log Name: Application

Source: MSExchange Store Driver

Event ID: 1020

Level: Error

Description:

The store driver couldn’t deliver the public folder replication message “Hierarchy ([email protected])” because the following error occurred: The Active Directory user wasn’t found.

But apart from replication not working there were other, more severe issues impacting end users who can still all be on Exchange 2007. The hanging of the outlook clients and mail enabled folders no longer being available. Dave Stork blogged about this in http://blogs.dirteam.com/blogs/davestork/archive/2010/03/16/mail-enabled-public-folder-recipient-not-found.aspx

Now the first mentions of the replication issue have been reported back in November 2009 (see http://get-exchange.blogspot.com/2009/11/public-folder-mayhem-exchange-2010.html) but still hasn’t been fixed. For the moment that fix is planned to be included in E2K10 RU5. Currently we’re at RU3, so that might well be august 2010.

The workaround described in above mentioned blog posts works & is effective immediately. Now they described the issue and the fix very well but I’ll add to tips.

Tip 1

“Practical End User Friendly Detection” of this issue can be done using exfolders.exe. You can read more about this tool here: “Exchange, meet ExFolders” (http://msexchangeteam.com/archive/2009/12/04/453399.aspx).The error only occurs when you create a public folder on Exchange 2010 and can be very annoying for the users so I’ll share this tip with you. Download the tool here http://msexchangeteam.com/files/12/attachments/entry453398.aspx and install it on an Exchange 2010 server in the bin directory (follow the readme.txt and don’t forget to merge the .reg file or the tool will crash). Running exfolders.exe and connect against any Exchange 2007 public folder. When you get this error …

—————————

ExFolders

—————————

An error occurred while trying to establish a connection to the Exchange server. Exception: The Active Directory user wasn’t found.

—————————

OK  

—————————

… you know you are affected. Deleting the empty Servers containers from ALL legacy Administrative Groups fixes the error. You then can connect successfully to a Exchange 2007 public folder with exfolder.exe. Which is a cool way to test for this issue and if the fix works as you don’t need to create a public folder and possibly hinder you users.

Tip 2

Also note that you need to delete  (using ADSIEDIT) every empty servers container out of every legacy Administrative Group, not just or only the one in the “First Administrative Group”. Don’t worry if you renamed that one to something more descriptive, that doesn’t matter at all. All the servers containers in the legacy Administrative Group should be empty I you have no more E2K3 servers left in your exchange organization. Feel free to leave comments on your experiences.

Perversions of IT: License Dongles

The Sum of All Evil resulting in the Mother of All Fears

One of the most annoying things in the life of infrastructure support are the sometimes convoluted schemes software vendors come up with to protect their commercial interests. The “solution” we despise the most is a license dongle. It sounds easy and straight forward but it never ever is. Murphy’s Law!

Vendors think they are unique

What’s one dongle? What are you complaining about? Huh … we have customers who have more than more than 10. Only one application is actually capable of finding multiple dongles and able to survive the loss of one. Do we need multiple machines to be on the safe side? Yes. How many? You need at least two. Why to allow “quick” redirection if possible at all (using scripts, GPO’s, etc.)

We have a USB hub attached to the rack, so we can stick in many dongles. It saves on hardware. Now once in a while the USB will get a hick up and unplugging it and plugging in back it usually does the trick. As mentioned before, this is not very handy when that dongle is in a secured room or data center. To add insult to injury network issues might stop applications from working that otherwise would be fine just because they can’t find a license server.

Today they are almost always USB dongles, but the parallel port dongle was quite popular as well. If you’re lucky the vendor provides you with a USB dongle when your hardware doesn’t have a parallel port anymore. But sometimes you are not that lucky. Today most laptops and indeed most PC’s don’t come with a parallel port anymore. And no, to road warriors with laptops a USB to parallel converter isn’t really user friendly. Furthermore a dongle sticking out are accidents waiting to happen (broken, lost) and finally some laptops, especially the smaller ones road warriors love sometimes only have one USB port that is taken up by their G3 broadband stick. Heaven forbid that some users actually have two applications that require a dongle and an internet connection. These are only the silly, practical issues you have to deal with when license dongles come into the picture but it gets worse fast when things like uptime, redundancy, high availability etc. are added.

Some dongles are attached to a network license server; some are attach locally to the PC or server running the software. In all cases they need drivers/software to run. The server software is sometimes very basic, rather flimsy and error prone. Combine this with various vendors who all require license dongles with various brands/versions of software. USB ports itself have been known to malfunction now and then. As you can imagine you end up with a USB hub lit up like a X-Mas tree and lots of finger crossing.

Reliable, Predictable, High Available

Dongles and high availability are almost always by definition mutually exclusive. If they are not then it’s a very expensive we’ll work around it type of solution trying to make a dongles highly available. This is only possible when the software that requires it supports redundant setups. I have only seen this possibility once. With some vendors you need to buy extra licenses to get an extra dongles … if the software package is 50.000 € that hurts. Some vendors will show leniency in such cases, some are plain arrogant and complacent. The fact that they are still in business proves that the world runs on bull shit I guess. But even when you do have multiple dongles, multiple servers hosting the dongles most dongle protected software is no written to deal with losing a dongle so you can’t get high availability only some form of hot stand by. Supporting that is also a lot of work that requires cloning, copying, scripts, manual interventions etc.

Some vendors are even so paranoid they check 5 times within a minute … if they can still find a license and if not the application fails. That means that even rebooting a dongle host for patching or another intervention takes down the application. Zero tolerance for failure … dongle wise, power wise, human error wise … pretty unrealistic. And even if the dongle is attached to a redundant server in a secure data center you’ll see that the USB port will fail for some reason. The only reliable and predictable thing in this story is the fact that you will fail.

Security

This is a good one. Do you really want to hurt some companies I work/consult for? Walk around their offices or data center, unplug any dongle you can and flush ‘m down the toilet. That will take ‘m down for a while. Yes I known, they should have better physical security. Either we have a license dongle on a network server which makes it a bit more realistic or we lock up all those PC’s in a secure room. That is not always feasible, either due to cost or just practically. And by the way that doesn’t protect you from a pissed of contractor or employee that has access. Even when security cameras can identify them fast the damage is already done.

Dongles sticking out of a 1 U server prevent the use of bezels to help lock down access to the server. The USB ports in the back are used for KVM over IP or keyboard and mouse.

In some models you can try to plug the dongle into an USB port inside the server chassis but than the old trick of unplugging/inserting the dongle when it goes haywire isn’t that easy anymore, let alone the fact that the dongle sits in a data center somewhere so getting to it might not be feasible so you need to allow someone to access the server to be able to get to the dongle.

Dongles and Virtualization

When you need to virtualize server applications that need a locally attached dongle you need to start looking for USB over Ethernet solutions that are reliable. When you find one you need to manage it very carefully and well. You need to manage the versions of the server software and the client software. We’ve seen network connectivity loss when the versions don’t match up, even if the software didn’t complain about different versions. You need to test its stability, have extra hardware and extra dongles for testing as not all dongles respond well to this type of setup. We can’t afford to bring down production environments with USB over Ethernet software “upgrades of faith”. The need for dongles adds an extra layer of complexity and management, one that is very error prone and hard to make redundant let alone highly available. It’s not a pretty picture.

We used to buy Fabulatech for such implementations. Version 4.1.1 was rock solid but ever since version 4.2/4.3 & 4.4 Beta they have brought us nothing but “Blue Screen of Death” hell. We now implement KernelPro (2.5.5) which seems to be functioning very well for the moment.

Dongles are a virtualization show stopper in some environments due to these issues and risks. Behold dongle David brings down virtualization Goliath.

The Bottom Line

The biggest perversion, in what is essentially a big mess, is the fact that the only people affected by this are your paying customers. Software vendors should take note of the fact that paying clients despise your convoluted, error prone, “accidents waiting to happen” dongle licensing schemes. You not only have no clue what it means to run reliable IT operations but you don’t even care about your customer’s needs. There is only one rule. Software & hardware should work under all circumstances without the need for dongles. That darn piece of 50 cent plastic & silicon could well bring an entire application down. Let us just hope that it isn’t the geo routing software for 911 or 112 services.

There are two possibilities when you sell software. One is that your application is very popular and as such is being “keygenned” and cracked all over the place and the only ones you’re hurting are your paying customers. The other possibility is that your software is so unique and expensive it’s only bought by specialized firms and entities that couldn’t even operate it without being exposed as thieves. Stop fooling yourselves and stop making life hell for your customers. Protect you rights as well as you can but not at the expense of paying customers. You might even sell more if you care about their needs. Go figure. Maybe I’m just to demanding? Nah!