Exchange 2007 & 2010 Event ID’s: 2601, 2604, 2501 & Users Can’t Access Mailboxes / Public Folders On My Day Off

I took the day off as I needed some time to deal with government administration. Good thing this is a blog about IT issues because holey crap what a time eating, confusing and rather pointless mess government administration can be. The process to get to the desired outcome is very tedious, prone to misunderstanding & pretty inefficient . What the entire duration of the process and the number of administrative entities involved contribute to the desired result is a mystery. It’s pure show and window dressing. But OK, we took the day of to finally get it all sorted after 5 months of patiently waiting for this day.

So I sleep until 08:00, get up and head for the kitchen for a jar of coffee. With the only Java I truly like in my hand I make my way to the home office. I check mails/alerts from System Center, Support Requests etc. I’m like a responsible guy dude, even when I need a day off. I do monitor the condition of my projects in production and I do step in when needed and document my findings. It keeps me honest when I design and sell my solutions. Beware of some architects that are not the ones having to deal with the crap architectures they design, they are often empty suits. Anyway, I see an issue that could be a warning of more to come. Someone has a problem with Outlook 2007 which reports the following error (translation from Dutch):

“Unable to expand the folder. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance.(/o=<DOMAIN>/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=servers/cn=<dagmember1>)”

Now I know that user. Smart, diligent and reliable. That user even provides the relevant and necessary information in their support request. Yes they do exist and HRM should hire those exclusively. So in combination with that error we knew we did not have an PEBKAC or ID-10T on our hands but a real issue.

I quickly check that DAG member node Outlook of that user is trying to connect to but I know that due to maintenance their mailboxes currently reside on another member of the DAG. So i could very well be just the public folders. Bingo. A quick test reveals this to be the case. Also the Windows 2008 R2 server and Exchange 2010 itself are running perfectly fine, happy as can be, except on that one node we see the Application Event Log messages:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2601
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When initializing a remote procedure call (RPC) to the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the SID for account <WKGUID=XXXXXXXXXXNOREALIDXXXXXXXXXXXXXX,CN=Microsoft Exchange,CN=Services,CN=Configuration,…> – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2604
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.demo
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When updating security for a remote procedure call (RPC) access for the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the security descriptor for Exchange server object DAGMEMBER1 – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2501
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). The site monitor API was unable to verify the site name for this Exchange computer – Call=DsctxGetContext Error code=8007077f. Make sure that Exchange server is correctly registered on the DNS server.

I think I’m OK when I see the possible cause. Why? Because I also know even if that probable cause isn’t the problem, it’s a hiccup I’ve seen before and I know how to fix its one. When you search those errors you can find a TechNet article describing a possible cause: “An inactive network connection is first on the binding list” http://technet.microsoft.com/en-us/library/dd789571(EXCHG.80).aspx. The fix is quite simple. Correct the NIC order and restart the MSExchange ADTopology Service. I had my scare about Active Directory and DNS horrors the first time I ever saw this one. So no gut wrenching panic here 🙂

But why do servers ever get in to this state when the NIC ordering is just fine? We did some firmware and upgrade recently after hours but that didn’t affect the NIC binding order. Now I’m pretty weird at times but I still know what I’m doing. Those NIC where OK when I configured those servers. Checking that has become a second nature on multi homed and clustered servers. I also remember happening this to me once before somewhere in February 2010 with another setup of Exchange 2010 on Windows 2008 R2. And in that case the NIC order in the binding list was also OK. I checked back then as well just to make sure. But since I build those Exchange 2010 setups myself I just know they are close to godliness both in design and implementation :-). Back then the issue went away by restarting the server, restarting the MSExchange ADTopology Service will do however, and the problem never came back. For some reason the AD Site information query fails. Now Windows retries and is OK after a while. Exchange, tries to get the AD Site information once, fails and keeps thinking there is an issue. With as a result clients have no connectivity and those errors that initially make you think you could have DNS issues, AD problems etc. But fortunately it’s a lot less serious.

So when the NIC binding order is OK why does this happen? I can’t tell you for sure but I do know that I’m not the only one (not that weird after all) since Microsoft published KB Article “MSExchange ADAccess Event ID’s 2601, 2604, 2501” http://support.microsoft.com/kb/2025528 . This article is a so called FAST PUBLISH from Microsoft Support and states that the issue only occurs on Windows 2008 R2 and that it affect Exchange 2007 and Exchange 2010. The cause? Well this is where they provide only what I already knew:

“During a restart of the server, the operating system queries Active Directory to get its AD Site information.  On a Windows 2008 R2 server, this will sometimes fail.  As the Exchange services are starting, it also will do a query for its AD Site and that too will fail. Windows will continue to try and determine its AD Site name and will eventually succeed.  However, Exchange does not re-try the query and the above errors are logged in the application log every 15 minutes.”

And yes the workaround/fix is also nothing new:

“After the server has been up for a minute or two, run NLTest /DSGetSite to verify that that the proper Active Directory Site is being returned by Windows.  Once that has been verified, restart the MSExchange ADTopology Service.”

Do note that this will also restart a slew of dependant Exchange services so it takes a little while.

  • Microsoft Exchange Transport Log Search
  • Microsoft Exchange Transport Log
  • Microsoft Exchange Service Host
  • Microsoft Exchange Search Indexer
  • Microsoft Exchange Replication Service
  • Microsoft Exchange Mail Submission
  • Microsoft Exchange Mailbox Assistants
  • Microsoft Exchange File Distribution
  • Microsoft Exchange EdgeSync
  • Microsoft Exchange Anti-spam Update

So after some manual intervention we had the users back in business. And all is well for them, as they rise and sleep under the watchful eye of a bunch of good IT Pro’s who’ll protect them form further harm and problems 😉 Now I need to get an auto fix for this I think until Microsoft fixes this one for good. SCOM where are you? No, no, no … It’s my day off for getting that administration done!

12 thoughts on “Exchange 2007 & 2010 Event ID’s: 2601, 2604, 2501 & Users Can’t Access Mailboxes / Public Folders On My Day Off

  1. I have been fighting this issue for several weeks and found your article. Thanks for posting this information.

    My setup is Exchange 2010 SP1 running on Server 2008 R2 hosted on Hyper-V 2008 R2.

    I have talked with another Exchange admin who did not have this problem on his Ex 2010 implementation on vSphere 4 and found that these error/warning events do not appear if the IP address on the primary NIC (first in the binding order) is statically assigned.

    I just found that my hub and mailbox servers were generating these events just like my CAS server was but they are now all running with static addresses. I have validated the static IP fix with a restart of the CAS server this morning and no further events have been posted – only the 2080 event every 15 minutes.

    • Your welcome.Why didn’t you use use static IP addresses for all Exchange servers from the start? I can see how clients & server get this error when the IP address changes & they use the cache or all DNS changes haven’t replicated yet.For the DAG Cluster IP address it does not matter if you can allow a dynamic IP address there (firewall, security policies) but for the other ones I highly recommend it. The issue indeed doesn’t (or shouldn’t) appear if the NIC binding order is correct normally but there will need to be a fix for the KB article 2025528 by Microsoft. Until then, even with static IP addresses you’re not 100% safe. You’ll need to monitor it.

      • Sorry for the confusion, I am at a major University adn we have Unix/Linux DNS and all of the addresses are static when delivered by DHCP. So that leads me to believe that the problem is either with the IP subsystem not getting the address from DHCP before the Exchange AD Topology needs to make its check or with 2008 R2 not initializing the NIC correctly or in a timely manner.

        • I understand the situation better now. Overhere a lot of universities also use MAC address reservations as a means of controlling IP addresses/client connectivity. Any other things in play? Port authentication? Rapid spanning tree is enabled?

  2. What if your getting these errors and there is no binding? Any idea’s on that?

    Got 2 NIC’s using different subnets. Thats about it. DNS records on DC’s are just fine.

    • As stated in the blog post, this error can happen in Windows 2008 R2 even without a binding order issue. Also please not that with 2 NICs on different subnets yoy must make sure the domain bound NIC is the first in the binding order. Binding order iis in play the moment you have 2 NICs regardless of subnet.
      “During a restart of the server, the operating system queries Active Directory to get its AD Site information. On a Windows 2008 R2 server, this will sometimes fail. As the Exchange services are starting, it also will do a query for its AD Site and that too will fail. Windows will continue to try and determine its AD Site name and will eventually succeed. However, Exchange does not re-try the query and the above errors are logged in the application log every 15 minutes.”

  3. Thank you *so* much for posting this. So much hair pulling (only about 10 hours) and googling for my issue – lead me here.

    Like yours, mine presented that “well everything else in exchange works – look EXBPA gives me a clean bill of health” but RPC decided not to.

    And we all know just how stupid it is to type “outlook fails to start” into Google.

    I’m pretty ropable that there is *still* no fix to this, and the explanation of “sometimes it just happens” is pretty weak.

  4. Hi workinghardinit,
    thanks for your artical. i am having similar issue. i am in process of transition form 2003 to 2010 exchange. currenlty only inbound and outbound emails are going via exchange 2010. All maiil boxes and public folder are still on 2003. i notice these error log could you please advise what is best to do. Server got two NIC but only one is used other one is disabled. these are the logs i am getting.

    Log Name: Application
    Source: MSExchange ADAccess
    Date: 13/06/2012 09:23:51
    Event ID: 2601
    Task Category: General
    Level: Warning
    Keywords: Classic
    User: N/A
    Computer: SVR-EXC01.fma.uk.com
    Description:
    Process MSEXCHANGEADTOPOLOGY (PID=1396). When initializing a remote procedure call (RPC) to the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the SID for account – Error code=8007077f.
    The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

    Log Name: Application
    Source: MSExchange ADAccess
    Date: 13/06/2012 09:23:51
    Event ID: 2604
    Task Category: General
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: SVR-EXC01.fma.uk.com
    Description:
    Process MSEXCHANGEADTOPOLOGY (PID=1396). When updating security for a remote procedure call (RPC) access for the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the security descriptor for Exchange server object SVR-EXC01 – Error code=8007077f.
    The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

    Log Name: Application
    Source: MSExchange ADAccess
    Date: 13/06/2012 09:23:52
    Event ID: 2501
    Task Category: General
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: SVR-EXC01.fma.uk.com
    Description:
    Process MSEXCHANGEADTOPOLOGY (PID=1396). The site monitor API was unable to verify the site name for this Exchange computer – Call=DsctxGetContext Error code=8007077f. Make sure that Exchange server is correctly registered on the DNS server.

    after all above this gets logged
    Log Name: Application
    Source: MSExchange ADAccess
    Date: 13/06/2012 09:25:29
    Event ID: 2080
    Task Category: Topology
    Level: Information
    Keywords: Classic
    User: N/A
    Computer: SVR-EXC01.fma.uk.com
    Description:
    Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1396). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
    (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
    In-site:
    SVR-DC01.fma.uk.com CDG 1 7 7 1 0 1 1 7 1
    SVR-DC02.fma.uk.com CDG 1 7 7 1 0 1 1 7 1
    Out-of-site:

    hope you can help me with this.

    Thanks

Leave a Reply