No ADSI Edit required to fix “Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).”

During the removal of the last Exchange 2007 SP3 Mailbox server after completing the transition of Exchange 2007 to Exchange 2010 SP1 we ran into the following well known error: Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).

The issue is that due to the coexistence of Exchange 2007 & Exchange 2010 we can no longer remove the public folder database with the Exchange 2007 GUI (EMC). But the public folder is not visible in the Exchange 2010 GUI (EMC) as it lives on an Exchange 2007 server. Trying to remove the public folder database manually using the Exchange 2007 GUI confirms this, you’ll get the same error.

This error has been described in some blogs as early as October 2009 on http://www.proexchange.be/blogs/exchange2010/archive/2009/10/28/remove-exchange-2007-mailbox-role-fails-with-error-object-is-read-only-because-it-was-created-by-a-future-version-of-exchange-0-10-14-0-100-0-current-supported-version-is-0-1-8-0-535-0.aspx and later on as recently as October 2010 on http://www.howexchangeworks.com/2010/10/object-is-read-only-because-it-was.html

The described solution/work around in these blogs get the job done perfectly, using ADSI Edit to delete the offending Exchange 2007 public folder database. It wouldn’t be the first time ADSI Edit saves an Exchange Consultants proverbial bacon. But if it can be done without using it I often recommend not to do it.  I’ve seen to many over eager deletions in ADSI Edit get people into trouble (like deleting a public folder database before it could be dumped safely without data loss).

For this problem, it’s not required to use ADSI Edit to get rid of the public folder on the Exchange 2007 Mailbox server. You can just fire up the Exchange Command Shell (EMS) in Exchange 2010 and execute following PowerShell command:


Remove-PublicFolderDatabase "E2K7MBXSGPublicFoldersStoreSGPublicFolders"

Confirm   4: Are you sure you want to perform this action?5: Removing public folder database "E2K7MBXSGPublicFoldersStoreSGPublicFolders".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The specified database has been removed. You must remove the database file located in K:E2K7DataSGPublicFoldersPublicFolderDatabase.edb from your computer manually if it exists. Specified database: PublicFolderDatabase

This works just fine. I have no objections using ADSIEdit when needed but I don’t advise using it to others unless really necessary.In this case it just isn’t needed to fix the problem. For good measure I also deleted the storage group in which the public folder lived. After that the install went well end without any issues.

Workaround for Exchange 2010 & Outlook 2003 Shared Calendars Connectivity Issues “The connection to the Microsoft Exchange Server is unavailable. Outlook must be online or connected to complete this action.”

This is a rather long blog about an issue we had during an Exchange 2007 to Exchange 2010 transition project @ a 1.500 mail boxes sized company. This was last summer but I needed to find the time to blog this so that it might help people who also need to find a solution. We ran in to this very annoying situation for users who still have outlook 2003 (SP3, fully patched) and were moved over to Exchange 2010 Roll Up 4. The users have rather a lot of shared calendars open & after accessing some of them (the sweet spot seemed to be around 6 to 7) they got this error:

"The connection to the Microsoft Exchange Server is unavailable. Outlook must be online or connected to complete this action."

Edit November 15th 2010: Microsoft released a KB article http://support.microsoft.com/kb/2299468 titled  “Error message when Outlook 2003 clients try to open multiple shared calendars in Exchange Server 2010: "The connection to the Microsoft Exchange server in unavailable. Outlook must be online or connected to complete this action" on this issue that is based on the work done on this case. The support engineer who worked on this case with me notified me by mail that the article had been released. Thanks Kovai! They only mention the throttling Policy as the /resetnavpane is not always required, this depends on the environment history. 

Edit February 3th 2011: The support engineer published a very content rich blog about all this and other possible causes of the error notification on his TechNet blog: Things you need to know about “The connection to the Microsoft Exchange server in unavailable. Outlook must be online or connected to complete this action” prompts in an Outlook 2003–Exchange 2010 world. It’s a must read for all people dealing with this error message.

When the user closes and reopens Outlook 2003 access sometimes worked for the calendar that threw the error before but than other calendars throw the error. So it becomes game of closing and opening Outlook 2003 in the hope you can open the desired calendar. This does not make users very happy as you can imagine. We stopped the migration to Exchange 2010 for users still on Outlook 2003. On the Exchange 2010 forums you’ll find http://social.technet.microsoft.com/Forums/en/exchange2010/thread/93db8cd7-3380-443f-8dbe-fbfb79cd9978 and http://social.technet.microsoft.com/Forums/en/exchange2010/thread/85c75bae-93dc-453e-be28-0425de3d5227 which both discuss this problem. It’s also pretty random. Sometimes it’s a shared calendar of a user on Exchange 2010 or on Exchange 2007. There is also a mention of this issue on others sites and one of them mentioned a private discussion with Microsoft about a fix coming in E2K10 Roll Up 5.I asked MS support later on if they had any knowledge of this and they said that they had none what so ever. By the way SP1 for Exchange 2010 arrived before we even got to roll up 5 for Exchange 2010 RTM.

We investigated and searched for a solution. One very promising work around was related to the throttling mechanism in Exchange 2010.

See also http://blogs.msdn.com/b/pepeedu/archive/2010/01/13/exchange-2010-client-access-throttling.aspx for more information on this. This is a good right up as well: http://eightwone.wordpress.com/2010/06/22/exchange-2010-throttling-policies-rtm-sp1/

The default throttling policy limits the RPC connections to 20. Now when Outlook 2003 opens a shared calendar it consumes a RPC connection. It doesn’t release that until outlook is closed. We taught we had a winner here as this could lead to the error while sending and receiving mail keeps working.

So in an attempt to fix the problem we tried this:

New-ThrottlingPolicy –name Outlook2003Calendar

Set-ThrottlingPolicy –identity Outlook2003Calendar –RCAMaxConcurrency 100

Set-Mailbox –Identity “annoyed user” –ThrottlingPolicy Outlook2003Calendar

We’re supposed to have some patience before this would work, due to Active Directory replication so we did. We even left it overnight. No joy. We also tried disabling throttling by using $NULL. Unfortunately this didn’t work either. We verified if the throttling was indeed the issue by counting the connections for a troublesome Outlook 2003 user and found that the error even occurred below the throttle limit. You can figure out the number of RPC connections by using:

Get-LogonStatistics –Identity <annoyed usser> | fl applicationid

Well we ran out of ideas (can you believe that?!) and so we called Microsoft Support to log a case. One of the goodies with our TechNet subscriptions via Software Assurance is that we have free support calls! Open-mouthed

The feedback was not coming fast. We also did not get any requests for more information. Not good sign. And when we pinged ‘m for info they acknowledged the bug and told us that it would probably be fixed in Exchange 2010 SP1. They were not 100% certain that our issue was due to this bug. But we did not get any request for further information or diagnostics. They asked us to go to Exchange 2007 SP3. At the time we were on Exchange 2007 SP2 Roll Up 4. There was not really an indication this would fix anything. The users who had the issue were the ones who had been moved to Exchange 2010. But we went along, put in the overtime past office hours and we went to Exchange 2007 SP3. As expected this did not help at all.

Then finally advanced support came in to play, now that was a different ballgame. Lots of questions request for logs, network traces, executing tests both client side and server side related. The engineer was really engaged and was working hard on this. He has my thanks and it took some time to do all the testing and work trough the results.

We found out some interesting things. When a user with a mailbox on Exchange 2007 or 2010 opens his shared calendars using Outlook 2003 the list is different than what you see in OWA or using Outlook 2007 and Outlook 2010. This is due to changes in the way those shared calendars are accessed. I was not able to find out any more details on the subject but this could be due to the fact that Outlook 2003 by default used referenced MDB model when additional calendars/mailboxes were opened. This feature isn’t supported in Exchange 2010 and due to the underlying design server side changes outlook 2003 now establishes more connections than it did with previous exchange versions. I assume it was still supported in Exchange 2007.

With Exchange 2010 the combination of the throttling policy and the changes in how shared calendars are accessed and the fact they contain “persistence information” caused this issue. We created a new throttling policy and i was necessary to delete all shared calendars from outlook, close outlook, start outlook and add them again. Manually this is done one calendar at the time and I pretty tedious. At least you can speed up things by using outlook.exe /resetnavpane. This will delete all the shared calendars for you and saves a lot of time. It also resets the entire navigation pane so any other “issues” lingering around are thrown out as well.

In summary (this an adaption of the support case conclusions. The escalation engineer working on the case was a great guy and he really put in an effort).

Problem:

  • When Outlook 2003 clients whose mailbox is on Exchange 2010 try to open additional calendars (on Exchange 2010 or on Exchange 2007) they get error popup "The connection to the Microsoft Exchange server in unavailable. Outlook must be online or connected to complete this action"

Observations:

  • The popups might happen while opening any additional calendar , no specific pattern on occurrence of pop up message based on specific number of additional calendars opened or a specific additional calendar/calendars being opened.
  • The issue didn’t happen when the mailboxes were on Exchange 2007, issue doesn’t happen with versions of outlook higher than Outlook 2003.
  • All the concerned mailboxes are on Exchange 2010.
  • There is an indication that mailboxes that existed in the organization since Exchange 2000 days face the problem. Mailboxes that were created since the introduction of Exchange 2007 and never had a mailbox prior to that version didn’t have the problem.

Causes:

  1. While the mailbox is on Exchange 2010, outlook 2003 will use Exchange 2010’s Address Book service to query the legacyDN of target shared mailboxes/calendars that needs to be opened and Exchange 2010 returns this as mungedDNs (not typical legacy exchange DNs) which forces outlook 2003 to think it’s a new server every time and establish more connections so eventually the default allowed maximum of 20 connections would exhaust.
  2. Outlook 2003 maintains hidden persistence messages containing various information about the shared mailboxes/calendars, such as the user name and the server where the mailbox resides. Thus it doesn’t have updated information and could very well reach a non-existing server as well.

Solutions:

  1. For the cause1, we set up a custom throttling policy with RCAMaxConcurrency set at 100 and applied this policy on mailboxes still using Outlook 2003.  For the change to take effect immediately, we need to force AD replication, restart of throttling service, RPCClientAccess service on CAS and MBX servers
  2. For the cause2, we either need to remove those stale entries and re-add the shared calendar entries (need to sync the deletion with server, then close and reopen outlook prior to re-adding entries again) or run ‘outlook.exe /resetnavpane’ switch.

Most issues with Outlook 2003 are well documented in KB articles and on the Microsoft Exchange Team Blog (http://msexchangeteam.com/) but not this one and it hurt us when the users complaints came in. We did not catch this one in the labs prior to the transition. We can’t win ‘m all, I know that. But the most obvious solution, upgrade to Outlook 2007/2010, is often not an acceptable option for financial, timing, practical and compatibility reasons. A couple of weeks ago I saw a tweet by Jetze Mellema about what he read somewhere “Friends don’t let friends upgrade to Exchange 2010 when using Outlook 2003.” I wouldn’t go that far. But this shared calendar issue should have been documented by now and made public I think.

New Version of ExFolders that is Exchange 2010 SP1 Compatible

I’ve mentioned the tool Exfolders before in https://blog.workinghardinit.work/2010/05/25/exchange-2007-2010-public-folders-issues-the-active-directory-user-wasnt-found/.   It’s a great tool and a worthy successor for PFDAVadmin.Now please note that when you upgrade to Exchange 2010 SP1 you’ll need to update the Exfolders tool as well. You can find the E2K10SP1 compatible version here: http://msexchangeteam.com/files/12/attachments/entry456255.aspx I’m happy to see that the new version of the tool is released in sync with the service pack. It’s a very handy an valuable tool to have. PFDAVadmin users already know this from experience. You can also use it to connect to an Exchange 2007 server but you need to run if on an Exchange 2010 server.

For more information about the tool take a look at this blog post by the Exchange Team: http://msexchangeteam.com/archive/2009/12/04/453399.aspx Don’t forget to read the instructions and follow them, especially regarding the import of the TurnOffSNVerificationForExFolders.reg file or the tool will crash.

EMC Does Not Show All Database Copies After Upgrade To Exchange 2010 SP1– Still Investigating

LATEST UPDATE March 9th 2011: I have installed Exchange 2010 SP1 Rollup 3 at customer and this did indeed fix this issue finally.

Updates to this post are being added as we get them below. Last update was October 13th 2010. The have identified the cause of the issue. It’s a case sensitivity bug. The fix is WILL be contained in Exchange 2010 Sp1 Roll Up 3? But they ARE working on a incremental update in between. See below for more details and the link to the Microsoft blog entry.

At a customer we have a 3 node geographically dispersed DAG. This DAG has two nodes in the main data center and one in the recovery site in another city, but it is in the same AD Site. This works but is not ideal as DAC in Exchange 2010 RTM presumes that the node will be in another Active Directory site. As you can imagine at that location we’re very interested in Exchange 2010 SP1 since that adds support for the DAC to be used with a geographically dispersed DAG node in the same Active Directory site.

We did an upgrade to SP1 following the guidelines as published in http://technet.microsoft.com/en-us/library/bb629560.aspx and we made sure all prerequisites where satisfied. We upgraded the backup software to a version that supported Exchange 2010 SP1 and made sure no services that hold a lock on Exchange resources are running. The entire process went extremely well actually. We did have to reconfigure redirection for OWA as the SP1 installation resets the settings on the Default Web Site on the CAS Servers. But apart from that we had no major issues apart from one very annoying GUI problem. Everything was fully functional, which we verified using EMS and by testing failovers. But in the EMC GUI we had the problem under Organization / Mailbox / Database Management we only see the database copies listed on one server and not on all tree.

When you check the properties of the databases shows all three servers that are hosting copies. We used EMS commands to test for problems but it all checks out and works. Failing over a server works, both in the GUI and in PowerShell, just like activating a database.

The same issue can be seen in Server Configuration /Database Copies as demonstrated in the screenshots below. In the first figure you we selected the mailbox server where the database copies are visible.

But on the other two nodes nothing shows up, just “There are no items to show in this view”.

No errors in the vent logs or installation logs. All is working fine. So what gives? We tried all the usual suspects like throwing away any user related MMC cache information and cleaning out the Exchange specific information in the user profile up to deleting the profile etc. But nothing worked.

Running the script below, which is given to you by Microsoft to check your DAG before upgrading to SP1, confirms all is well.

(Get-DatabaseAvailabilityGroup -Identity (Get-MailboxServer -Identity $env:computername).DatabaseAvailabilityGroup).Servers | Test-MapiConnectivity | Sort Database | Format-Table -AutoSize

Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | Format-Table -AutoSize

function CopyCount

{

$DatabaseList = Get-MailboxDatabase | Sort Name

$DatabaseList | % {

$Results = $_ | Get-MailboxDatabaseCopyStatus

$Good = $Results | where { ($_.Status -eq "Mounted") -or ($_.Status -eq "Healthy") }

$_ | add-member NoteProperty "CopiesTotal" $Results.Count

$_ | add-member NoteProperty "CopiesFailed" ($Results.Count-$Good.Count)

}

$DatabaseList | sort copiesfailed -Descending | ft name,copiesTotal,copiesFailed -AutoSize

}

CopyCount

Searching the internet we find some folks who have the same problem. Also with a 3 node DAG that is geographically distributed. Is this a coincidence or is this related? http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/37d96c3d-433e-4447-b696-c0c00e257765/#5071f470-13cb-4256-8aa7-ade05bb4d67d. At first I taught it might have been related to the issue described in the following blog post http://blogs.technet.com/b/timmcmic/archive/2010/08/29/exchange-2010-sp1-error-when-adding-or-removing-a-mailbox-database-copy.aspx but in the lab we could not reproduce this. The only thing we managed to confirm is that you can delete the Dumpsterinfo registry key without any problem or nasty side effects. I’m still looking into this, but I’ll need to get Microsoft involved on this one.

Updates:

  • As an other test we created a new mailbox database and by the time we got the copies set up to the 3 nodes that brand new database and its copies showed the same behavior. For that new database the registry key Dumpsterinfo doesn’t even exist (yet?). So  That’s another nail in the coffin of the idea that behavior being related to the Dumpsterinfo key I guess.
  • Next test was that I added two static IP addresses to the DAG. One for each subnet in use. Until now we had a DCHP address and I noticed it was an address for the subnet of the node that is showing the database copies. I might as well give it a try right? But nope, that didn’t make a difference either. Still waiting for that call back from Microsoft Support.
  • Meanwhile I’m thinking, hey this DAG is only showing the database copies with the lowest preference (3). So I change the preference on a test database to 1 and refresh the EMC. No joy. This must really be just a GUI hiccup or bug. Now what would prevent the EMC GUI from displaying that information?
  • Some one on the newsgroup has the same issue with a 2 node DAG in the same subnet. So not related to a 3 Node geographically dispersed DAG.
  • MS Support got in touch. They have heard it before. But unless it was related to net logon errors they don’t have a cause or solutions. There are other cases and they will escalate my support call.

On September 27th 2010:

  • After a call from an MS support engineer to confirm the issue and pass on more feedback last week, we got an update via e-mail. After completing a code review and analysis they believe to have identified the problem.  They have also been able to reproduce the issue. More information is being gathered with reference customers to confirm the findings. More updates will follow hen they have more information on how to proceed. Indeed all is well with Exchange 2010 SP1 and PowerShell is your friend 🙂 Well progress is being made. That’s good.

On October 4th 2010:

We requested feedback today and tonight we got an e-mail with a link to a blog post confirming the issue and the cause. When the Exchange Management Console draws the database copies pane, it compares the host server name of a database copy to the server name of a database copy status.  This comparison is case sensitive and if they do not match up like in DAG-SERVER-1 <> Dag-Server-1 the database copies are not shown in the GUI. Again in EMS all works just fine. A fix is still in the make. You can find the Microsoft bug here: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

On October 10th:

I received another mail from Microsoft support just now. They expect this issue to be fully resolved in Exchange 2010 Service Pack 1 Rollup Update 3.  At this time they also intend to release an incremental update that corrects the issue. But this has some caveats.

1)  The incremental update would have to be applied to all servers where administrators would be utilizing the Exchange Management Console.  I think this is expected, like with most updates.

2) The incremental update cannot be applied with other incremental updates – for example if later an issue is encountered that is fixed in a different incremental update one would have to be removed prior to installing the second.  This can be a problem for people in that situation, so pick what is most important to you

3) The incremental update would only be valid for a particular Rollup Update.  For example, if the incremental update is installed for Exchange 2010 SP1 RU1, and you desire to go to Exchange 2010 SP1 RU2, you would have to contact Microsoft to have the incremental update built and released for Exchange 2010 SP1 RU2.  This may inadvertently delay the application of a rollup update.  Nothing new here, we’ve seen this before with interim fixes.

The workaround for customers not desiring to install an incremental update would be to continue using the Exchange Management Shell with the Get-MailboxDatabaseCopyStatus command. Nothing new here Smile

They have also updated their blog: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

I’m planning on keeping the case open in order to get my hands on the fix to test in the lab and have it for customers who so desire.

October 13th:

The fix WILL be included in Exchange 2010 SP1 Roll Up 3. They ARE working on the interim updates but this will take several weeks or longer.