Category Archives: Consulting

No ADSI Edit required to fix “Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).”

Posted on by
6

During the removal of the last Exchange 2007 SP3 Mailbox server after completing the transition of Exchange 2007 to Exchange 2010 SP1 we ran into the following well known error: Object is read only because it was created by a future version of Exchange: 0.10 (14.0.100.0). Current supported version is 0.1 (8.0.535.0).

The issue is that due to the coexistence of Exchange 2007 & Exchange 2010 we can no longer remove the public folder database with the Exchange 2007 GUI (EMC). But the public folder is not visible in the Exchange 2010 GUI (EMC) as it lives on an Exchange 2007 server. Trying to remove the public folder database manually using the Exchange 2007 GUI confirms this, you’ll get the same error.

This error has been described in some blogs as early as October 2009 on http://www.proexchange.be/blogs/exchange2010/archive/2009/10/28/remove-exchange-2007-mailbox-role-fails-with-error-object-is-read-only-because-it-was-created-by-a-future-version-of-exchange-0-10-14-0-100-0-current-supported-version-is-0-1-8-0-535-0.aspx and later on as recently as October 2010 on http://www.howexchangeworks.com/2010/10/object-is-read-only-because-it-was.html

The described solution/work around in these blogs get the job done perfectly, using ADSI Edit to delete the offending Exchange 2007 public folder database. It wouldn’t be the first time ADSI Edit saves an Exchange Consultants proverbial bacon. But if it can be done without using it I often recommend not to do it.  I’ve seen to many over eager deletions in ADSI Edit get people into trouble (like deleting a public folder database before it could be dumped safely without data loss).

For this problem, it’s not required to use ADSI Edit to get rid of the public folder on the Exchange 2007 Mailbox server. You can just fire up the Exchange Command Shell (EMS) in Exchange 2010 and execute following PowerShell command:


Remove-PublicFolderDatabase "E2K7MBXSGPublicFoldersStoreSGPublicFolders"

Confirm   4: Are you sure you want to perform this action?5: Removing public folder database "E2K7MBXSGPublicFoldersStoreSGPublicFolders".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The specified database has been removed. You must remove the database file located in K:E2K7DataSGPublicFoldersPublicFolderDatabase.edb from your computer manually if it exists. Specified database: PublicFolderDatabase

This works just fine. I have no objections using ADSIEdit when needed but I don’t advise using it to others unless really necessary.In this case it just isn’t needed to fix the problem. For good measure I also deleted the storage group in which the public folder lived. After that the install went well end without any issues.

EMC Does Not Show All Database Copies After Upgrade To Exchange 2010 SP1– Still Investigating

Posted on by
2

LATEST UPDATE March 9th 2011: I have installed Exchange 2010 SP1 Rollup 3 at customer and this did indeed fix this issue finally.

Updates to this post are being added as we get them below. Last update was October 13th 2010. The have identified the cause of the issue. It’s a case sensitivity bug. The fix is WILL be contained in Exchange 2010 Sp1 Roll Up 3? But they ARE working on a incremental update in between. See below for more details and the link to the Microsoft blog entry.

At a customer we have a 3 node geographically dispersed DAG. This DAG has two nodes in the main data center and one in the recovery site in another city, but it is in the same AD Site. This works but is not ideal as DAC in Exchange 2010 RTM presumes that the node will be in another Active Directory site. As you can imagine at that location we’re very interested in Exchange 2010 SP1 since that adds support for the DAC to be used with a geographically dispersed DAG node in the same Active Directory site.

We did an upgrade to SP1 following the guidelines as published in http://technet.microsoft.com/en-us/library/bb629560.aspx and we made sure all prerequisites where satisfied. We upgraded the backup software to a version that supported Exchange 2010 SP1 and made sure no services that hold a lock on Exchange resources are running. The entire process went extremely well actually. We did have to reconfigure redirection for OWA as the SP1 installation resets the settings on the Default Web Site on the CAS Servers. But apart from that we had no major issues apart from one very annoying GUI problem. Everything was fully functional, which we verified using EMS and by testing failovers. But in the EMC GUI we had the problem under Organization / Mailbox / Database Management we only see the database copies listed on one server and not on all tree.

When you check the properties of the databases shows all three servers that are hosting copies. We used EMS commands to test for problems but it all checks out and works. Failing over a server works, both in the GUI and in PowerShell, just like activating a database.

The same issue can be seen in Server Configuration /Database Copies as demonstrated in the screenshots below. In the first figure you we selected the mailbox server where the database copies are visible.

But on the other two nodes nothing shows up, just “There are no items to show in this view”.

No errors in the vent logs or installation logs. All is working fine. So what gives? We tried all the usual suspects like throwing away any user related MMC cache information and cleaning out the Exchange specific information in the user profile up to deleting the profile etc. But nothing worked.

Running the script below, which is given to you by Microsoft to check your DAG before upgrading to SP1, confirms all is well.

(Get-DatabaseAvailabilityGroup -Identity (Get-MailboxServer -Identity $env:computername).DatabaseAvailabilityGroup).Servers | Test-MapiConnectivity | Sort Database | Format-Table -AutoSize

Get-MailboxDatabase | Sort Name | Get-MailboxDatabaseCopyStatus | Format-Table -AutoSize

function CopyCount

{

$DatabaseList = Get-MailboxDatabase | Sort Name

$DatabaseList | % {

$Results = $_ | Get-MailboxDatabaseCopyStatus

$Good = $Results | where { ($_.Status -eq "Mounted") -or ($_.Status -eq "Healthy") }

$_ | add-member NoteProperty "CopiesTotal" $Results.Count

$_ | add-member NoteProperty "CopiesFailed" ($Results.Count-$Good.Count)

}

$DatabaseList | sort copiesfailed -Descending | ft name,copiesTotal,copiesFailed -AutoSize

}

CopyCount

Searching the internet we find some folks who have the same problem. Also with a 3 node DAG that is geographically distributed. Is this a coincidence or is this related? http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/37d96c3d-433e-4447-b696-c0c00e257765/#5071f470-13cb-4256-8aa7-ade05bb4d67d. At first I taught it might have been related to the issue described in the following blog post http://blogs.technet.com/b/timmcmic/archive/2010/08/29/exchange-2010-sp1-error-when-adding-or-removing-a-mailbox-database-copy.aspx but in the lab we could not reproduce this. The only thing we managed to confirm is that you can delete the Dumpsterinfo registry key without any problem or nasty side effects. I’m still looking into this, but I’ll need to get Microsoft involved on this one.

Updates:

  • As an other test we created a new mailbox database and by the time we got the copies set up to the 3 nodes that brand new database and its copies showed the same behavior. For that new database the registry key Dumpsterinfo doesn’t even exist (yet?). So  That’s another nail in the coffin of the idea that behavior being related to the Dumpsterinfo key I guess.
  • Next test was that I added two static IP addresses to the DAG. One for each subnet in use. Until now we had a DCHP address and I noticed it was an address for the subnet of the node that is showing the database copies. I might as well give it a try right? But nope, that didn’t make a difference either. Still waiting for that call back from Microsoft Support.
  • Meanwhile I’m thinking, hey this DAG is only showing the database copies with the lowest preference (3). So I change the preference on a test database to 1 and refresh the EMC. No joy. This must really be just a GUI hiccup or bug. Now what would prevent the EMC GUI from displaying that information?
  • Some one on the newsgroup has the same issue with a 2 node DAG in the same subnet. So not related to a 3 Node geographically dispersed DAG.
  • MS Support got in touch. They have heard it before. But unless it was related to net logon errors they don’t have a cause or solutions. There are other cases and they will escalate my support call.

On September 27th 2010:

  • After a call from an MS support engineer to confirm the issue and pass on more feedback last week, we got an update via e-mail. After completing a code review and analysis they believe to have identified the problem.  They have also been able to reproduce the issue. More information is being gathered with reference customers to confirm the findings. More updates will follow hen they have more information on how to proceed. Indeed all is well with Exchange 2010 SP1 and PowerShell is your friend 🙂 Well progress is being made. That’s good.

On October 4th 2010:

We requested feedback today and tonight we got an e-mail with a link to a blog post confirming the issue and the cause. When the Exchange Management Console draws the database copies pane, it compares the host server name of a database copy to the server name of a database copy status.  This comparison is case sensitive and if they do not match up like in DAG-SERVER-1 <> Dag-Server-1 the database copies are not shown in the GUI. Again in EMS all works just fine. A fix is still in the make. You can find the Microsoft bug here: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

On October 10th:

I received another mail from Microsoft support just now. They expect this issue to be fully resolved in Exchange 2010 Service Pack 1 Rollup Update 3.  At this time they also intend to release an incremental update that corrects the issue. But this has some caveats.

1)  The incremental update would have to be applied to all servers where administrators would be utilizing the Exchange Management Console.  I think this is expected, like with most updates.

2) The incremental update cannot be applied with other incremental updates – for example if later an issue is encountered that is fixed in a different incremental update one would have to be removed prior to installing the second.  This can be a problem for people in that situation, so pick what is most important to you

3) The incremental update would only be valid for a particular Rollup Update.  For example, if the incremental update is installed for Exchange 2010 SP1 RU1, and you desire to go to Exchange 2010 SP1 RU2, you would have to contact Microsoft to have the incremental update built and released for Exchange 2010 SP1 RU2.  This may inadvertently delay the application of a rollup update.  Nothing new here, we’ve seen this before with interim fixes.

The workaround for customers not desiring to install an incremental update would be to continue using the Exchange Management Shell with the Get-MailboxDatabaseCopyStatus command. Nothing new here Smile

They have also updated their blog: http://blogs.technet.com/b/timmcmic/archive/2010/10/04/database-copies-fail-to-display-after-upgrading-to-exchange-2010-service-pack-1.aspx

I’m planning on keeping the case open in order to get my hands on the fix to test in the lab and have it for customers who so desire.

October 13th:

The fix WILL be included in Exchange 2010 SP1 Roll Up 3. They ARE working on the interim updates but this will take several weeks or longer.

 

Microsoft Project 2007-2010 Default Format When Saving Group Policy

Posted on by
Reply

For a roll out of Office 2010 we needed to set the default format when saving to Microsoft Project 2007 and not Microsoft Project 2010. This was the prevent any accidental file format issues with people still using Microsoft Project 2007. The rest of the Office 2007 formats such as Word, Excel, Visio, … have remained the same as in 2007 (in the case of Visio even the same as 2003).

We had already loaded the ADMX GPO templates for Office 2010 some and intend to set the default to Microsoft Project 2007. So we navigate to User Configuration => Policies => Administrative Templates: Policy definitions (ADMX files) retrieved from central store => Microsoft Project 2010

Where we find following options:

Funny we thought we’d find three options: Project 2010, Project 2007 and Project 2000-2003. Now what mpp format would that “Project” setting refer to? Microsoft Project 2007 or 2010? Well we checked the contents of the ADMX file proj14.admx which can be found, when using the central store that is, at:

\blog.demoSYSVOLblog.demoPoliciesPolicyDefinitionsproj14.admx

There we see the following:

Aha it’s the mpp format of Microsoft Project 2007 as indicated by MSProject.MPP.12. An entry for Microsoft Project 2010 doesn’t even exist. We quickly had a look at the proj12.admx (Microsoft Project 2007) and there the entries are exactly the same for that setting for “Save” policy.

It seems that even when the file formats is different between Microsoft Project 2007 and Microsoft Project 2010 they offer no way to easily identify what mpp format you’re setting it to in the GPO. Why they don’t offer Microsoft Project 2010 I don’t know. I guess their take is that you could leave it disabled in the GPO. Which is true but GPO setting are normally very unambiguous and you can explicitly set every option when you enable it, even the default when disabled. Perhaps they just forgot as they did not need this for the other office products in the Office 2010 release. I don’t know.

Anyway we tested this and indeed it set the default format for saving to Microsoft Project 2007. So all is well. Tip of the day: check the content of the ADMX file when in doubt.

Exchange 2007 & 2010 Event ID’s: 2601, 2604, 2501 & Users Can’t Access Mailboxes / Public Folders On My Day Off

Posted on by
12

I took the day off as I needed some time to deal with government administration. Good thing this is a blog about IT issues because holey crap what a time eating, confusing and rather pointless mess government administration can be. The process to get to the desired outcome is very tedious, prone to misunderstanding & pretty inefficient . What the entire duration of the process and the number of administrative entities involved contribute to the desired result is a mystery. It’s pure show and window dressing. But OK, we took the day of to finally get it all sorted after 5 months of patiently waiting for this day.

So I sleep until 08:00, get up and head for the kitchen for a jar of coffee. With the only Java I truly like in my hand I make my way to the home office. I check mails/alerts from System Center, Support Requests etc. I’m like a responsible guy dude, even when I need a day off. I do monitor the condition of my projects in production and I do step in when needed and document my findings. It keeps me honest when I design and sell my solutions. Beware of some architects that are not the ones having to deal with the crap architectures they design, they are often empty suits. Anyway, I see an issue that could be a warning of more to come. Someone has a problem with Outlook 2007 which reports the following error (translation from Dutch):

“Unable to expand the folder. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server computer is down for maintenance.(/o=<DOMAIN>/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=servers/cn=<dagmember1>)”

Now I know that user. Smart, diligent and reliable. That user even provides the relevant and necessary information in their support request. Yes they do exist and HRM should hire those exclusively. So in combination with that error we knew we did not have an PEBKAC or ID-10T on our hands but a real issue.

I quickly check that DAG member node Outlook of that user is trying to connect to but I know that due to maintenance their mailboxes currently reside on another member of the DAG. So i could very well be just the public folders. Bingo. A quick test reveals this to be the case. Also the Windows 2008 R2 server and Exchange 2010 itself are running perfectly fine, happy as can be, except on that one node we see the Application Event Log messages:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2601
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When initializing a remote procedure call (RPC) to the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the SID for account <WKGUID=XXXXXXXXXXNOREALIDXXXXXXXXXXXXXX,CN=Microsoft Exchange,CN=Services,CN=Configuration,…> – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2604
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.demo
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). When updating security for a remote procedure call (RPC) access for the Microsoft Exchange Active Directory Topology service, Exchange could not retrieve the security descriptor for Exchange server object DAGMEMBER1 – Error code=8007077f. The Microsoft Exchange Active Directory Topology service will continue starting with limited permissions.

Log Name:      Application
Source:        MSExchange ADAccess
Date:          8/19/2010 7:12:43 AM
Event ID:      2501
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      dagmember1.company.blog
Description:
Process MSEXCHANGEADTOPOLOGY (PID=1620). The site monitor API was unable to verify the site name for this Exchange computer – Call=DsctxGetContext Error code=8007077f. Make sure that Exchange server is correctly registered on the DNS server.

I think I’m OK when I see the possible cause. Why? Because I also know even if that probable cause isn’t the problem, it’s a hiccup I’ve seen before and I know how to fix its one. When you search those errors you can find a TechNet article describing a possible cause: “An inactive network connection is first on the binding list” http://technet.microsoft.com/en-us/library/dd789571(EXCHG.80).aspx. The fix is quite simple. Correct the NIC order and restart the MSExchange ADTopology Service. I had my scare about Active Directory and DNS horrors the first time I ever saw this one. So no gut wrenching panic here 🙂

But why do servers ever get in to this state when the NIC ordering is just fine? We did some firmware and upgrade recently after hours but that didn’t affect the NIC binding order. Now I’m pretty weird at times but I still know what I’m doing. Those NIC where OK when I configured those servers. Checking that has become a second nature on multi homed and clustered servers. I also remember happening this to me once before somewhere in February 2010 with another setup of Exchange 2010 on Windows 2008 R2. And in that case the NIC order in the binding list was also OK. I checked back then as well just to make sure. But since I build those Exchange 2010 setups myself I just know they are close to godliness both in design and implementation :-). Back then the issue went away by restarting the server, restarting the MSExchange ADTopology Service will do however, and the problem never came back. For some reason the AD Site information query fails. Now Windows retries and is OK after a while. Exchange, tries to get the AD Site information once, fails and keeps thinking there is an issue. With as a result clients have no connectivity and those errors that initially make you think you could have DNS issues, AD problems etc. But fortunately it’s a lot less serious.

So when the NIC binding order is OK why does this happen? I can’t tell you for sure but I do know that I’m not the only one (not that weird after all) since Microsoft published KB Article “MSExchange ADAccess Event ID’s 2601, 2604, 2501” http://support.microsoft.com/kb/2025528 . This article is a so called FAST PUBLISH from Microsoft Support and states that the issue only occurs on Windows 2008 R2 and that it affect Exchange 2007 and Exchange 2010. The cause? Well this is where they provide only what I already knew:

“During a restart of the server, the operating system queries Active Directory to get its AD Site information.  On a Windows 2008 R2 server, this will sometimes fail.  As the Exchange services are starting, it also will do a query for its AD Site and that too will fail. Windows will continue to try and determine its AD Site name and will eventually succeed.  However, Exchange does not re-try the query and the above errors are logged in the application log every 15 minutes.”

And yes the workaround/fix is also nothing new:

“After the server has been up for a minute or two, run NLTest /DSGetSite to verify that that the proper Active Directory Site is being returned by Windows.  Once that has been verified, restart the MSExchange ADTopology Service.”

Do note that this will also restart a slew of dependant Exchange services so it takes a little while.

  • Microsoft Exchange Transport Log Search
  • Microsoft Exchange Transport Log
  • Microsoft Exchange Service Host
  • Microsoft Exchange Search Indexer
  • Microsoft Exchange Replication Service
  • Microsoft Exchange Mail Submission
  • Microsoft Exchange Mailbox Assistants
  • Microsoft Exchange File Distribution
  • Microsoft Exchange EdgeSync
  • Microsoft Exchange Anti-spam Update

So after some manual intervention we had the users back in business. And all is well for them, as they rise and sleep under the watchful eye of a bunch of good IT Pro’s who’ll protect them form further harm and problems 😉 Now I need to get an auto fix for this I think until Microsoft fixes this one for good. SCOM where are you? No, no, no … It’s my day off for getting that administration done!