Exchange 2010 DAG Issue: Cluster IP address resource ‘Cluster IP Address’ cannot be brought online

Today I was called upon to investigate an issue with an Exchange 2010 Database Availability Group that had serious backup issues with Symantec Backup Exec not working. As it turned out, while the DAG was still providing mail services and clients did not notice anything the underlying Windows Cluster Service had an issue with. The cluster resource could not be brought on line, instead we got an error:

“Cluster IP address resource ‘Cluster IP Address’ cannot be brought online because the cluster network ‘Cluster Network 1’ is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.”

I have been dealing with Windows 2008 (R2) clusters since the beta’s and had seen some causes of this so I started to check the cluster & Exchange DAG configuration. Nothing was wrong, not a single thing. Weird. I had seen such weird behavior once before with a Hyper-V R2 cluster. There I fixed it by disabling and enabling the NIC’s on the nodes that were having the issue, thus resetting the network. I you don’t have DRAC/ILO or KVM over IP access you can temporarily allow client access via another cluster network or you’ll need physical access to the server console.

In the event viewer I found some more errors:

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          6/18/2010 2:02:41 PM
Event ID:      1069
Task Category: Resource Control Manager
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      node1.company.com
Description: Cluster resource ‘IPv4 DHCP Address 1 (Cluster Group)’ in clustered service or application ‘Cluster Group’ failed.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          6/18/2010 1:54:47 PM
Event ID:      1223
Task Category: IP Address Resource
Level:         Error
Keywords:     
User:          SYSTEM
Computer:     node1.company.com
Description: Cluster IP address resource ‘Cluster IP Address’ cannot be brought online because the cluster network ‘Cluster Network 1’ is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          6/18/2010 1:54:47 PM
Event ID:      1223
Task Caegory: IP Address Resource
Level:         Error
Keywords:     
User:          SYSTEM
Counter:      node1.company.com
Description: Cluster IP address resource ‘IPv4 DHCP Address 1 (Cluster Group)’ cannot be brought online because the cluster network ‘Cluster Network 3’ is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.

So these cluster networks (it’s a geographically dispersed cluster with routed subnets) are indicating they do not have “Allow clients to connect through this network” set.  Well, I checked and they did! Both “Allow cluster network communications on this network” and “allow clients to connect through this network” are enabled. 

Weird, OK but as mentioned I’ve encountered something similar before. In this case I did not want to do just disable/enable those NICs. The DAG was functioning fine and providing services tot clients, so I did not want to cause any interruption or failover now the cluster was having an issue.

So before going any further I did a search and almost within a minute I found following TechNet blog post: Cluster Core Resources fail to come online on some Exchange 2010 Database Availability Group (DAG) nodes (http://blogs.technet.com/b/timmcmic/archive/2010/05/12/cluster-core-resources-fail-to-come-online-on-some-exchange-2010-database-availability-group-dag-nodes.aspx)

Well, well, the issue is known to Microsoft and they offer three fixes. Which is actually only one, but can be done using  the Failover Cluster Manager GUI, cluster.exe or PowerShell. The fix is to simply disable and enable  “Allow clients to connect through this network” on the affected cluster network. The “long term fix” will be included in Exchange 2010 SP1. The work around does work immediately and their Backup Exec started functioning again. They’ll just have to keep an eye on this issue until the permanent fix arrives with SP1.

Exchange 2007-2010 Public Folders Issues “The Active Directory user wasn’t found.”

I was working on an Exchange 2007 to Exchange 2010 project when we ran into trouble creating our first public folder database on an Exchange 2010 server. Mind you, this was just creating the database. We did not even set up replication for this database yet. All mailboxes still resided in Exchange 2007 databases pointing to an Exchange 2007 public folder. Very soon after creating the database we got notified users could no longer send mails to mail enabled public folders. The exact error was this:

554 5.6.0 STOREDRV.Deliver.Exception:ObjectNotFoundException; Failed to process message due to a permanent exception with message The Active Directory user wasn’t found.

Also browsing of the public folders in Outlook was slow and the application froze/hung. These issues where fixed very fast by getting rid of the still unused public folder database all together. Now we could commence our search for the root cause. The error seemed related to the issue described in Public Folder Replication Fails Due To Empty Legacy Administrative Group which can be found @ http://msexchangeteam.com/archive/2010/05/05/454821.aspx  The blog describes this error during replication:

Log Name: Application

Source: MSExchange Store Driver

Event ID: 1020

Level: Error

Description:

The store driver couldn’t deliver the public folder replication message “Hierarchy ([email protected])” because the following error occurred: The Active Directory user wasn’t found.

But apart from replication not working there were other, more severe issues impacting end users who can still all be on Exchange 2007. The hanging of the outlook clients and mail enabled folders no longer being available. Dave Stork blogged about this in http://blogs.dirteam.com/blogs/davestork/archive/2010/03/16/mail-enabled-public-folder-recipient-not-found.aspx

Now the first mentions of the replication issue have been reported back in November 2009 (see http://get-exchange.blogspot.com/2009/11/public-folder-mayhem-exchange-2010.html) but still hasn’t been fixed. For the moment that fix is planned to be included in E2K10 RU5. Currently we’re at RU3, so that might well be august 2010.

The workaround described in above mentioned blog posts works & is effective immediately. Now they described the issue and the fix very well but I’ll add to tips.

Tip 1

“Practical End User Friendly Detection” of this issue can be done using exfolders.exe. You can read more about this tool here: “Exchange, meet ExFolders” (http://msexchangeteam.com/archive/2009/12/04/453399.aspx).The error only occurs when you create a public folder on Exchange 2010 and can be very annoying for the users so I’ll share this tip with you. Download the tool here http://msexchangeteam.com/files/12/attachments/entry453398.aspx and install it on an Exchange 2010 server in the bin directory (follow the readme.txt and don’t forget to merge the .reg file or the tool will crash). Running exfolders.exe and connect against any Exchange 2007 public folder. When you get this error …

—————————

ExFolders

—————————

An error occurred while trying to establish a connection to the Exchange server. Exception: The Active Directory user wasn’t found.

—————————

OK  

—————————

… you know you are affected. Deleting the empty Servers containers from ALL legacy Administrative Groups fixes the error. You then can connect successfully to a Exchange 2007 public folder with exfolder.exe. Which is a cool way to test for this issue and if the fix works as you don’t need to create a public folder and possibly hinder you users.

Tip 2

Also note that you need to delete  (using ADSIEDIT) every empty servers container out of every legacy Administrative Group, not just or only the one in the “First Administrative Group”. Don’t worry if you renamed that one to something more descriptive, that doesn’t matter at all. All the servers containers in the legacy Administrative Group should be empty I you have no more E2K3 servers left in your exchange organization. Feel free to leave comments on your experiences.

Hiring practices – The Dilbert Life

What’s going on?

For the last 2 years I’ve been in a number of situations where management was pushing to hire extra FTE’s. The numbers on the personnel lists had to be met whether it made sense or not I guess. Sometimes they wanted to give someone a raise or do them a favor with a better contract.

Furthermore in their effort to reach the numbers there was pressure to drop our standards, to lower the bar so to speak. Well, not on my watch. My standards are in place for a reason. To make sure that the people on the job can actually handle that job under all circumstances. Look, when the shit hits the fan and things get tough you’ll need every bit of team spirit and camaraderie that you can get to make sure the team can withstand the stress and get the job done. If you put people in the teams that don’t cut it you’ll already have dysfunctional unit in “normal” times, what do you think will happen when things get tough? They’ll just cut the rope and let the dead weight fall.

People, those standards, the difficulty & complexity in selection, the so called “bar” are there for a reason. If you cannot meet those demands it is not those standards that are the problem. The only real issue is that you’re not capable to meet those standards. So stop complaining about them. Instead start working on yourself so you can meet them. If you can’t do that you’ll need to find job satisfaction somewhere else.

But why this pressure to hire people even if they don’t seize up? Why this obsession with more and more personnel?

Filling the numbers

On paper all is well, you’ve got plenty of warm bodies to fill the cubicles and to meet productivity demands. You can’t produce a baby in one month with 9 women, it doesn’t work that way. Perhaps methodologies are putting so much pressure on having all forms and numbers right that achieving just that is priority one. You might have the best functional team in the business but if those numbers don’t match up to the methodology documents that doesn’t matter. You get no bonus but lots of hassle. So what’s a poor weak manager who wants to climb the ladder to do? Sell out the team to comply. Go ahead. Kiss your team’s respect goodbye. You might as well have flushed their motivation down the toilet.

Other times people simply can’t manage. So when they have underperforming team members the easy way out is getting extra FTE’s. That’s way easier than dealing with the problem. So bloat your FTE needs to the needed numbers + the number of miss hires already in place. And then there is the danger of hiring good people, they are a treat to the peaceful little world management is in. Remember, “A people” hire “A people”, “B people” hire “B people” … and it goes downhill from there.

Sometimes you might need to have x amount of people to legitimize your management position so … bring on the FTE’s 🙂

There are lots of reasons why this happens. Managers, especially middle management often suck at management. It is really a very hard, demanding and crucial job. Management at least gets the perks. Middle management is caught between a rock and a hard place. But the dangling carrot and their ambition keeps them from walking out and the keep choosing the easy way to deal with. Perhaps they don’t even have a choice. That’s the best they can do.

Giving a raise / doing a favor

This is bad, very bad. You are inflicting real damage here. This is the equivalent of making me an NCO in the airborne rangers because I love to shoot guns and like to brag in the bar with a ranger patch on my sleeve. It’s pure madness. I’m overweight, I’m as myopic as a mole and I have several medical issues that would put me (and as a consequence my mates) in danger, even on an exercise. What do you think will happen? Everybody will lose and will be unhappy in this situation. Me, because I know I’m fooling myself, I know I’m a burden to my colleagues and endangers myself and them. I will never be accepted let alone be respected. Nope they’ll come to despise me. They have their job to do, plus now they need to do mine, correct my failures, drag me along and rescue me. So basically they’ll resent the hell out of me. They’ll be pissed off at the officers who passed me because they have devalued their skills and show no professional recognition of what they are capable of. The officers won’t be happy because they’ll have to command and work with a dysfunctional unit. It is madness. Is cannot work, it will not work.

If this metaphor doesn’t get the message across than perhaps this one will: imagine your brain surgeon was given the position because nagged long enough to hospital management for a raise

Conclusion

Managers, don’t lower the bar to fill the numbers, to give people a raise or do them a favor. You’ll just create frustration and dysfunctional teams. They won’t function as well as they could and when they need to stand tall to face serious issues you don’t have the number you fool yourself to have on paper. At best you have your good people who’ll  need to work around the bad ones. In the worst case the team is so dysfunctional they won’t get the job done as well and fast as they should or they’ll fail all together. But in reality management is the real failure. They didn’t do their job. Unit operational efficiency might not be on your radar screen but it really should be.

The do’s and don’ts when engaging consultants Part II

In part II we take a look at the common mistakes when using consultants. This is all common sense and well documented in literature and on line. So why does it go wrong so often? Politics and just not caring (I just work here dude) often in combination with “fake it until you make it”. Follow this link for Part I

Common Mistakes

Buying advice to settle a score

Now please all acts as grownups! Don’t settle your disagreements this way. One side will feel defeated, the other victorious with a vengeance. Great start for a project that depends on collaboration<SARCASM>. What if the result of such advice is very mixed with lots of pros and cons on both sides? Then what? You’re doomed. No agreement, no common goals, no purpose or will to cooperate. You are now riding a dead horse. Guess what? You are not going to get anywhere 🙂 What your team really needs people, is a coach!

Bait & Switch

So the guru walks in, explains with great passion and knowledge the solution and how to achieve it. You make the deal. You are very happy to have such a highly skilled operator to take care of your interests. But soon after the project starts the guru is nowhere to be found. Some pimple faced youths and a washed up senior are handling your case. What the hell happened? Will things work out? Sure the guru will present the finished project which has taken too long, isn’t as trouble free as it should be and was rather hard going the entire time. Well, nothing that can’t be fixed on a follow up consulting engagement, right? This is the oldest trick in the book. They trained their young potentials at your expense and now know which ones can’t hack it. You’ve become the free university for the consulting firm. Avoid this at all cost. Agree on who will do the work. Fix those numbers and names so you get the people you signed up for.

Giving away your business

Well those IT people are a tough crowd. They are opinionated and don’t communicate to well. Hell those guys & gals prefer to work with machines! They are not up to speed with what is politically correct or fashionable, hate faking, can’t stand save asses and don’t tolerate kiss asses. Good IT people live by the sword and die by the sword. It’s all very direct. Either their solution works or it doesn’t. You can’t hide behind reports, gold plated words or lies when you work in IT. The evidence is there every single second of every single day, staring you right in the face. So basically it’s no wonder that weak management and incompetent employees can’t get along with them very well, they are a bit to direct. So why not get rid of that problem? Get some consultants to do all that IT stuff. You’ll be pleasantly surprised by how courteous they are, well willing and facilitating. Sure it cost a bundle but hey, life seems good. Then your best IT people leave (physically or mentally) and your worst just hang around for the paycheck. No worries, no one is indispensible and firm X has more consultants for you! And so, slowly but surely, you become completely dependent on very expensive external staff and what you thought was consulting for your company has turned into a money generating market for a body shop that generates revenue by sending you their juniors to be trained at your expense. Congratulations, you fucked up big time. Your organization is now nothing more than a cash cow. When you no longer produce the gold, you get slaughtered. Game over.

Instead of outsourcing you should have invested in the IT team. Train them, communicate with them, be honest about projects and listen to them when they tell you it won’t work. They don’t do that to piss you off but because they think you’re making a mistake. Get rid of the idiots and assholes (they shouldn’t be there in the first place) and take care of your key IT personnel. They are hard to come by. Get external help from consultants where and when needed, but don’t hand over your business to them. If you do, you lose everything.

Buying the wrong stuff from the wrong people

This one seems so easy to avoid … Don’t buy the advice on what to buy from the company that sells the products. Don’t buy CRM services from a hardware company. Don’t let the marketing guys write software. What kind of switch will Cisco advise you to buy? Think about your interests and those of the vendors. Make sure they align. Buy where you get the best deal for your needs. Get the right expertise from the right experts.

Not knowing what you buy

You’ve seen the possible consulting engagements already.

· Advice

· Implementation

· Coaching

· Skills Acquisition

Now decide what you need and communicate this very clearly. Make sure everyone understands what is expected. Put it in writing. That way there will be no surprises such as getting an advice about e-mail infrastructures instead of a functioning Exchange 2007 system like you taught you’d get .

Buying efforts instead of results

Efforts are commendable, nothing more. They are not the desired outcome. It really doesn’t matter how much time, work, people, and resources you throw against a problem. The only things that matters are the achieved results. Buying efforts is a great way to make a consulting firm very wealthy. It’s usually the result of ill defined scopes, not getting clearly defined deliverables in writing and bad project management. The worst type of this mistake is “consultants” who start churning. Those bastards give true consultants a bad name. In essence this is fraud. They steal time and as such money by doing unnecessary work, prolonging issues or problems to augment the billable time, etc. Never forget that our economy a society is fueled by bull shit. Half of the consultants are selling just that. Hire the other half. Don’t feel bad if you’re a consultant, what do you think permanent staff sells half of the time? Right!

Falling in love with the misuse of “methodologies”

Sometimes both clients and consultant firms fall in love with methodologies or make mistake manuals for methodologies. They focus so much on the methodology everything else becomes secondary to it. This often has a couple reasons. One of them is that a lot of businesses lie when they say their personnel are their biggest capital or most important asset. They hate the fact that they are dependent on specific people and skill sets. It’s way too risky and expensive. They want to modularize people like parts of a car. If it’s broken, replace it with an identical one. The other reason is that talent doesn’t scale very well. This is quite normal, people just don’t scale. Talent needs to be cultivated and that takes time and effort. Performing complex tasks to produce high quality results takes talent. This makes things expensive as availability to talent is limited and thus growth is stalled.

So we’re in trouble here. We can’t attract enough talent and the bean counters insist on reproducible identical drones (“Human Resources”), which is impossible to achieve. So the solution consultancy firms come up with is the misuse of methodologies as a religion. They create guide lines, methodologies, scripts etc. You see, one way people try to scale talent is by creating a cookbook the “less talented” to follow. But books don’t make an excellent cook! This is because talent cannot be methodized completely. This is a farce. In the end it leads to mediocre firms, run by mediocre people producing mediocre products. Talented and motivated people will use methodologies correctly when and where appropriate. But they have also learned to be creative and the use of experience combined with knowledge and guts allows them to produce excellent results. Joel Spolsky did a great write up about this in Big Macs vs. The Naked Chef on his web site (http://www.joelonsoftware.com/articles/fog0000000024.html).

Now methodologies are useful when used correctly! But focusing on them where talent and creativity are needed will only drive the talent away or in hiding. Is this what you hired consultants to do? Is your organization in such a bad shape? Do you really have that many people on the payroll that are not capable of performing their jobs? Next thing you’ll be buying innovation and creativity as process from those same consultants.

Conclusion

Good consultants can and will help you to realize well defined, complex and important goals. However, before engaging consultants, you need to be aware of the pitfalls. Know what consultancy can and cannot do for you given the circumstances of your organization and your needs. Using consultants for mere staff augmentation is not a good idea. Good consultants know this and act upon it. You need to be smart, proactive and involved in selecting and working with consultancy firms. You just can’t expect to hire some consultants, lay back in your easy chair and hope all will be well from that moment on. Far from it, that attitude will only lead to more and very expensive problems.