A Hardware Load Balancing Exercise With A Kemp Loadmaster 2200

I recently had the opportunity to get my hands on a hardware load balancer for a project where, due to limitations in the configuration of the software, Windows Network Load Balancing could not be used. The piece of kit we got was a LoadMaster 2200 by Kemp Technologies. A GPS network/software services solution (NTRIP Caster) for surveyors needed load balancing, not only for distributing the load, but also to help with high availability. The software could not be configured to use a Virtual IP address of a Windows Load Balancer cluster. That meant when had to take the load balancing of the Windows server nodes. I had been interested in Kemp gear for a while now (in function of some Exchange implementations) but until recently I did not get my hands on a LoadMaster.

We have two networks involved. One the 192.1683.2.0/24 network serves as a management, back-office network to which the dial access calls are routed and load-balanced to 2 separate servers WebSurvey01 and WebSurvey02 (running VMs running on Hyper-V). The Other network is 192.168.1.0/24 and that serves the internet traffic for the web site and the NTRIP data for the surveyors, which is also load balanced to WebSurvey01 and WebSurvey02. The application needs to see the IP addresses of the clients so we want transparency. To achieve this we need to use the gateway of the VIP on the Kemp load balancer as the gateway. That means we can’t connect to those apps from the same subnet, but this is not required. The clients dial in or come in from the internet. A logical illustration (it’s not a complete overview or an exact network diagram) of such a surveyor’s network configuration is shown below.

Why am I using layer 7 load balancing? Well, layer 4 is a transport layer (which is transparent but not very intelligent) and as such is not protocol aware while layer 7 is an application layer and is protocol aware. I want the latter as this gives me the possibility to check the health of the underlying service, filter on content, do funky stuff with headers (which allows us to give the clients IP to the destination server => X-Forwarded-For header when using layer 7), load balance traffic based on server load or service etc. Layer 7 not as fast as layer 4, as there is more things to do, code to run, but when you don’t overload the device that not a problem as it has plenty of processing power.

The documentation for the KEMP LoadMaster is OK. But I really do advise you to get one, install it in a lab and play with all the options to test it as much as you can. Doing so will give you a pretty good feel for the product, how it functions, and what you can achieve with it. They will provide you with a system to do just that when you want. If you like it and decide to keep it, you can pay for it and it’s yours. Otherwise, you can just return it. I had an issue in the lab due to a bad switch and my local dealer was very fast to offer help and support. I’m a happy customer so far. It’s good to see more affordable yet very capable devices on the market. Smaller projects and organizations might not have the vast amount of server nodes and traffic volume to warrant high-end load balancers but they have needs that need to be served, so there is a market for this. Just don’t get in a “mine is bigger than yours” contest about products. Get one that is the best bang for the buck considering your needs.

One thing I would like to see in the lower end models is a redundant hot-swappable power supply. It would make it more complete.  One silly issue they should also fix in the next software update is that you can’t have a terminal connection running until 60 seconds after booting or the appliance might get stuck at 100% CPU load. Your own DOS attack at your fingertips. Update: I was contacted by KEMP and informed that they checked this issue out. The warning that you should not have the vt100 connected during a reboot is an issue the used to exist in the past but is no longer true. This myth persists as it is listed on the sheet of paper that states “important” and which is the first thing you see when you open the box. They told me they will remove it from the “important”-sheet to help put the myth to rest and your mind at ease when you unbox your brand new KEMP equipment. I appreciate their follow up and very open communication. From my experience, they seem to make sure their resellers are off the same mindset as they also provided speedy and correct information. As a customer, I appreciate that level of service.

The next step would be to make this he setup redundant. At least that’s my advice to the project team. Geographically redundant load balancing seems to be based on DNS. Unfortunately, a lot of surveying gear seems to accept only IP addresses so I’ll still have to see what possibilities we have to achieve that. No rush, getting that disaster recovery and business continuity site designed and setup will take some time anyway.

They have virtual load balancers available for both VMware and Hyper-V but not for their DR or Geo versions. Those are only on VMware still. The reason we used an appliance here is the need to make the load balancer as independent as possible of any hardware (storage, networking, host servers) used by the virtualization environment.

Event ID: 11 From Microsoft-Windows-RPC-Events Are Indicating Possible Memory Leaks With MMC

After finishing putting some brand new servers in place with Windows 2008 R2, installing its rolls and leaving a happy client I’m usually very happy about a job well done. That feeling can last for a while when doing the paperwork involved with the project. It can also go away blazingly fast when you get a call that there is an “RPC memory leak or something no right” on the servers.  Not good. So you remotely access the server and start looking. Luckily for me this was to be a non issue. The event logged was the following:

Log Name:      Application

Source:        Microsoft-Windows-RPC-Events

Date:          06/01/2011 22:26:18

Event ID:      11

Task Category: None

Level:         Warning

Keywords:     

User:          BIGBillyTheServerAdmin

Computer:      infra01.big.corp

Description:

Possible Memory Leak.  Application ("C:Windowssystem32mmc.exe" "C:Windowssystem32dhcpmgmt.msc" ) (PID: 5000) has passed a non-NULL pointer to RPC for an [out] parameter marked [allocate(all_nodes)].  [allocate(all_nodes)] parameters are always reallocated; if the original pointer contained the address of valid memory, that memory will be leaked.  The call originated on the interface with UUID ({6bffd098-a112-3610-9833-46c3f874532d}), Method number (2).  User Action: Contact your application vendor for an updated version of the application.

If you do a search for this you’ll find several unresolved news group and support site questions but also a Microsoft knowledge base article http://support.microsoft.com/kb/974814. It states that when you run the Server Manager Snap-in (servermanager.msc) for extended periods of time, the application event log warning as seen above is logged. It also says it only happens on DHCP servers, which is exactly a roll these servers have and the warning entry we see in the application even log. As long as the UUID is {6bffd098-a112-3610-9833-46c3f874532d} and you have no other indications of a memory leak you’re good to go. Armed with the link we quickly put the owners mind at easy and all is well again. Back to the paperwork.

Windows 2008 R2 & Windows 7 SP1 RTM Today!!!!

UPDATE: The Russian TechNet blog retracted it’s statement about SP1 being RTM. We’ll see.

A quick heads up. According to WinRumors Microsoft has confirmed the release of Windows 7 / Windows 2008 R2 SP1. http://www.winrumors.com/microsoft-confirms-windows-7-sp1-rtm-released-to-oems-today/?utm_source=twitterfeed&utm_medium=twitter&utm_campaign=Feed%3A+WinRumors+%28WinRumors%29. My busy days just got busier. Cluster nodes with Hyper-V in the lab are being fired up already for final testing before wisely introducing it into production. My current workstation of cause is going to be updated faster than I can download the service pack Smile I’ll update this post with a download link when I get it.

DCDIAG.EXE Problem On Windows 2008(R2): VerifyEnterpriseReferences indicates problem “Missing Expected Value” & points to Knowledge Base Article: Q312862

I was preparing to replace some 5 year old DELL PE1850 servers running Active Directory with new DELL R610 servers when the DCDIAG.exe output showed a possible issue with SYSVOL FRS and some missing expected value.

Starting test: VerifyEnterpriseReferences

The following problems were found while verifying various important DN

references.  Note, that  these problems can be reported because of

latency in replication.  So follow up to resolve the following

problems, only if the same problem is reported on all DCs for a given

domain or if  the problem persists after replication has hadreasonable time to replicate changes.

[1] Problem: Missing Expected Value

Base Object: CN=DC1,OU=CITY,OU=Domain Controllers,DC=corp,DC=com

Base Object Description: "DC Account Object"

Value Object Attribute Name: msDFSR-ComputerReferenceBL

Value Object Description: "SYSVOL FRS Member Object"

Recommended Action: See Knowledge Base Article: Q312862

The log points to a knowledge base article at  but that has no relevance here.This is a phantom error when found under following circumstances. It occurs on Windows 2008 or Windows 2008 R2 when you are running in Windows 2008 or Windows 2008 R2 domain functional level. Since Windows 2008 the File Replication Service (FRS) that sysvol uses has been replaced with the  Distributed File Replication service (DFRS) as used by DFS. If you’re not yet running DFRS when you can (which is highly recommend  http://blogs.technet.com/b/askds/archive/2010/04/22/the-case-for-migrating-sysvol-to-dfsr.aspx but not required), you’ll see this error show up when running DCDIAG.exe, so no real issue at all.

There are lots of posts on the internet pointing to various possible issues or causes: http://social.technet.microsoft.com/Forums/en-US/winserverDS/thread/2ce07c3f-9956-4bec-ae46-055f311c5d96/  & http://social.technet.microsoft.com/Forums/en-IE/winserverDS/thread/3062d40a-b73e-42ea-b27a-e817ee29abc1. But before you worry to much I suggest you check that everything that has to do with replication is running well. Is so and you’re running in Windows 2008 or Windows 2008 R2 domain functional level you’ll see this error go way once you complete your migration to DFRS.

So, to recapture, if you have a well maintained & working Active Directory, do not panic when you see some warning or failures in diagnostic test results. Make sure things are indeed fine and if you conclude that you don’t have any lingering problems, do some further research on what the real reason might . This pahnatom error is a fine example of this.

There is an absolute brilliant step by step guide to get the move from FRS to DFRS completed without a problem in a series by the storage team at Microsoft . You can find the first of a 5 part blog series over here http://blogs.technet.com/b/filecab/archive/2008/02/08/sysvol-migration-series-part-1-introduction-to-the-sysvol-migration-process.aspx.

While you are at it. If your still running DFS in Windows 2000 native mode, you might want to upgrade that as well. More on that later Smile