Dell iDRAC 6 Remote Console Connection Failed

Posted on by
26

Dell iDRAC 6 Remote Console Connection Failed

I recently had the honor to fix a real annoying issue with the iDRAC on rather old DELL hardware, R710 servers that are stilling puling their weight. They have been upgraded to the latest firmware naturally and DELL allows access to those updates to anyone without the need for a support contract (happy users/customers).You can perfectly configure Java site exceptions and use Firefox or Chrome to connect to it (IE is different story, you can connect but the view is messed up). Anyway the browser isn’t the big issue. The  problem was that Dell iDRAC 6 remote console connection failed consistently at the very last moment with “Connection Failed”

image

image

Note: are you nuts?

Yes I like 25/50/100Gbps RDMA, S2D, All Flash etc. I do live the vanguard live on the bleeding edge, but part of that is funding solutions that fit the environment. In this case. They have multiple spare servers and extra disks on top the ones they use in the lab or even in production. So even when a server or a component fails they can use that to fix it. They have the hands on and savvy staff members to do that. No problem. This is not an organization driven by fear of risk and responsibility but by results and effective TCO/ROI. They know very well what they can handle and what not. On top of that they know very well what part of IT sectors sales and marketing promises/predictions are FUD and which are reality. This means they can make decisions based on optimizing for their needs delivering real results.

Leveraging old hardware does mean that sometimes you’ll  run into silly issues but annoying issues like older DRAC cards with modern client operating systems, browsers and recent Java versions.

Most tricks are to be found on line to get those to work together but sometimes even those fails. First of all make sure all network requirements are in order (ports, firewall etc) and on top of that:

  • Upgraded the DRAC Firmware to the latest v2.85
  • Add DRAC IP into the Java Exception List.
  • Change Java Network Setting from Browser to Direct Connect
  • Hack the Java config files
  • Disable Encrypted Video on the DRAC
  • Reset the DRAC
  • On top of this you can run and older version of the browser and Java but at a certain point this becomes a silly option. You see at a given moment the entire stack as moved ahead and one trick like running an old version of Java won’t do it anymore and keeping a VM around that’s at a 10 year old tech/version level is a pain.

The missing piece for me: generate & upload SHA256 certs

So let me share you what extra step got the remote console of the DELL R710 iDRAC to work with the most recent version of Java, Windows 10 and the latest of the greatest Firefox browser at the time of writing.

The trick that finally did it is to generate a CSR on the DRAC while you are connected to it. You see, many people never upload their own certs and if they did, it might have been many years ago. Those old SHA1 certs are frowned upon by modern browsers and Java.

image

image

Open the CSR file, copy the content and submit it to a PKI you have or a free one on line like at getacert.com. Just fill out some random info in the request and you’ll get a SHA256 cert for download immediately that “valid” a couple of months. Enough for testing or getting out of a pickle. Your own corporate CA will do better for long term needs.

image

On top of that you’ll need to reset the DRAC card and give it a few minutes.

image

Reconnect to the DRAC and after that, without failure, we could connect to the on all R710 servers where before we kept getting the dreaded “Connection Failed” error otherwise.

That’s it! Good luck.

Hyper-V integration components 6.3.9600.18692

Posted on by
6

After the July 2017 round of patching we got a new version of the Hyper-V integration components on Windows Server 2012 R2. Yes, something that you no longer need to deal with manually since Windows Server 2016. But hey, my guess is that many of you are still taking care of Windows Server 2012 R2 Hyper-V deployments. I’m still taking care of a couple of Windows Server 2012 R2 Clusters, so don’t be shy now.

The newest version (at the time of writing) is 6.3.9600.18692 and 1st appeared in the June 27, 2017—KB4022720 (Preview of Monthly Rollup) update. It has since  been release in the July 11, 2017—KB4025336 (Monthly Rollup) update. You can follow up on the versions of the IC via this link Hyper-V Integration Services: List of Build Numbers

image

That means that you’ll need to upgrade the integration components for the VMs running on your Hyper-V (cluster) nodes after patching those.

image

And yes despite some issues we have seen with QA on updates in the past we still keep our environment very well up to date as when doing balanced risk management the benefits of a modern, well patched environment are very much there. Both for fixing bugs and mitigating security risks. Remember WannaCry ?

So my automation script has run against my Windows Server 2012 R2  Clusters. have you taken care of yours? I did adapt it to deal with the ever growing number of Windows Server 2016 VMs we see running, yes even on Windows Server 2012 R2 Hyper-V hosts.

image

Vast Experience in NoSQL design

Posted on by
Reply

Staff with a vast experience in NoSQL design?

Some days you come across implementations that make you doubt the sanity of the IT industry. That and our collective ability to learn and make progress.

Note: No fungi, plants, animals or humans were hurt due to the below. It did hurt the psychological well being of one DBA. He’s recovering well en doing fine. thank you for asking.

Some time back I came across this “beauty” at a company that spends many millions per year on developer staff and consultants. It makes uses of modern tools, technologies, frame works. They have coaches for anything you can imagine etc. From a budget and resource perspective they checks all the right boxes. They have analysts, project managers and ITIL with change board and all. Apparently they’re also very deeply invested in and have a staff with a vast experience in NoSQL design. Pun intended.

So on a Windows Server 2016 virtual machine, running SQL Server 2016 there was a request to implement a change to a table: add a ‘MonthNumber’ column to a table. That’s when you find these gems of table design:

image

It’s in Dutch but most of you will notice what’s the issue here.  The optimists call this “NOSQL” I’m sure, and they’re welcome to it Smile.

You don’t have to fight all fights

The good news is they don’t design mission critical systems, don’t do civil engineering or develop software for life or death surgery with robotics. Normally when it’s something that matters this sets off all the alarms and we have to intervene. But they’re quite happy with it and it’s all ITIL approved and compliant with the DevOps  &  agile principles they adhere to. That’s quite a challenge for any organization, to combine these successfully.

I have no skin in this game, so I quietly walked away and when the column was added they were happy and at peace. This fight is not mine, not today.

The Federation Service was unable to create the federation metadata document as a result of an error.Document Path: /FederationMetadata/2007-06/FederationMetadata.xml

Posted on by
2

While working on upgrading a Windows 2012 R2 ADFS Farm to Window Server 2016 I noticed the worried looks of the systems administrators while looking at a warning in the ADFS event log, which they wanted to trouble shoot. I knew they had a hardware load balancer in place which made me 99.999% sure it wasn’t a real issue. You see, early documentation on configuring load balancing for and ADFS farm was often configured with a health check for the following url: /FederationMetadata/2007-06/FederationMetadata.xml. This leads you to an XML file that should be available on a working ADFS node.

This works fine. The Kemp Loadmaster knows the ADFS nodes are functional or not and can do it’s job. There’s a nagging issue however. The ADFS log on the ADFS farm node keep logging every health check with a warning

Event ID 143 AD FS

The Federation Service was unable to create the federation metadata document as a result of an error.Document Path: /FederationMetadata/2007-06/FederationMetadata.xml

Additional Data
Exception details:
System.Net.HttpListenerException (0x80004005): The specified network name is no longer available at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size) at Microsoft.IdentityServer.Service.FederationMetadata.SamlMetadataListener.OnGetContext(IAsyncResult result)

As you cans see it just fills the logs every 9 seconds (the frequency of the health check).

image

This leads to hunting for a “ghost” issue that’s actually only an artefact of checking for .

Kemp has updated their documentation with 2 other values for the health check url to use. The good news is these don’t cause the above artefact of logging warning to the ADFS event log. These 2 options are:

  • /adfs/services/trust/mex

image

This leads to an XML file as well but it doesn’t cause the warning to be logged.

image

  • /adfs/ls/idpInitiatedSignon.aspx.

This leads to the ADFS login page which also doesn’t cause a warning to be logged.

So by changing your health check to any of the  above you get a functional health check for your nodes and you don’t have the phantom warning entries in the ADFS event log. That’s a lot better and at least doesn’t cause any unneeded concerns by the initiated accidental ADFS administrator.