Tag Archives: Firmware

Upgrade the firmware on a Brocade Fibre Channel Switch

Posted on by
3

NOTE: content available as pdf download here.

Upgrade the firmware on a Brocade Fibre Channel Switch

In order to maintain a secure, well-functioning fibre channel fabric over the years you’ll need to perform a firmware upgrade now and again. Brocade fibre channel switches are expensive but they do deliver a very solid experience. This experience is also obvious in the firmware upgrade process. We’ll walk through this as a guide on how to upgrade the firmware on a Brocade fibre channel switch environment.

Have a FTP/SFTP/SCP server in place

If you have some switches in your environment you’re probably already running a TFTP or FTP server for upgrading those. For TFTP I use the free but simple and good one provided by Solarwinds. They also offer a free SCP/SFTP solution. For FTP it depends either we have IIS with FTP (and FTPS) set up or we use FileZilla FTP Server which also offers SFTP and FTPS. In any case this is not a blog about these solutions. If you’re responsible for keeping network gear in tip top shape you should this little piece of infrastructure set up for both downloads and uploads of configurations (backup/restore), firmware and boot code. If you don’t have this, it’s about time you set one up sport! A virtual machine will do just fine and we back it up as well as we store our firmware and backups on that VM as well. For mobile scenarios I just keep TFTP & FilleZilla Server installed and ready to go on my laptop in a stopped state until I need ‘m.

Getting the correct Fabric OS firmware

It’s up to your SAN & switch vendors to inform you about support for firmware releases. Some OEMs will publish those on their own support sites some will coordinate with Brocade to deliver them as download for specific models sold and supported by them. Dell does this. To get it select your switch version on the dell support site and under downloads you’ll find a link.

clip_image002

That link takes you to the Brocade download page for DELL customers.

clip_image004

Make sure you download the correct firmware for your switch. Read the release notes and make sure you’re the hardware you use is supported. Do your homework, go through the Brocade Fabric OS (FOS) 7.x Compatibility Matrix. There is no reason to shoot yourself in the foot when this can be avoided. I always contact DELL Compellent CoPilot support to verify the version is support with the Compellent Storage Center firmware.

When you have downloaded the firmware for your operating system (I’m on Windows) unzip it and place the content of the resulting folder in your FTP root or desired folder. I tend to put the active firmware under the root and archive older one as they get replaced. So that root looks like this. You can copy it there over RDP or via a FTP client. If the FTP server is running your laptop, it’s just a local copy.

clip_image005

The upgrade process

A word on upgrading the firmware

I you move from a single major level/version to the next or upgrade within a single major level/version you can do non-disruptive upgrades with a High Availability (HA) reboot meaning that while the switch reloads it will not impact the data flow, the FC ports stay online. Everything keeps running, bar that you lose connectivity to the switch console for a short time.

Some non-disruptive upgrade examples:

V6.3.2e to V6.4.3g

V7.4.0a to v7.4.0b

V7.3.0c to v7.4.0b

Note that this way you can step from and old version to a new one step by step without ever needing downtime. I have always found this a really cool capability.

You can find Brocades recommendations on what the desired version of a major release is in https://www.brocade.com/content/dam/common/documents/content-types/target-path-selection-guide/brocade-fos-target-path.pdf

I tend to way a bit with the latest as the newer ones need some wrinkles taken care of as we can see now switch 7.4.1 which is susceptible to memory leaks.

Some disruptive upgrade examples (FC ports go down):

7.1.2b to 7.4.0a

6.4.3.h to 7.4.0b

Our upgrade here from 7.4.0a to 7.4.0b is non-disruptive as was the upgrade from to 7.3.0c to 7.4.0a. You can jump between version more than one version but it will require a reboot that takes the switch out of action. Not a huge issue if you have (and you should) to redundant fabrics but it can be avoided by moving between versions one at the time. IT takes longer but it’s totally non-disruptive which I consider a good thing in production. I reserve disruptive upgrades for green field scenarios or new switches that will be added to the fabric after I’m done upgrading.

Prior to the upgrade

There is no need to run a copy run or write memory on a brocade FC switch. It persists what you do and you have to save and activate your zoning configuration anyway when you configure those (cfgsave). All other changes are persisted automatically. So in that regards you should be all good to go.

Make a backup copy of your configuration as is. This gives you a way out if the shit hits the fan and you need to restore to a switch you had to reset or so. Don’t forget to do this for the switches in both fabrics, which normally you have in production!

You log on switch with your username and password over telnet or ssh (I use putty or kitty)

MySwitchName:admin> configupload

Hit ENTER

Select the protocol of the backup target server you are using

Protocol (scp, ftp, sftp, local) [ftp]: ftp

Hit ENTER

Server Name or IP Address [host]: 10.1.1.12

HIT ENTER

Enter the user, here I’m using anonymous

User Name [user]: anonymous

Hit ENTER

Give the backup file a clear and identifying name

Path/Filename [<home dir>/config.txt]: MySwitchNameConfig20151208.txt

Hit ENTER

Select all (default)

Section (all|chassis|switch [all]): all

configUpload complete: All selected config parameters are uploaded

That’s it. You can verify you have a readable backup file on your FTP server now.

clip_image007

The Upgrade

A production environment normally has 2 fabrics for redundancy. Each fabric exists out of 1 or more switches. It’s wise to start with one fabric and complete the upgrade there. Only after all is proven well there should you move on to the second fabric. To avoid any impact on production I tend plan these early or late in the day also avoiding any backup activity. Depending on your environment you could see some connectivity drops on any FC-IP links (remote SAN replication FC to IP ó IP to FC) but when you work one fabric at the time you can mitigate this during production hours via redundancy.

Log on to first brocade fabric switch with your username and password over telnet or ssh (I use putty or kitty). At the console prompt type

firmwaredownload

This is the command for the non-disruptive upgrade. If you need or want to do a disruptive one, you’ll need to use firmwaredownload –s.

Hit Enter

Enter the IP address of the FTP server (of the name if you have name resolution set up and working)

Server Name or IP address: 10.1.1.12

User name: I fill out anonymous as this gives me the best results. Leaving it blank doesn’t always work depending on your FTP server.

User Name: anonymous

Enter the path to the firmware, I placed the firmware folder in the root of the FTP server so that is

Path: /v7.4.0b

Hit enter

At the password prompt leave the password empty. Anonymous FTP doesn’t need one.

Password:

Hit enter, the upgrade process preparation starts. After the checks have passed you’ll be asked if you want to continue. We enter Y for yes and hit Enter. The firmware download starts and you’ll see lost of packages being downloaded. Just let it run.

clip_image009

This goes on for a while. At one point you’ll see the prom update happening.clip_image011

When it’s done it starts removing unneeded files and when done it will inform you that the download is done and the HA rebooting starts. HA stands for high availability. Basically it fails over to the next CP (Control Processor, see http://www.brocade.com/content/html/en/software-upgrade-guide/FOS_740_UPGRADE/GUID-20EC78ED-FA91-4CA6-9044-E6700F4A5DA1.html) while the other one reboots and loads the new firmware. All this happens while data traffic keeps flowing through the switch. Pretty neat.

When you keep a continuous ping to the FC switch running during the HA reboot you’ll see a short drop in connectivity.

image

But do realize that since this is a HA reboot the data traffic is not interrupted at all. When you get connectivity back you SSH to switch and verify the reported version, which here is now 7.4.0b.

clip_image014

That’s it. Move on to the switch in the same fabric until you’re done. But stop there before you move on to your second fabric (failure domain). It pays to go slow with firmware upgrades in an existing environment.

This doesn’t just mean waiting a while before installing the very latest firmware to see whether any issues pop up in the forums. It also means you should upgrade one fabric at the time and evaluate the effects. If no problems arise, you can move on with the second fabric. By doing so you will always have a functional fabric even if you need to bring down the other one in order to resolve an issue.

On the other hand, don’t leave fabrics unattended for years. Even if you have no functional issues, bugs are getting fixed and perhaps more importantly security issues are addressed as well as browser and Java issues for GUI management. I do wish that the 6.4.x series of the firmware got an update in order for it to work well with Java 8.x.

Upgrading The DELL MD3600F Controller Firmware Using the Modular Disk Storage Manager

Posted on by
4

As part of our hardware maintenance we’re deploying the updates of SUU 7.3.1 right now. As part of that effort we’re also making sure the PowerVault disk bays are getting their updates. We’ve got a couple of PowerVault MD3600F with 7 MD1200 extension bays filled with disks attached to R710  Power Edge servers that act as Disk2Disk backup media servers with 10Gpbs networking and running W2K12R2.

In general you should not be applying firmware updates and such constantly and all over the place just to keep busy. It is, however, a good practice to keep an eye on releases and see if they fix any risky bugs you might be vulnerable to. You see that sometimes these fix issues you don’t want to run into and those are often marked urgent. You also don’t get to far behind on firmware updates as this can complicate thing later in the useful service life or you hardware. First of all read the readme.txt and such to make sure you’re not missing any special instructions for particular cases that might exist. The standard disclaimer of my blog applies Smile, you’re responsible for your own actions.

First of all get the latest version of your PowerVault Modular Disk Storage Manager software if you haven’t already. Often you’ll need to have a least a certain version before you can even do newer firmware updates. So just get this in order before starting, especially if they tell you to, don’t try and outsmart the system. Download the disk firmware form the DELL support website or via the DELL Storage Community Wiki http://en.community.dell.com/techcenter/storage/w/wiki/4234.dell-powervault-md-downloads.aspx?dgc=SM&cid=257966&lid=4630585 and safe it to disk.

Here’s an example of a read me file you’d better not ignore. If you do, the Modular Disk Storage Manager will not allow you to upgrade anyway.

*** ATTENTION ***
If your PowerVault MD32xx/MD36xx series storage array is currently running a firmware version lower than 07.75.28.60, you cannot directly install the RAID controller firmware version 07.84.47.60 included in this release.
Instead, you must first install the bridge firmware version 07.75.28.60. After the bridge firmware is successfully installed, you can then install RAID controller firmware 07.84.47.60.

Specifically, follow these steps:
1. *** IMPORTANT *** Install Dell PowerVault MD Storage Manager (MDSM) software from the Resource DVD version 4.1.2.29 or higher.
2. Extract the following files from the Bridge_Firmware_07_75_28_60.zip archive:
* Bridge_Firmware_07_75_28_60.dlp
3. Download and activate RAID controller bridge firmware version 07.75.28.60 (contained in the Bridge_Firmware_07_75_28_60.dlp file)
4. Once the bridge firmware installation is successfully completed, wait at least 30 minutes to allow the attached hosts to re-discover the storage array.
5. Download and activate the RAID controller firmware version 07.84.47.60 (contained in the MD3xxx_MD3xxx_Firmware_07_84_47_60.dlp file) together with the NVSRAM configuration file version N26X0-784890-X04 (contained in the MD3xxx_MD3xxx_NVSRAM_N26X0-784890-X04.dlp file) using the Dell PowerVault MD Storage Manager for your corresponding array type.
* MD3200_MD3220_Firmware_07_84_47_60.dlp and MD3200_MD3220_N26X0_784890_004.dlp (6G Non-Dense SAS)
* MD3200i_MD3220i_Firmware_07_84_47_60.dlp and MD3200i_MD3220i_N26X0_784890_004.dlp (1G Non-Dense iSCSI)

* MD3600i_MD3620i_Firmware_07_84_47_60.dlp and MD3600i_MD3620i_N26X0_784890_904.dlp (10G Non-Dense iSCSI)
* MD3600f_MD3620f_Firmware_07_84_47_60.dlp and MD3600f_MD3620f_N26X0_784890_904.dlp (8G Non-Dense FC)
* MD3260_Firmware_07_84_47_60.dlp and MD3260_N26X0_784890_004.dlp (6G Dense SAS)
* MD3260i_Firmware_07_84_47_60.dlp and MD3260i_N26X0_784890_004.dlp (1G Dense iSCSI)
* MD3660i_Firmware_07_84_47_60.dlp and MD3660i_N26X0_784890_904.dlp (10G Dense iSCSI)
* MD3660f_Firmware_07_84_47_60.dlp and MD3660f_N26X0_784890_904.dlp (8G Dense FC)

*** WARNING ***
If you have a single controller PowerVault MD32/MD36 series storage array you must stop all I/O operations before starting the RAID controller firmware upgrade.

Launch the PowerVault Modular Disk Storage Manager, right click your array and select Upgrade Raid Controller Module Firmware.image

Wait while the status is being refreshed.Image 017

You might run into a an issue if the event log contains to many entries. In that case you’ll be warned you can’t upgrade until you’ve cleared it as shown below. Here are some screenshots from another previous upgrade where I ran into this issue.Image 018

In that case open up the Manage Storage Array  for your array.image just

Navigate to the support tab on your and got to “View Event Log”. image

It’s always wise to have a look to see if you have any issues before upgrading anyway. If you need to save the log for some reason do so otherwise just clear it with the Clear All button.image

The lawyers need assurance you confirm that you know what you’re doing so type yes in the warning form and click OK.

image 

You get an entry telling you the log has been cleared and you can close the form.image

If you launch the Upgrade Raid Controller Module Firmware menu option again you’ll now see that you’re good to go with the upgradeImage 024

Click on Download, Firmware and browse to the  and the NVSRAM. I prefer to do both in one go but you don’t have to. Just make sure that when both need an upgrade you don’t forget to do it when upgrading them separately. You can opt to download the firmware and  NVSRAM but activate them later. I normally do it all in one go (the default option).image

They’ll warn you no to do silly things, meaning you have to make sure that the firmware and NVSRAM versions are compatible. Read the documentation to make sure your OK.

image

If you’re good to go, click Yes and the update kicks of. First it’s the firmware update that runs.

image

When that’s done you’ll see the new firmware version in the Pending Version column. After that the NVSRAM update kicks in automatically. This takes a bit longer.Image 013

Once it’s done you get the green check indicating your firmware has upgraded successfully. We chose to activate in the same run but it’s not visible yet.image

Now to see this reflected I the version columns you‘ll need to close and reopen the Upgrade Raid Controller Module Firmware wizard again.
image

Close the wizard. You’re done upgrading the Raid Controller Firmware. Next things to do would be to upgrade disk firmware and perhaps the EMM (Enclosure Management Module) firmware of the extension disk bays if applicable. They

Upgrading Firmware Of Mellanox RoCE Cards for Final Windows Server 2012 RDMA Testing

Posted on by
3

Upgrading Mellanox Firmware

As we are preparing to roll out Windows Server 2012 R2 we are also updating the Mellanox cards we have. At the moment of writing the final driver & firmware for Windows Server 2012 R2 isn’t out yet, but let’s take a look at the process so you’re ready for prime time. If you need the latest public Mellanox driver for Windows Server 2012 R2 it’s here. Installing the driver is a straight forward process (upgrading servers with Mellanox drivers in place has been an issue however).

Mellanox provides good documentation on their site (http://www.mellanox.com/page/firmware_HCA_FW_identification &  http://www.mellanox.com/page/firmware_NIC_FW_update) but for Mellanox newbies & many Windows server admins the process might be a bit more hands on than via a single installer they are used to.

What do you need?

The Windows Mellanox Firmware Tools (WinMFT). This gives you all the tools you need to get the job done.

It helps us with two things: find out Card ID and using that we can determine the PSID (Board ID) which tells us what firmware we need to down load.

The Win MFT tools are also used to burn the firmware.

Practical Tip 1: I have found that it pays to launch the installers Mellanox provides from an elevated command prompt as other wise UAC might trip up some clean finalization of a launched msi. The driver installer is more sensitive to this that the firmware installer.

Practical Tip 2: I you have OEM Mellanox cards from DELL/ HP/IBM … and they haven’t released the new firmware yet you can always burn your own. Please find the instructions here.

Walkthrough

I have a Windows Server 2012 R2 RTM running and I already installed the latest beta drivers I could find on the Mellanox site. But I’m a firmware version behind. So let’s fix this.

image

I put all the files I need in one handy spot

image

I launch an elevated command prompt

image

And from there I lauch the WinMFT installer

image

Just follow the instructions. image

image

image

image

image

Now you’re ready to determine the Device ID of your Mellanox card. From that same elevated command prompt navigate to C:Program FilesMellanoxWinMFT and run mst status

image

Grab the Device ID (marked in green) and execute following command:

flint -d /dev/mst/mt4099_pci_cr0 query

image

The Board ID (marked in yellow) is actually the PSID (more information here) will tell you what firmware to download from the Mellanox site). By the way, note this also tells you the current firmware.

You download the firmware from http://www.mellanox.com/page/firmware_download by selecting the card you have. In my case a ConnectX®-3 EN PCI-Ex Network Interface Cards (Ethernet Only NICs) and is use the Board ID to find my download.

image

All that’s left to do is burn the firmware image by executing the following command:

flint -d /dev/mst/mt4099_pci_cr0  -i C:SysAdminMellanoxFirmwarefw-ConnectX3-rel-2_30_3000-MCX312A-XCB_A2-A6-3.4.142_EN.bin burn

This requires you to confirm by typing in “y” and you can follow the process via a counter.image

When done you’ll need to reboot the server I order for the new firmware to actually get used. You can verify success by running the command again or by checking the information tab of you cards configuration settings. As you can see we’re running 2.30.3000 now.

image

So here you go. You might need to do this again after October 18th 2013 but you’re ready for now and all the testing you do is on the latest version of both the driver and the firmware. Happy testing!

Upgrading Your DELL Compellent Storage Center Firmware (Part 2)

Posted on by
7

This is Part 2 of this blog. You’ll find Part 1 over here.

In part 1 we prepared our Compellent SAN to be ready and install Storage Center 6.3.10 that has gone public.  As said, 6.3.10 brings interesting features like ODX and UNMAP to us Windows Server 2012 Hyper-V users. It also introduces some very nice improvements to synchronous replication and Live Volumes. But here we’ll just do the actual upgrade, the preparations & health check have been done in part 1 so we can get started here right away.

Log in to your Compellent system and navigate to the Storage Management menu. Click on “System”, select Update and finally click on “Install Update”.  It’s already there as we downloaded it in Part 1. Click on “Install Now” to kick it all off.

image

Click on Install now to launch the upgrade.

image

After initialization you can walk away for 10 minutes but you might want to keep an eye on things and the progress of the process.

image

So go have a look at your storage center. Look at the Alert Monitor for example and notice that the “System is undergoing maintenance”.

image

When the controller the VIP address of the SAN reboots it becomes unavailable. After a while you can login again to the other controller via the VIP, if you cant’ wait a few seconds just use the IP address of the active controller. That will do.

image

When you log in again you’ll see the evidence of an ongoing SAN firmware upgrade. Nothing to panic about.image

This is also evident in Alert Monitor. CoPilot knows you’re doing the upgrade so no unexpected calls to make sure your system is OK will come in. They’re there every step of the way. The cool thing is that is the very first SAN we ever owned that we don’t need engineers on site or complex and expensive procedure to do all this. It’s all just part of an outstanding customer service Compellent & DELL deliver.image

You can also take a peak at your Enterprise manager software to see paths going down and so on. The artifacts of a sequential controller failovers during an upgrade. Mind you you’re not suffering downtime in most cases.image

Just be patient and keep an eye on the process. When you log in again after the firmware upgrade and your system is up and running again, you’ll be asked to rebalance the ports & IO load between the controllers on the system. You do, so click yes.image

image

When done you’ll return to the Storage Center interface. Navigate to “Help”" and click on About Compellent Storage Center. image

You can see that both controllers are running 6.3.10.

image

You’re rocking the new firmware. As you kept an eye on your hosts you should know these are good to go. Send of an e-mail to CoPilot support and they’ll run a complete health check on your system to make sure you’re good to go. Now it’s time to start leveraging the new capabilities you just got.