Upgrade the firmware on a Brocade Fibre Channel Switch

NOTE: content available as pdf download here.

Upgrade the firmware on a Brocade Fibre Channel Switch

In order to maintain a secure, well-functioning fibre channel fabric over the years you’ll need to perform a firmware upgrade now and again. Brocade fibre channel switches are expensive but they do deliver a very solid experience. This experience is also obvious in the firmware upgrade process. We’ll walk through this as a guide on how to upgrade the firmware on a Brocade fibre channel switch environment.

Have a FTP/SFTP/SCP server in place

If you have some switches in your environment you’re probably already running a TFTP or FTP server for upgrading those. For TFTP I use the free but simple and good one provided by Solarwinds. They also offer a free SCP/SFTP solution. For FTP it depends either we have IIS with FTP (and FTPS) set up or we use FileZilla FTP Server which also offers SFTP and FTPS. In any case this is not a blog about these solutions. If you’re responsible for keeping network gear in tip top shape you should this little piece of infrastructure set up for both downloads and uploads of configurations (backup/restore), firmware and boot code. If you don’t have this, it’s about time you set one up sport! A virtual machine will do just fine and we back it up as well as we store our firmware and backups on that VM as well. For mobile scenarios I just keep TFTP & FilleZilla Server installed and ready to go on my laptop in a stopped state until I need ‘m.

Getting the correct Fabric OS firmware

It’s up to your SAN & switch vendors to inform you about support for firmware releases. Some OEMs will publish those on their own support sites some will coordinate with Brocade to deliver them as download for specific models sold and supported by them. Dell does this. To get it select your switch version on the dell support site and under downloads you’ll find a link.

clip_image002

That link takes you to the Brocade download page for DELL customers.

clip_image004

Make sure you download the correct firmware for your switch. Read the release notes and make sure you’re the hardware you use is supported. Do your homework, go through the Brocade Fabric OS (FOS) 7.x Compatibility Matrix. There is no reason to shoot yourself in the foot when this can be avoided. I always contact DELL Compellent CoPilot support to verify the version is support with the Compellent Storage Center firmware.

When you have downloaded the firmware for your operating system (I’m on Windows) unzip it and place the content of the resulting folder in your FTP root or desired folder. I tend to put the active firmware under the root and archive older one as they get replaced. So that root looks like this. You can copy it there over RDP or via a FTP client. If the FTP server is running your laptop, it’s just a local copy.

clip_image005

The upgrade process

A word on upgrading the firmware

I you move from a single major level/version to the next or upgrade within a single major level/version you can do non-disruptive upgrades with a High Availability (HA) reboot meaning that while the switch reloads it will not impact the data flow, the FC ports stay online. Everything keeps running, bar that you lose connectivity to the switch console for a short time.

Some non-disruptive upgrade examples:

V6.3.2e to V6.4.3g

V7.4.0a to v7.4.0b

V7.3.0c to v7.4.0b

Note that this way you can step from and old version to a new one step by step without ever needing downtime. I have always found this a really cool capability.

You can find Brocades recommendations on what the desired version of a major release is in https://www.brocade.com/content/dam/common/documents/content-types/target-path-selection-guide/brocade-fos-target-path.pdf

I tend to way a bit with the latest as the newer ones need some wrinkles taken care of as we can see now switch 7.4.1 which is susceptible to memory leaks.

Some disruptive upgrade examples (FC ports go down):

7.1.2b to 7.4.0a

6.4.3.h to 7.4.0b

Our upgrade here from 7.4.0a to 7.4.0b is non-disruptive as was the upgrade from to 7.3.0c to 7.4.0a. You can jump between version more than one version but it will require a reboot that takes the switch out of action. Not a huge issue if you have (and you should) to redundant fabrics but it can be avoided by moving between versions one at the time. IT takes longer but it’s totally non-disruptive which I consider a good thing in production. I reserve disruptive upgrades for green field scenarios or new switches that will be added to the fabric after I’m done upgrading.

Prior to the upgrade

There is no need to run a copy run or write memory on a brocade FC switch. It persists what you do and you have to save and activate your zoning configuration anyway when you configure those (cfgsave). All other changes are persisted automatically. So in that regards you should be all good to go.

Make a backup copy of your configuration as is. This gives you a way out if the shit hits the fan and you need to restore to a switch you had to reset or so. Don’t forget to do this for the switches in both fabrics, which normally you have in production!

You log on switch with your username and password over telnet or ssh (I use putty or kitty)

MySwitchName:admin> configupload

Hit ENTER

Select the protocol of the backup target server you are using

Protocol (scp, ftp, sftp, local) [ftp]: ftp

Hit ENTER

Server Name or IP Address [host]: 10.1.1.12

HIT ENTER

Enter the user, here I’m using anonymous

User Name [user]: anonymous

Hit ENTER

Give the backup file a clear and identifying name

Path/Filename [<home dir>/config.txt]: MySwitchNameConfig20151208.txt

Hit ENTER

Select all (default)

Section (all|chassis|switch [all]): all

configUpload complete: All selected config parameters are uploaded

That’s it. You can verify you have a readable backup file on your FTP server now.

clip_image007

The Upgrade

A production environment normally has 2 fabrics for redundancy. Each fabric exists out of 1 or more switches. It’s wise to start with one fabric and complete the upgrade there. Only after all is proven well there should you move on to the second fabric. To avoid any impact on production I tend plan these early or late in the day also avoiding any backup activity. Depending on your environment you could see some connectivity drops on any FC-IP links (remote SAN replication FC to IP ó IP to FC) but when you work one fabric at the time you can mitigate this during production hours via redundancy.

Log on to first brocade fabric switch with your username and password over telnet or ssh (I use putty or kitty). At the console prompt type

firmwaredownload

This is the command for the non-disruptive upgrade. If you need or want to do a disruptive one, you’ll need to use firmwaredownload –s.

Hit Enter

Enter the IP address of the FTP server (of the name if you have name resolution set up and working)

Server Name or IP address: 10.1.1.12

User name: I fill out anonymous as this gives me the best results. Leaving it blank doesn’t always work depending on your FTP server.

User Name: anonymous

Enter the path to the firmware, I placed the firmware folder in the root of the FTP server so that is

Path: /v7.4.0b

Hit enter

At the password prompt leave the password empty. Anonymous FTP doesn’t need one.

Password:

Hit enter, the upgrade process preparation starts. After the checks have passed you’ll be asked if you want to continue. We enter Y for yes and hit Enter. The firmware download starts and you’ll see lost of packages being downloaded. Just let it run.

clip_image009

This goes on for a while. At one point you’ll see the prom update happening.clip_image011

When it’s done it starts removing unneeded files and when done it will inform you that the download is done and the HA rebooting starts. HA stands for high availability. Basically it fails over to the next CP (Control Processor, see http://www.brocade.com/content/html/en/software-upgrade-guide/FOS_740_UPGRADE/GUID-20EC78ED-FA91-4CA6-9044-E6700F4A5DA1.html) while the other one reboots and loads the new firmware. All this happens while data traffic keeps flowing through the switch. Pretty neat.

When you keep a continuous ping to the FC switch running during the HA reboot you’ll see a short drop in connectivity.

image

But do realize that since this is a HA reboot the data traffic is not interrupted at all. When you get connectivity back you SSH to switch and verify the reported version, which here is now 7.4.0b.

clip_image014

That’s it. Move on to the switch in the same fabric until you’re done. But stop there before you move on to your second fabric (failure domain). It pays to go slow with firmware upgrades in an existing environment.

This doesn’t just mean waiting a while before installing the very latest firmware to see whether any issues pop up in the forums. It also means you should upgrade one fabric at the time and evaluate the effects. If no problems arise, you can move on with the second fabric. By doing so you will always have a functional fabric even if you need to bring down the other one in order to resolve an issue.

On the other hand, don’t leave fabrics unattended for years. Even if you have no functional issues, bugs are getting fixed and perhaps more importantly security issues are addressed as well as browser and Java issues for GUI management. I do wish that the 6.4.x series of the firmware got an update in order for it to work well with Java 8.x.

Windows Server 2016 Technical Preview Version 3 Cluster Upgrades

I was eagerly awaiting the release of Windows Server 2016 Technical Preview 3 for further experimenting and testing and August 19th 2015was the big day with a truck load of announcements and press releases including the arrival of TPv3 which also made containers publicly available for testing to all of us.

image

After a swift download II set out upgrading the labs, both PC hardware based and enterprise grade server hardware. I always test out the less wise things as well just to kick the tires and test behavior left and right.

As always I tested some in place upgrades just to see how well that goes before doing clean installs . Not recommend in production but hey,Testing is good. At first all networking seem to be OK but it wasn’t. So I ended up with doing clean installs which are advisable, even more so with non production versions of the OS. The product is not finished yet! This is also the supported way of doing a new cluster build. imageThe end result is a lab at home on PC hardware and an enterprise grade lab to work with in the datacenter. Busy times ahead.

For help on what’s new in this build go here What’s New in Windows Server 2016 Technical Preview 3 and good luck on your Windows Server 2016 Technical Preview Version 3 cluster upgrades!

Happy testing!

Upgrading Hyper-V Cluster Nodes to Windows 8 (Beta) – Part 2

This is a multipart series based on some lab test & work I did.

  1. Part 1 Upgrading Hyper-V Cluster Nodes to Windows Server 8 (Beta) – Part 1
  2. Part 2 Upgrading Hyper-V Cluster Nodes to Windows 8 (Beta) – Part 2
  3. Part 3 Upgrading Hyper-V Cluster Nodes to Windows 8 (Beta) – Part 3

Here’s part two of my adventures while upgrading or rather “transitioning” my Hyper-V cluster nodes to Windows 8. Transition is more correctly as you can not upgrade a cluster, you create a new cluster en recuperate the node. I did however not reinstall them but upgrade them. Why, because I can and I wanted to try it out to see what happens. For production purposes I do advise you to rebuild nodes from scratch using a well defined and automated plan if possible. I already mentioned this in Upgrading Hyper-V Cluster Nodes to Windows Server 8 (Beta) – Part 1

So we stopped Part 1 with a evicted and upgraded node. We’ll want to create a new cluster with that node and then transition the other nodes over to the new Windows 8 cluster one by one, or in batches, depending on how many you can afford to take down at one time. In this part we’ll just build our new Window 8 cluster with a single node. It’s a good thing this is possible as we can start a transition with just one node. This an easy part.

First of all we create a new cluster. I will all look very familiar if you’ve ever created a Windows 2008 (R2) cluster.

image

 

The Create Cluster Wizard appears, read all the advice you want and click “Next”

02

 

We select the node that we evicted from the old cluster and upgraded to Windows 8

03

04

 

You now run the validation test for your cluster

06

Let’s run ‘m all and see what it has to say.

07

 

We get a summary of what notes will be tested and what tests will be run. Click “Next”

08

 

The tests are running.

09

 

We get a pass with some warnings. So we click “View Report” to take a look. It’s OK we only have one node, we don’t have storage yet and networking wise we still need to configure some things but we can create a one node cluster, So click “Finish”

10

image

 

I named my new cluster “warriors”, the old one was called “warrior”.

12

 

I define the IP Address for the Access Point for administering the cluster

13

 

We’re ready to create the cluster so we click “Next” and the creation process starts

15

16

 

And we’re informed we’ve have successfully created a cluster. Click Finish. Any experienced cluster builder should find this process very familiar without surprises.

17

 

So now we have a cluster existing out of one node and we haven’t got any storage assigned yet.

We have several options for storage here. We could assign new storage but we cannot do a Quick Storage Migration between cluster using SCVMM2008R2 but that doesn’t fly as SCVMM2008R2 can’t manage Windows 8 clusters and I don’t know if it ever will.  We can do a good old manual or scripted export and import of the VMs what takes a considerable amount of time.

We can recuperate the old storage with the VMs still on there. This could get tricky as no two cluster should be able to see & use the storage at the same time. The benefit could be that we can just use the import type in Windows 8 ("Register the virtual machine in-place" (use the existing unique ID) and be done with it. We’ll try that one. We’ll still have some down time but it should be pretty fast. It’s only from Windows 8 on that we’ll be able to do Shared Nothing Live Migrations between clusters Smile We’ll address that in Part 3.

Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 1

This is a multipart series based on some lab test & work I did.

  1. Part 1 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 1
  2. Part 2 Upgrading Hyper-V Cluster Nodes to Windows Server 2012 (Beta) – Part 2
  3. Part 3 Upgrading Hyper-V Cluster Nodes to Windows 8 (Beta) – Part 3

After I got back from the MVP Summit 2012 in Bellevue/Redmond I could wait to start playing with a Windows 8 Hyper-V cluster so I decided to upgrade my Windows 2008 R2 cluster nodes to Windows 8. That means evicting them on by one, upgrading them and adding them to a new Windows 8 cluster. As we can build a one node cluster this can be done a node at the time. This isn’t a fail proof definite “How To”, I’m just sharing what I did.

Evicting a node

Before evicting a node make sure all virtual machines are running on the other node(s). As you can see the cluster warrior has 2 nodes, crusader & saracen (I was listening to some Saxon heavy metal at the time I built that lab setup). We evacuated node saracen prior to evicting it.

image

Evict the node & confirm when asked.

image

image

When this is done all storage is off line to the node evicted from the cluster. No need to worry about that.

Upgrade that node to Windows 8

To anyone having installed/upgraded to Windows 2008 R2 this should all be a very recognizable experience. Being lazy, I left the iSCSI initiator configuration in there with the Hyper-V & failover cluster roles installed during the upgrade. Now for production environments I like to build my nodes from scratch to have an exactly known, new and clean installation base. But for my test lab at home I wanted to get it done as fast as possible. If only the days had more hours …For extra safety you can pull the plug (or disable the switch ports) on your iSCSI or FC connections and make sure no storage is presented to the node during the upgrade process. Now please do mind is use Intel server grade NIC adaptors for which Windows 8 beta has drivers. Your situation may vary so I can’t guarantee the 7 year old FC HBA in your lab server will just work, OK!?

So run setup.exe from the Windows 8 (Beta) ISO you extracted to a folder on the server or  from the (bootable) USB you created with the downloaded ISO.

image

 

The Windows Setup installer will start.

04 run setup

 

Click on “Install now” to proceed and start the setup process.

image

 

Select to “Go online to get the latest updates for Setup (Recommended)”

image

 

So it looks for updates on line.

image

 

It didn’t find any but that’s OK.

image

 

Select the installation you want. I went with for Server with a GUI as I want screen shots. But as I wrote in the blog post Windows 8 Server With GUI, Minimal Server Interface & Server Core Lesson with the Desktop Experience Feature you can turn it into a Server Core Installation and back again now. So no regrets with any choice you make here, which is a nice improvement that can save us a lot of time.

image

Accept the EULA

image

 

We opt to upgrade (in production I go for a clean install)

image

 

I get notified that I have to remove PerfectDisk. I had an evaluation copy of Raxco PerfectDisk installed I used to do some testing with redirected CSV traffic and defragmentation (see Some Feedback On How to defrag a Hyper-V R2 Cluster Shared Volume).

image

 

So the upgrade was cancelled.

image

 

I uninstalled PerfectDisk but still it was a no go. I  had to remove all traces of it in the registry & files systems that the uninstall left or the upgrade just wouldn’t start. But after that it worked.

image

 

That means we can kick of the upgrade! It all looks very familiar Smile It takes a couple of reboots and some patience. But all in all it’s a fast process.

image

image

image

image

After this step it takes a couple of reboots and some patience. But all in all it’s a fast process. After some reboots and a screen that goes dark in between those …we get our restyled beta fish.

image

image

image

And voila we’re where we need to be … Smile

image

 

After the upgrade process I ran into one error. The GUI for Failover Clustering would not start. The solution if found for that was simply to remove that role and add it again. That did the trick.

ClusGUI

 

So this was a description of the first steps to transition a  Windows 2008 R2 SP1 cluster to a  Windows 8 (Beta) Cluster. As seen we evict the nodes one by one to upgrade them or do a clean install. In the latter case you’ll need to do the iSCSI initiator configuration again,  install the Failover Cluster role and in the case of a Hyper-V cluster the Hyper-V role. The nodes can than be added to a new Windows 8 cluster, starting out with a one node cluster. More on that in the second part of this blog post.