Category Archives: Networking

Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

Posted on by
Reply

Introduction

As you might remember I wrote a blog post about SFP+ and SFP28 compatibility. In this i discuss future proofing your network investments and not having to upgrade everything all at once. One example is that buying 25Gbps NICs when your main network infrastructure is still on 10Gbps is not an issue. 25Gbps normally handles 10Gbps well so you don’t have do replace all parts in the fabric at the same time but you can start with either the network fabric or the server NICs. It’s a way of future proofing the investments you make today.

When installing Mellanox ConnectX-4 Lx 25Gbps NICs in a bunch of servers we hit an issue when connected them to the DELLEMC N4000 10Gbps switches. The intent is to replace these with 25/50/100Gbps in the future.

The links did not come up.

The links did not come up. The switch ports are normally forced 10 Gbps in our setups so we check that. The speed was indeed set fix to 10Gbps. When changing that to auto-negotiate the link would come up at 1Gbps.

Naturally you check everything from cabling to used transceivers (BCM84754 on the switches) but that all checked out. We also check the firmware on the switches to determine if they were up to date and perhaps a new version fixed a known issue related to this. But no hardware wise everything was up to date on the switches and on the NICs.

Note that these links worked fine when used with 10 Gbps cards like the ConnectX-3 Pro. The DELL branded transceivers on the switches were BCM84754 (Broadcom)

The fix: Force Mellanox ConnectX-4 Lx 25Gbps to 10Gbps speed

I do not need to tell you that when you want 10Gbps getting 1Gbps doesn’t fly well. The fix was easy enough. We put the switch ports back to 10Gbps fixed speed. Auto-negotiate doesn’t deliver. No worries we fix the ports anyway. We then used mlc5cmd.exe Mellanox tool to change the NIC ports from auto-negotiate to fixed.

On hosts with Mellanox Connect-X4 NICs you open an elevated command prompt.

Navigate to C:\Program Files\Mellanox\MLNX_WinOF2\Management Tools. Run the below command to check the current link speed.

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Query

Note 10 and 25 Gbps are supported, so it’s autonegotiate.

We force the link speed to 10Gbps:

mlx5cmd.exe -LinkSpeed -Name “MyNicName ” -Set 10

Link speed is forced to 10Gbps

The link comes up at 10Gbps

Likewise you can force the link to 25Gbps. If you want to change it back to the default you can force the link speed to auto-negotiate again.

mlx5cmd.exe -LinkSpeed -Name “MyNicName” -Set 0

See https://community.mellanox.com/s/article/mlx5cmd-exe for more information on this tool.

Do note that the switch port also needs to be set to 10Gbps fixed. As you can see below the command will notify you when those are still on auto.

The change was done but still no uplink when the switch port isn’t fixed to 10Gbps.

Conclusion

So my statement hold true the path to 25/50/100Gbps is one you can do step by step with future proofing. You might run into some issues but these are normally fixable. I have shared with you how to fix failing or wrong speed negotiations on 25 Gbps RDMA NICs (Mellanox ConnectX-4 Lx) when connecting to 10Gbps ports. I’m pretty sure the same holds true for other models. I have also had cards where things work out of the box but don’t give up when you hit an issue. I hope this helps some of you out there.

Windows Server 2019 SMB Direct Best Practices

Posted on by
Reply

Windows Server 2019 SMB Direct Best Practices

The Hyper-V amigos, @Hypervserver and working@workinghardinithardinit recently did a webinar together about Windows Server 2019 SMB Direct Best Practices. We also discuss some trends and put some things into perspective. Rachfahl IT Solutions does more of these cool webinars for you to check out (see the info at the end of the video). You can watch the webinar below on Vimeo. It’s quite an honor to be invited to talk on this subject as Carsten is one the worlds most experienced S2D practitioners.

https://vimeo.com/319818020
Windows Server 2019 SMB Direct Best Pratices Webinar

Need to know more?

I hope this get’s you started and updated. Need to know more? Want more details, advice and a deeper and more elaborate discussion. I will be talking about this on various occasions this year. One of them is the Cloud & Datacenter Conference Germany 2019 in Hanau. Register to secure your spot. It is a great conference with a lot of hands on, real life knowledge being shard. I will be around for the Hyper-V pre day and during the entire conference. This means you can find me to talk shop. Be warned, I can go one about the subject or a while

Running the Ubiquiti UniFi Controller as a service

Posted on by
7

Introduction

I recently had to prepare replacing an aging Aruba Wi-FI deployment with an effective, more capable and budget friendly solution. It needed to offer both corporate (Radius Server) and guest Wi-Fi access for modern workplace needs.

We selected Ubiquiti equipment to comply with the requirements. Apart from the WAPs all gear goes into server & network racks. The controller had to be implemented on-premises (self-managed, not via a service provider). As they have a modern Hyper-V environment we opted to deploy the controller on a Windows 2019 virtual machine. By the time the solution is deployed that will have become generally available. A Cloud Key appliance or Raspberry PI was less interesting in this environment that had professional racks in available in dedicated server & network rooms.

OK, you can use Windows Server 2016 or Windows Server 2012 R2 as well. Note that I don’t like using a client OS for an infrastructure role. I would also not use older server versions because I like longevity in support. I dislike solutions that are out of support a week after I deployed it. The big take away here is that you want to tweak the standard deployment of the controller a bit.

  1. Change the install so it is not tied to a user profile
  2. Run the controller as a service rather than an app you need to start manually or add to auto start.
  3. Configure a certificate for a decent user experience with the UniFi dashboard

Below are my lab notes as reference to myself and my readers in regards to running the Ubiquiti UniFi Controller as a service on Windows Server 2019.

Installation

For some reason the installer dumps all the files in the user profile of the person running the installer. Which is easy in terms of permissions. But people leave and profiles get deleted. Multiple people need to manage systems so having it tied to an individual isn’t that great.

For a UniFi install is first install java (x64) and a x64 bit browser. Chrome & Firefox are support, others may be as well or just work. The controller runs on Java so that’s a no brainer you need it. You don’t need a browser on the virtual machine per se, but it acts as a console access to the controller via the VM in case of network issues. Having multiple options is good.  If you don’t need that, Windows Core will do.

Step by Step

1. Install the controller with the UniFi-installer.exe installer. It will put the installation under C:\Users\USERNAME\Ubiquiti UniFi
2. To move the UniFi controller app you copy the entire folder to the desired location. That could be C:\Program Files or C:\ProgramData. You can even create your own root folder if you don’t want any admin permission to be needed for the folder. For this demo I used C:\ProgramData\Ubiquiti UniFi.
3. I create a shortcut https://unifi.workinghardinit.work:8443 and change the Icon to one I created for this purpose.

clip_image001

4. I then also change the “Target” path to “C:\ProgramData\Ubiquiti UniFi\lib\ace.jar” ui and “Start in” path to “C:\ProgramData\Ubiquiti UniFi”path. That way that short cut points to the right location. However, I want my controller to run as a service so we won’t be using that shortcut too much.

clip_image003

Anyway, we have a clean nice setup right now to continue with. Please note you do not need to install a browser on the server itself. This was done to give people a virtual machine console access option in case they have network issue. If don’t want that you can use Windows Server Core

Running as a service

Since we want the controller to always run and behave like a service, we just have some extra work to do. This is documented here: https://help.ubnt.com/hc/en-us/articles/205144550-UniFi-Run-the-Controller-as-a-Windows-service I just adapted this to my path.

1. Close any instances of the UniFi software on the computer. If you just installed the UniFi controller, make sure to open it once by using the icon on the desktop or within the start menu. Once it says “UniFi Controller (a.b.c) started.” you can close the controller program. This is needed to generate some required files for the service to work.
2. Open the command prompt as an Administrator. For example, on Windows 10, right click on the Start Menu and choose “Command Prompt (Admin)”.
3. Change directory to the location of UniFi in your computer using the following command (exactly as it is here, no substituting needed): cd “C:\ProgramData\Ubiquiti UniFi\”
4. Once in the root of the UniFi folder, issue the following (this installs the UniFi Controller service): java -jar lib\ace.jar installsvc
5. Once you’re at a new command prompt line, after it says “Complete Installation…”, issue the following: java -jar lib\ace.jar startsvc

Installing a proper certificate

After entering the FQDN A record or CNAME to your DNS infra you will still get a security warning as we haven’t installed a proper certificate yet.

clip_image005

Let’s fix this unprofessional looking fist view of your controller web application! We’ll use a recent cert from either a corporate or public PKI. Take your pick, there are free ones out there if you need that.

I’m using a wild card certificate and will show you how to implement it with the Unifi controller. The trick is to replace default Keystore with a custom one in which you added your certificate. There is are nice tools for that and the exact method will vary a bit. This is what I did. Note that you can do this on your workstation, no need o do all this on your server with the UniFi Controller. Keep that tidy.

Make sure you have your cert available (exported) as a pfx file.

The Windows application method

Download KeyStore Explorer (http://keystore-explorer.org/downloads.html) and install in on your PC, the default settings are just fine.

Have your certificate exported as pfx file with private key and the option “Include all certificates in the certification path if possible”.

Run KeyStore Explorer and under tools select “Import Key Pair”

clip_image006

As type select PKCS #12

clip_image007

Browse to your pfx cert you created, fill out the correct password and click “Import”

clip_image008

I’m happy with my default alias of * as I have a wild card cert. You should use unifi.domain.ext if you don’t have a wild card to be clear.

clip_image009

Enter the new Key Pair password, again I use “aircontrolenterprise”

clip_image010

Click OK and your see that your import was successful. Click OK.

clip_image012

Now select your keypair and under the File menu select “Save As”

clip_image013

For the password, again, use aircontrolenterprise, click OK and fill out keystore for the file name.

clip_image015

Click save, your done here.

I actually delete the imported key pair form KeyStore Explorer and also shift delete the export pfx. It’s better not to have these sorts of files lingering around on your workstation even when using bitlocker. You must have a cert management process.

The results of your work

Now on your controller VM navigate to your data path, in my case it’s C:\ProgramData\Ubiquiti UniFi\data. Rename the original keystore file to keystor.ori and past the one you created in this folder.

clip_image019

You then need to restart the UniFi Controller service, either in the GUI or via the command prompt.

clip_image020

clip_image021

Give the controller 10 second to get going properly and click your UniFi Dashboard shortcut to browse to the application. And now, as you can see, below we have a much better user experience. This is actually the logon screen after you’ve run through the initial install wizard when you first launch the application.

clip_image023

We now have a well-behaved web application to securely access the UniFi controller and manage the Wi-Fi setup.jjj

The native java tools method

If you want you can use native Java tools to do the same as with the KeyStore Explorer app replace those steps above by the one below.

C:\Program Files\Java\jre1.8.0_181\bin>Keytool.exe -list -keystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx -storetype pkcs12  which prompts for your password and outputs:

Enter keystore password:
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 1 entry

1524853e062d1785ac5ebedb44a61065, Aug 30, 2018, PrivateKeyEntry,
Certificate fingerprint (SHA1): 7A:82:FC:6E:2D:4D:79:F2:43:7A:FE:57:48:BE:13:FB:C4:AF:ED:71

C:\Program Files\Java\jre1.8.0_181\bin>keytool -importkeystore -srcstoretype pkcs12 -srcalias 1524853e062d1785ac5ebedb44a61065 -srckeystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx -keystore C:\SysAdmin\Certs\keystore -destalias *.workinghardinit.work

Importing keystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx to C:\SysAdmin\Certs\keystore…

Enter destination keystore password: aircontrolenterprise
Re-enter new password: aircontrolenterprise
Enter source keystore password: aircontrolenterprise => the password use to protect the pfx exported, can be anything

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore C:\SysAdmin\Certs\keystore -destkeystore C:\SysAdmin\Certs\keystore -deststoretype pkcs12”.

Conclusion

Ubiquiti delivers value for money Wi-fi solutions. The gear is good and affordable with manageability options that serve the majority of needs for the SME. It perfect for the more demanding SOHO environment.

Ubiquiti offers flexibility but also requires some “tweaking” to get just right. This goes for both the software installation (fixing some default installation choices and installing a certificate) as well as some of the hardware (installing less loud fans) shortcomings.

For many people a virtual machine with Windows is something they already have the infrastructure for. It fist perfectly into their existing operational processes. A virtual machine also fits well into many customers their existing backup and restore scenarios. A virtual machine can also easily be “checkpointed” to revert to a known good situation. This is an extra benefit in case something goes wrong during an upgrade or update wrong. This combined with the Auto Backup Configuration of the UniFi controller cover most bases for quick recovery. Not too many people can restore their raspberry PI or appliance that fast.

We chose to use Windows Server 2019 in this demo as we wanted to future proof the deployment . So we want to deliver the controller on an OS that will serve them well for many years to come.

To recap, first I showed you how to improve on the default installation. We than made the UniFi controller runs as a service. Finally I configured an SSL certficate for the controller app. I hope you liked it and that it helps you out.

SFP+ and SFP28 compatibility

Posted on by
4

Introduction

As 25Gbps (SFP28) is on route to displace 10Gbps (SFP+) from its leading role as the work horse in the datacenter. That means that 10Gbps is slowly but surely becoming “the LOM option”. So it will be passing on to the role and place 1Gbps has held for many years. What extension slots are concerned we see 25Gbps cards rise tremendously in popularity. The same is happening on the switches where 25-100Gbps ports are readily available. As this transition takes place and we start working on acquiring 25Gbps or faster gear the question about SFP+ and SFP28 compatibility arises for anyone who’s involved in planning this.

SPF+ and SFP28 compatibility

Who needs 25Gbps?

When I got really deep into 10Gbps about 7 years ago I was considered a bit crazy and accused of over delivering. That was until they saw the speed of a live migration. From Windows Server 2012 and later versions that was driven home even more with shared nothing and storage live migration and SMB 3 Multichannel SMB Direct.

On top of that storage spaces and SOFS came onto the storage scene in the Microsoft Windows server ecosystem. This lead us to S2D and storage replica in Windows Server 2016 and later. This meant that the need for more bandwidth, higher throughput and low latency was ever more obvious and clear. Microsoft has a rather extensive collection of features & capabilities that leverage SMB 3 and as such can leverage RDMA.

In this time frame we also saw the strong rise of All Flash Array solutions with SSD and NVMe. Today we even see storage class memory come into the picture. All this means even bigger needs for high throughput at low latency, so the trend for ever faster Ethernet is not over yet.

What does this mean?

That means that 10Gbps is slowly but surely becoming the LOM option and is passing on to the role 1Gbps has held for many years. In our extension slots we see 25-100Gbps cards rise in popularity. The same is happening on the switches where we see 25, 50, 100Gbps or even higher. I’m not sure if 50Gbps is ever going to be as popular but 25Gbps is for sure. In any case I am not crazy but I do know how to avoid tech debt and get as much long term use out of hardware as possible.

When it comes to the optic components SFP+ is commonly used for 10Gbps. This provides a path to 40Gbps and 100Gbps via QSFP. For 25Gbps we have SFP28 (1 channel or lane for 25Gbps). This give us a path to 50Gbps (2225Gbps – two lanes) and to 100Gbps (4*25Gbps – 4 lanes) via QSFP28. In the end this a lot more economical. But let’s look at SFP+ and SFP28 compatibility now.

SFP+ and SFP28 compatibility

When it comes to SFP+ and SFP28 compatibility we’re golden. SFP+ and SFP28 share the same form factor & are “compatible”. The moment I learned that SFP28 share the same form factor with SFP+ I was hopeful that they would only differ in speed. And indeed, that hope became a sigh of relief when I read and experimentally demonstrated to myself the following things I had read:

  1. I can plug in a SFP28 module into an SFP+ port
  2. I can plug in a SFP+ module into an SFP28 port
  3. Connectivity is established at the lowest common denominator, which is 10Gbps
  4. The connectivity is functional but you don’t gain the benefits SFP28 bring to the table.

Compatibility for migrations & future proofing

For a migration path that is phased over time this is great news as you don’t need to have everything in place right away from day one. I can order 25Gbps NIC in my servers now, knowing that they will work with my existing 10Gbps network. They’ll be ready to roll when I get my switches replaced 6 months or a year later. Older servers with 10Gbps SFP+ that are still in production when the new network gear arrives can keep working on new SFP28 network gear.

  • SFP+: 10Gbps
  • SFP28: 25Gbps but it can go up to 28 so the name is SFP28, not 25. Note that SFP28 can handle 25Gbps, 10Gbps and even 1Gbps.
  • QSFP28: 100Gbps to 4*25Gbps or 2*50Gbps gives you flexibility and port density.
  • 25Gbps / SFP28 is the new workhorse to deliver more bandwidth, better error control, less cross talk and an economical sound upgrade path.

Do note that SFP+ modules will work in SFP28 ports and vice versa but you have to be a bit careful:

  • Fix the ports speed when you’re not running at the default speed
  • On SFP28 modules you might need to disable options such as forward error correction.
  • Make sure a 10Gbps switch is OK with a 25Gbps cables, it might not.

If you have all your gear from a vendor specializing in RDMA technology like Mellanox this detects this all this and takes care of everything for you. Between vendors and 3rd party cables pay extra attention to verifying all will be well.

SFP+ and SFP28 compatibility is also important for future proofing upgrade paths. When you buy and introduce new network gear it is nice to know what will work with what you already have and what will work with what you might or will have in the future. Some people will get all new network switches in at once while others might have to wait for a while before new servers with SFP28 arrive. Older servers might be around and will not force you to keep older switches around just for them.

SFP28 / QSFP28 provides flexibility

Compatibility is also important for purchase decision as you don’t need to match 25Gbps NIC ports to 25Gbps switch ports. You can use the QSFP28 cables and split them to 4 * 25Gbps SFP28.

SPF+ and SFP28 compatibility

QSFP28

The same goes for 50Gbps, which is 100Gbps QSFP to 2 * 50Gbps QSFP.

SPF+ and SFP28 compatibility

SPF+ and SFP28 compatibility

This means you can have switch port density and future proofing if you so desire. Some vendors offer modular switches where you can mix port types (Dell EMC Networking S6100-ON)

Conclusion

More bandwidth at less cost is a no brainer. It also makes your bean counters happy as this is achieved with less switches and cables. That also translates to less space in a datacenter, less consumption of power and less cooling. And the less material you have the less it cost in operational expenses (management and maintenance). This is only offset partially by our ever-growing need for more bandwidth. As converged networking matures and becomes better that also helps with the cost. Even where economies of scale don’t matter that much. The transition to 25Gbps and higher is facilitated by SFP+ and SFP28 compatibility and that is good news for all involved.