Running the Ubiquiti UniFi Controller as a service

Introduction

I recently had to prepare replacing an aging Aruba Wi-FI deployment with an effective, more capable and budget friendly solution. It needed to offer both corporate (Radius Server) and guest Wi-Fi access for modern workplace needs.

We selected Ubiquiti equipment to comply with the requirements. Apart from the WAPs all gear goes into server & network racks. The controller had to be implemented on-premises (self-managed, not via a service provider). As they have a modern Hyper-V environment we opted to deploy the controller on a Windows 2019 virtual machine. By the time the solution is deployed that will have become generally available. A Cloud Key appliance or Raspberry PI was less interesting in this environment that had professional racks in available in dedicated server & network rooms.

OK, you can use Windows Server 2016 or Windows Server 2012 R2 as well. Note that I don’t like using a client OS for an infrastructure role. I would also not use older server versions because I like longevity in support. I dislike solutions that are out of support a week after I deployed it. The big take away here is that you want to tweak the standard deployment of the controller a bit.

  1. Change the install so it is not tied to a user profile
  2. Run the controller as a service rather than an app you need to start manually or add to auto start.
  3. Configure a certificate for a decent user experience with the UniFi dashboard

Below are my lab notes as reference to myself and my readers in regards to running the Ubiquiti UniFi Controller as a service on Windows Server 2019.

Installation

For some reason the installer dumps all the files in the user profile of the person running the installer. Which is easy in terms of permissions. But people leave and profiles get deleted. Multiple people need to manage systems so having it tied to an individual isn’t that great.

For a UniFi install is first install java (x64) and a x64 bit browser. Chrome & Firefox are support, others may be as well or just work. The controller runs on Java so that’s a no brainer you need it. You don’t need a browser on the virtual machine per se, but it acts as a console access to the controller via the VM in case of network issues. Having multiple options is good.  If you don’t need that, Windows Core will do.

Step by Step

1. Install the controller with the UniFi-installer.exe installer. It will put the installation under C:\Users\USERNAME\Ubiquiti UniFi
2. To move the UniFi controller app you copy the entire folder to the desired location. That could be C:\Program Files or C:\ProgramData. You can even create your own root folder if you don’t want any admin permission to be needed for the folder. For this demo I used C:\ProgramData\Ubiquiti UniFi.
3. I create a shortcut https://unifi.workinghardinit.work:8443 and change the Icon to one I created for this purpose.

clip_image001

4. I then also change the “Target” path to “C:\ProgramData\Ubiquiti UniFi\lib\ace.jar” ui and “Start in” path to “C:\ProgramData\Ubiquiti UniFi”path. That way that short cut points to the right location. However, I want my controller to run as a service so we won’t be using that shortcut too much.

clip_image003

Anyway, we have a clean nice setup right now to continue with. Please note you do not need to install a browser on the server itself. This was done to give people a virtual machine console access option in case they have network issue. If don’t want that you can use Windows Server Core

Running as a service

Since we want the controller to always run and behave like a service, we just have some extra work to do. This is documented here: https://help.ubnt.com/hc/en-us/articles/205144550-UniFi-Run-the-Controller-as-a-Windows-service I just adapted this to my path.

1. Close any instances of the UniFi software on the computer. If you just installed the UniFi controller, make sure to open it once by using the icon on the desktop or within the start menu. Once it says “UniFi Controller (a.b.c) started.” you can close the controller program. This is needed to generate some required files for the service to work.
2. Open the command prompt as an Administrator. For example, on Windows 10, right click on the Start Menu and choose “Command Prompt (Admin)”.
3. Change directory to the location of UniFi in your computer using the following command (exactly as it is here, no substituting needed): cd “C:\ProgramData\Ubiquiti UniFi\”
4. Once in the root of the UniFi folder, issue the following (this installs the UniFi Controller service): java -jar lib\ace.jar installsvc
5. Once you’re at a new command prompt line, after it says “Complete Installation…”, issue the following: java -jar lib\ace.jar startsvc

Installing a proper certificate

After entering the FQDN A record or CNAME to your DNS infra you will still get a security warning as we haven’t installed a proper certificate yet.

clip_image005

Let’s fix this unprofessional looking fist view of your controller web application! We’ll use a recent cert from either a corporate or public PKI. Take your pick, there are free ones out there if you need that.

I’m using a wild card certificate and will show you how to implement it with the Unifi controller. The trick is to replace default Keystore with a custom one in which you added your certificate. There is are nice tools for that and the exact method will vary a bit. This is what I did. Note that you can do this on your workstation, no need o do all this on your server with the UniFi Controller. Keep that tidy.

Make sure you have your cert available (exported) as a pfx file.

The Windows application method

Download KeyStore Explorer (http://keystore-explorer.org/downloads.html) and install in on your PC, the default settings are just fine.

Have your certificate exported as pfx file with private key and the option “Include all certificates in the certification path if possible”.

Run KeyStore Explorer and under tools select “Import Key Pair”

clip_image006

As type select PKCS #12

clip_image007

Browse to your pfx cert you created, fill out the correct password and click “Import”

clip_image008

I’m happy with my default alias of * as I have a wild card cert. You should use unifi.domain.ext if you don’t have a wild card to be clear.

clip_image009

Enter the new Key Pair password, again I use “aircontrolenterprise”

clip_image010

Click OK and your see that your import was successful. Click OK.

clip_image012

Now select your keypair and under the File menu select “Save As”

clip_image013

For the password, again, use aircontrolenterprise, click OK and fill out keystore for the file name.

clip_image015

Click save, your done here.

I actually delete the imported key pair form KeyStore Explorer and also shift delete the export pfx. It’s better not to have these sorts of files lingering around on your workstation even when using bitlocker. You must have a cert management process.

The results of your work

Now on your controller VM navigate to your data path, in my case it’s C:\ProgramData\Ubiquiti UniFi\data. Rename the original keystore file to keystor.ori and past the one you created in this folder.

clip_image019

You then need to restart the UniFi Controller service, either in the GUI or via the command prompt.

clip_image020

clip_image021

Give the controller 10 second to get going properly and click your UniFi Dashboard shortcut to browse to the application. And now, as you can see, below we have a much better user experience. This is actually the logon screen after you’ve run through the initial install wizard when you first launch the application.

clip_image023

We now have a well-behaved web application to securely access the UniFi controller and manage the Wi-Fi setup.jjj

The native java tools method

If you want you can use native Java tools to do the same as with the KeyStore Explorer app replace those steps above by the one below.

C:\Program Files\Java\jre1.8.0_181\bin>Keytool.exe -list -keystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx -storetype pkcs12  which prompts for your password and outputs:

Enter keystore password:
Keystore type: PKCS12
Keystore provider: SunJSSE

Your keystore contains 1 entry

1524853e062d1785ac5ebedb44a61065, Aug 30, 2018, PrivateKeyEntry,
Certificate fingerprint (SHA1): 7A:82:FC:6E:2D:4D:79:F2:43:7A:FE:57:48:BE:13:FB:C4:AF:ED:71

C:\Program Files\Java\jre1.8.0_181\bin>keytool -importkeystore -srcstoretype pkcs12 -srcalias 1524853e062d1785ac5ebedb44a61065 -srckeystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx -keystore C:\SysAdmin\Certs\keystore -destalias *.workinghardinit.work

Importing keystore C:\SysAdmin\Certs\exported_wildcard_workinghardinit_work.pfx to C:\SysAdmin\Certs\keystore…

Enter destination keystore password: aircontrolenterprise
Re-enter new password: aircontrolenterprise
Enter source keystore password: aircontrolenterprise => the password use to protect the pfx exported, can be anything

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore C:\SysAdmin\Certs\keystore -destkeystore C:\SysAdmin\Certs\keystore -deststoretype pkcs12”.

Conclusion

Ubiquiti delivers value for money Wi-fi solutions. The gear is good and affordable with manageability options that serve the majority of needs for the SME. It perfect for the more demanding SOHO environment.

Ubiquiti offers flexibility but also requires some “tweaking” to get just right. This goes for both the software installation (fixing some default installation choices and installing a certificate) as well as some of the hardware (installing less loud fans) shortcomings.

For many people a virtual machine with Windows is something they already have the infrastructure for. It fist perfectly into their existing operational processes. A virtual machine also fits well into many customers their existing backup and restore scenarios. A virtual machine can also easily be “checkpointed” to revert to a known good situation. This is an extra benefit in case something goes wrong during an upgrade or update wrong. This combined with the Auto Backup Configuration of the UniFi controller cover most bases for quick recovery. Not too many people can restore their raspberry PI or appliance that fast.

We chose to use Windows Server 2019 in this demo as we wanted to future proof the deployment . So we want to deliver the controller on an OS that will serve them well for many years to come.

To recap, first I showed you how to improve on the default installation. We than made the UniFi controller runs as a service. Finally I configured an SSL certficate for the controller app. I hope you liked it and that it helps you out.

SFP+ and SFP28 compatibility

Introduction

As 25Gbps (SPF28) is on route to displace 10Gbps (SFP+) from its leading role as the work horse in the datacenter. That means that 10Gbps is slowly but surely becoming “the LOM option”. So it will be passing on to the role and place 1Gbps has held for many years. What extension slots are concerned we see 25Gbps cards rise tremendously in popularity. The same is happening on the switches where 25-100Gbps ports are readily available. As this transition takes place and we start working on acquiring 25Gbps or faster gear the question about SPF+ and SFP28 compatibility arises for anyone who’s involved in planning this.

SPF+ and SFP28 compatibility

Who needs 25Gbps?

When I got really deep into 10Gbps about 7 years ago I was considered a bit crazy and accused of over delivering. That was until they saw the speed of a live migration. From Windows Server 2012 and later versions that was driven home even more with shared nothing and storage live migration and SMB 3 Multichannel SMB Direct.

On top of that storage spaces and SOFS came onto the storage scene in the Microsoft Windows server ecosystem. This lead us to S2D and storage replica in Windows Server 2016 and later. This meant that the need for more bandwidth, higher throughput and low latency was ever more obvious and clear. Microsoft has a rather extensive collection of features & capabilities that leverage SMB 3 and as such can leverage RDMA.

In this time frame we also saw the strong rise of All Flash Array solutions with SSD and NVMe. Today we even see storage class memory come into the picture. All this means even bigger needs for high throughput at low latency, so the trend for ever faster Ethernet is not over yet.

What does this mean?

That means that 10Gbps is slowly but surely becoming the LOM option and is passing on to the role 1Gbps has held for many years. In our extension slots we see 25-100Gbps cards rise in popularity. The same is happening on the switches where we see 25, 50, 100Gbps or even higher. I’m not sure if 50Gbps is ever going to be as popular but 25Gbps is for sure. In any case I am not crazy but I do know how to avoid tech debt and get as much long term use out of hardware as possible.

When it comes to the optic components SFP+ is commonly used for 10Gbps. This provides a path to 40Gbps and 100Gbps via QSFP. For 25Gbps we have SFP28 (1 channel or lane for 25Gbps). This give us a path to 50Gbps (2*2*25Gbps – two lanes) and to 100Gbps (4*25Gbps – 4 lanes) via QSFP28. In the end this a lot more economical. But let’s look at SPF+ and SFP28 compatibility now.

SPF+ and SFP28 compatibility

When it comes to SPF+ and SFP28 compatibility we’re golden. SFP+ and SPF28 share the same form factor & are “compatible”. The moment I learned that SFP28 share the same form factor with SFP+ I was hopeful that they would only differ in speed. And indeed, that hope became a sigh of relief when I read and experimentally demonstrated to myself the following things I had read:

  1. I can plug in a SFP28 module into an SPF+ port
  2. I can plug in a SFP+ module into an SPF28 port
  3. Connectivity is established at the lowest common denominator, which is 10Gbps
  4. The connectivity is functional but you don’t gain the benefits SFP28 bring to the table.

Compatibility for migrations & future proofing

For a migration path that is phased over time this is great news as you don’t need to have everything in place right away from day one. I can order 25Gbps NIC in my servers now, knowing that they will work with my existing 10Gbps network. They’ll be ready to roll when I get my switches replaced 6 months or a year later. Older servers with 10Gbps SPF+ that are still in production when the new network gear arrives can keep working on new SPF28 network gear.

  • SPF+: 10Gbps
  • SFP28: 25Gbps but it can go up to 28 so the name is SFP28, not 25. Note that SFP28 can handle 25Gbps, 10Gbps and even 1Gbps.
  • QSPF28: 100Gbps to 4*25Gbps or 2*50Gbps gives you flexibility and port density.
  • 25Gbps / SPF28 is the new workhorse to deliver more bandwidth, better error control, less crosstalk and an economical sound upgrade path.

Do note that SPF+ modules will work in SPF28 ports and vice versa but you have to be a bit careful:

  • Fix the ports speed when you’re not running at the default speed
  • On SPF28 modules you might need to disable options such as forward error correction.
  • Make sure a 10Gbps switch is OK with a 25Gbps cables, it might not.

If you have all your gear from a vendor specializing in RDMA technology like Mellanox this detects this all this and takes care of everything for you. Between vendors and 3rd party cables pay extra attention to verifying all will be well.

SPF+ and SFP28 compatibility is also important for future proofing upgrade paths. When you buy and introduce new network gear it is nice to know what will work with what you already have and what will work with what you might or will have in the future. Some people will get all new network switches in at once while others might have to wait for a while before new servers with SPF28 arrive. Older servers might be around and will not force you to keep older switches around just for them.

SPF28 / QSFP28 provides flexibility

Compatibility is also important for purchase decision as you don’t need to match 25Gbps NIC ports to 25Gbps switch ports. You can use the QFSP28 cables and split them to 4 * 25Gbps SFP28.

SPF+ and SFP28 compatibility

QSFP28

The same goes for 50Gbps, which is 100Gbps QFSP to 2 * 50Gbps QFSP.

SPF+ and SFP28 compatibility

SPF+ and SFP28 compatibility

 

 

 

 

 

 

 

 

 

This means you can have switch port density and future proofing if you so desire. Some vendors offer modular switches where you can mix port types (Dell EMC Networking S6100-ON)

Conclusion

More bandwidth at less cost is a no brainer. It also makes your bean counters happy as this is achieved with less switches and cables. That also translates to less space in a datacenter, less consumption of power and less cooling. And the less material you have the less it cost in operational costs (management and maintenance). This is only offset partially by our ever-growing need for more bandwidth. As converged networking matures and becomes better that also helps with the cost. Even where economies of scale don’t matter that much. The transition to 25Gbps and higher is facilitated by SPF+ and SFP28 compatibility and that is good news for all involved.

Windows Server 2016 RDMA and the Hyper-V vSwitch – Part II

Introduction

In part I this article I demonstrated that some of the rules in regards to SMB Direct and the Hyper-V vSwitch as we know them for Windows Server 2012 R2 have changed with Windows Server 2016. We focused on the fact that you can expose RDMA to a vNIC exposed to the management OS created on a vSwitch. This means that while in Windows Server 2012 R2 you cannot expose RDMA capabilities via a vSwitch, even when you are using a non-teamed RDMA capable NIC, this is no longer true with Windows Server 2016.

While a demo with a vSwitch on a single NIC as we did in part I is nice it’s unlikely you’ll use this often if at all in the real world? Here we require redundancy and that means NIC teaming. To do so we normally use a vSwitch created on a native Windows NIC team. But a native NIC teaming does not expose RDMA capabilities. And as such a vSwitch created against a Windows native NIC team cannot leverage RDMA either. Which was the one of the reasons why a fully converged scenario in Windows Server 2012 R2 was too limited for many scenarios. Loss of RSS on the vNIC exposes to the management OS was another. The solution to this in Windows Server 2016 Hyper-V comes with Switch Embedded Teaming (SET). Now using SET in each and every situation might not be a good idea. It depends. But we do need to know how to configure it. So let’s dive in.

Switch Embedded Teaming (SET) exposes RDMA to the vSwitch

Switch Embedded Teaming (SET in Windows Server 2016 allows multiple identical (make, model, firmware, drivers to be supported) NICs to be used or “teamed” within the vSwitch itself. The important thing to note here this does not use windows NIC teaming or LBFO (Load Balancing and Fail Over).

SET is the future and is needed or use with the Network Controller and Software Defined Networking in Windows. SET can also be used without these technologies. While today it supports a good deal of the capabilities of native Windows NIC teaming it also lacks some of them. In general SET is meant for full or partial converged scenarios with 10GBps or better NICs, not 1Gbps networking in a (hyper)converged Hyper-V scenario.

Please see New Windows Server 2016 NIC and Switch Embedded Teaming User Guide for Download for more information as there is just too much to tell about it.

Setting it up

We start out with a 2-node cluster where each node has 2 RDMA NICs (Mellanox ConnectX-3) with RDMA enabled and DCB configured. Live migration of VMs between those nodes works over SMB Direct works. All NIC are on the same subnet 172.16.0.0/16 (thanks to Window Server 2016 Same Subnet Multichannel) and are on VLAN 110. In Failover Cluster Manager (FCM) that looks like below.

clip_image002

We’ll now use the rNICs to create a Switch Embedded Team.

clip_image004

Note that the teaming mode is switch independent, the only option supported with SET in Window Server 2016.

clip_image006

This also gives us a vNIC exposed to the management OS (default)

clip_image008

This is also visible as a vNIC in the mamagement OS called “vEthernet (RDMA-SET-vSwitch)”

clip_image010

This will be used to manage the host and to make its purpose clear we’ll rename it.

We’ll create 2 separate management OS vNICs for the RDMA traffic later. For now, we want the HOST-MGNT vNIC to have connectivity to the LAN and for that we need to tag it with VLAN 10.

image

The vNIC actually “inherited” the IP configuration of one of our physical NICs and we need to change that to either DHCP or a correct LAN IP address and settings.

clip_image014

You can use the code below to set the HOST-MGNT vNIC to DHCP

To finalize the HOST-MGNT vNIC configuration we enable priority tagging on it. If we don’t we won’t see any traffic other than SMB Direct tagged at all!

clip_image016

Before we go any further we’ll remove the VLAN tag from the rNICS as we don’t want it interfering with egress traffic being tagged by them or ingress traffic being filtered because it doesn’t match the VLAN ID on the rNICs.

From here on we’ll focus on the RDMA capable vNICs well create and use for SMB traffic.

We create 2 vNIC on the management OS for SMB Direct traffic.

clip_image018

Now these vNIC need an IP address, this can be in the same subnet because we have Windows Server 2016 SMB multichannel.

We than also need to put the vNICs in the correct VLAN. Remember that DCB / PFC priority tagging needs tagged VLAN so carry that priority. Right now, we can see that these are untagged.

clip_image020

So we tag them with VLAN ID 110

clip_image022

We enable jumbo frames on the vNICs. Remember that the physical NICs in the SET have jumbo frames enabled as well.

clip_image024

Normally all traffic that is originated from vNICs gets any QOS values set to zero. There is one exception to this and that’s SMB Direct traffic. SMB Direct traffic gets tagged with its QoS priority and that is not reset to 0 as it bypasses the vSwitch completely. But if we set other priorities on other types of traffic for DCB PFC and or ETS that passes over these vNICs we must enable priority tagging on these NICs as well or they’ll be stripped away.

clip_image026

The association of the vNIC to pNICs is random. This also changes during creation and destruction (disabling NICs, restarting the OS). We can map a vNCI to a particular pNIC. This prevents suboptimal use of the available pNICs and provides for a well know predictable path of the traffic. We do this with the below PowerShell commands.

Finally, last but not least, we should enable RDMA on our two vNICs or SMB Direct will not kick in at all.

Right now, we have it all configured correctly on one node of our 2-node cluster. The SMB network look like this now:

clip_image028

The cluster now looks like below.

clip_image030

We can live migrate VMs over SMB Direct in this mixed scenario where one node has pNICs RDMA NICs, 1 node has SET with vNICs for RMDA.

clip_image032

When looking at this in report mode we clearly see Node-A send SMB Direct traffic (tagged with priority 4, green) over its RDMA enabled SET vNICs to Node-B which still has a complete physical rNIC set up (blue).

clip_image034

As you can see in the screen shots above we now have RDMA / SMB Direct working with SET / RDMA vNICs on one node (Node-A) and pure physical RDMA NICs on the other (Node-B). This gives us bandwidth aggregation and redundancy. To complete the exercise, we configure SET on the other node as well. But it’s clear SET and RDAM will also work in a mixed environment.

We’ll discuss some details about certain aspects of the vNIC configuration in future articles. Things like the why and how of Set-VMNetworkAdapterTeamMapping and the use of -IeeePriorityTag. But for now, this is it. Go try it out! It’s the basis for anything you’ll do with SDNv2 in W2K16 and beyond.

You cannot connect multiple NICs to a single Hyper-V vSwitch without teaming on the host

Can you connect multiple NICs to a single Hyper-V vSwitch without teaming on the host

Recently I got a question on whether a Hyper-V virtual switch can be connected to multiple NICs without teaming. The answer is no. You cannot connect multiple NICs to a single Hyper-V vSwitch without teaming on the host.

This question makes sense as many people are interested in the ease of use and the great results of SMB Multichannel when it comes to aggregation and redundancy. But the answer lies in the name “SMB”. It’s only available for SMB traffic. Believe it or not but there is still a massive amount of network traffic that is not SMB and all that traffic has to pass through the Hyper-v vSwitch.

What can we do?

Which means that any redundant scenario that requires other traffic to be supported than SMB 3 will need to use a different solution than SMB Multichannel. Basically, this means using NIC teaming on a server. In the pre Windows Server 2012 era that meant 3rd party products. Since Windows Server 2012 it means native LBFO (switch independent, static or LACP). In Windows Server 2016 Switch Embedded Teaming (SET) was added to your choice op options. SET only supports switch independent teaming (for now?).

If redundancy on the vSwitch is not an option you can use multiple vSwitches connected to separate NIC and physical switches with Windows native LBFO inside the guests. That works but it’s a lot of extra work and overhead so you only do this when it makes sense. One such an example is SR-IOV which isn’t exposed on top of  a LBFO team.