Fixing slow RoCE RDMA performance with WinOF-2 to WinOF

In this post, you join me on my quest of fixing slow RoCE RDMA performance with traffic initiated on WinOF-2 system to or from a WinOF system. I was working on the migration to new hardware for Hyper-V clusters. This is in one of the locations where I transition from 10/40Gbps to 25/50/100Gbps. The older nodes had ConnectX-3 Pro cards for SMB 3 traffic. The new ones had ConnectX-4 Lx cards. The existing network fabric has been configured with DCB for RoCE and has been working well for quite a while. Everything was being tested before the actual migrations. That’s when we came across an issue.

All the RDMA traffic workes very well except when initiated from the new servers. So ConnectX-3 Pro to ConnectX-3 Pro works fine. So does ConnectX-4 to ConnectX-4. When initiating traffic (send/retrieve) from a ConnectX-3 Pro server to a ConnectX-4 host it is also working fine.

However, when I initiated traffic (send/retrieve) from a ConnectX-4 server to a ConnectX-3 host the performance was abysmally slow. 2.5-3.5 MB/s is really bad. This need investigation. It smelled a bit like an MTU size issue. As if one side was configured wrong.

The suspects

As things are bad only when traffic was initiated from a ConnectX-4 host I suspected a misconfiguration. But that all checked out. We could reproduce it on every ConnectX-4 to any ConnectX-3.

The network fabric is configured to allow jumbo frames end to end. All the Mellanox NICs have an MTU size of 9014 and the RoCE frame size is set to Automatic. This has worked fine for many years and is a validated setup.

If you read up on how MTU sizes are handled by Mellanox you would expect this to work out well.

So what is going on here? I started researching a bit more and I found this in the release notes of the WinOF-2 2.20 driver.

Description: In RoCE, the maximum MTU of WinOF-2 (4k) is greater than the maximum MTU of WinOF (2k). As a result, when working with MTU greater than 2k, WinOF and WinOF-2 cannot operate together.

Well the ConnectX-3 Pro uses WinOF and the ConnectX-4 uses WinOF-2 so this is what they call a hint. But still, if you look at the mechanism for negotiating the RoCE frame size this should negotiate fine in our setup, right?

The Fix

Well, we tried fixing the RoCE frame size to 2048 on both ConnectX-3 Pro and ConnectX-4 NICs. We left the NIC MTU at 9014. That did not help at all. What did help in the end wat set the NIC MTU sizes to 1514 (default) and set the Max RoCE frame size to automatic again? With this setting, all scenarios in all direction do work as expected.

Personally, I think the negotiation of the RoCE frame size, when set to automatic should figure this out correctly with the NIC MTU size set to 9014 as well. But I am happy I found a workaround where we can leverage RDMA during the migration, After that is done we can set the NIC MTU sizes back to 9014 when there is no more need for ConnectX-4 / ConnectX3 RDMA traffic.

Conclusion

The above behavior was unexpected based on the documentation. I argue the RoCE frame size negotiation should work thing out correctly in each direction. Maybe a fix will appear in a new WinOF-2 driver. The good news is that after checking many permutations I found came up with a fix. Luckily, we can leave jumbo frames configured across the network fabric. Reverting the NIC interfaces MTU size to 1514 on both side and leaving the RoCE frame size on auto on both sides works fine. So that is all that’s needed for fixing slow RoCE RDMA performance with WinOF-2 to WinOF. This will do just fine during the co-existence period and we keep it in mind whenever we encounter this behavior again.

KB4512534 fixes GUI activation issues with Windows Server 2019 MAK keys

Just a quick post to share some good news. In the recently released KB4512534 Microsoft fixed a rather long outstanding issue with the MAK activation of Windows Server 2019. I verified that KB4512534 fixes GUI activation issues with Windows Server 2019 MAK keys. Until now trying to activate a Windows Server 2019 installation with a MAK key via the GUI always failed. Using slmgr.vbs /ipk works.

This has been an issue since RTM. The error code is 0X80070490 and if you search for it you’ll find many people with that issue. See Getting error 0x80070490 while trying to activate win server 2019 and Server 2019 product key woes. There was no fix other than to use slmgr.vbs.

But that issue has now been resolved.

For the use cases where we need a MAK and a GUI is prefered by the support people, this was a bit annoying. For that reason, I was quite happy to read the following in the notes of KB4512534

Addresses an issue that prevents server editions from activating with a Multiple Activation Key (MAK) in the graphical user interface (GUI). The error is, “0x80070490”.

I had to try it and yes I can confirm that it works! It took Microsoft a while to fix this. As we had a working alternative (slmgr.vbs) and the VLK activation had no issues this problem was not a show stopper.

KB4512534 is available via Windows Updates, WSUS and the Windows Catalog. Please find more information here. So one more annoyance fixed that many people can encounter when starting out with Windows Server 2019. That is a good thing

Introduction

I made a rough video on how to configure Persistent Memory for Hyper-V. The server in this demo is a DELL R740. The type of persistent memory (PMEM) is NVDIMM-N. There were 6 modules in the system. In collaboration with my fellow MVPs Carsten Rachfahl and Anton Kolomyeytsev I hope to showcase storage class memory (SCM) as well (Intel Optane DC Persistent Memory, and later also NVDIMM-P). I can only hope that NRAM gets to market for real in 2020 so we can get memory class storage (MCS) out there. That would change the storage and memory world even more. If it can live up to its promise.

The video

In the video on how to configure Persistent Memory for Hyper-V, I walk you through the NVDIMM configuration in the BIOS first. Then we configure the PMEM regions on the host to get PMEM Disks.

After initializing and formatting them we create the new .vhdpmem type of virtual disk on them. The virtual machine we want to present them to requires a new type of storage controller, a virtual machine PMEM controller, to be able to add the .vhdpmem disks. Once you have done that the PMEM disks will be visible inside the virtual machine.

Inside the virtual machine, we initialize those PMEM disks and format them with NTFS. We put some large test files on them and run a diskspd test to show you how it performs. That’s it.

Enjoy the video on Vimeo here or below.

Powershell

Below you will find the PowerShell I used in the video. Adapt where needed for your setup and configuration.

PowerShell on the host

<#
Didier Van Hoye
Microsoft MVP Cloud & Datacenter Management
Twitter: @WorkigHardInIT
blog: https://blog.workinghardinit.work
#>

#Take a look at the unused PMEM regions
Get-PmemUnusedRegion

#We don't have PMEM disks yet
Get-pmemdisk

#We create PMEM disks from the unuses regions
Get-PmemUnusedRegion | New-PmemDisk -AtomicityType None
#Greate PMEM disk out of the unused regions
#New-PmemDisk -RegionId 1 -AtomicityType None

#List the PMEM disks
Get-pmemdisk


#Grab the physical disk that have Storage Class Memory (SCM) media or bus type.
Get-PhysicalDisk | Where Mediatype -eq SCM | ft -AutoSize
Get-Disk | Where BusType -eq SCM | ft -AutoSize


#Initialize the pmem disks
Get-Disk | Where BusType -eq SCM | Initialize-Disk –PartitionStyle GPT

#Take a look at the initialized disks
Get-Disk | Where BusType -eq SCM

#Create partitions and format them
New-Partition -DiskNumber 6 -UseMaximumSize -driveletter P  | Format-Volume -FileSystem NTFS -NewFileSystemLabel PMEM01 -IsDAX $True
New-Partition -DiskNumber 7 -UseMaximumSize -driveletter Q  | Format-Volume -FileSystem NTFS -NewFileSystemLabel PMEM02 -IsDAX $True

Get-Partition | fl *
Get-Volume -driveletter P,Q | Get-Partition | ft Driveletter, IsDax

#Show that the volumes are DAX
fsutil fsinfo volumeinfo P: 

#On those PMEM disk with a DAX file system we create a VHDPMEM virtual disk
New-VHD 'P:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem' -Fixed -SizeBytes 31GB
New-VHD 'Q:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem' -Fixed -SizeBytes 31GB

Get-VMPmemController PMEMVM
(Get-VMPmemController PMEMVM).Drives

#We add a virtual PMEM controller to our VM
#make sure the VM is shut down.
Add-VMPmemController PMEMVM

#Check if it is there
Get-VMPmemController PMEMVM

#We add our VHDPMEM virtual disks to the VM
Add-VMHardDiskDrive -VMName PMEMVM -ControllerType PMEM -ControllerNumber 0 -ControllerLocation 1 -Path 'P:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem'
Add-VMHardDiskDrive -VMName PMEMVM -ControllerType PMEM -ControllerNumber 0 -ControllerLocation 2 -Path 'Q:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem'

#Check if the disks are there
Get-VMPmemController PMEMVM
(Get-VMPmemController PMEMVM).Drives

#That's it on the host, we now go in to the VM and initialize and format our PMEM disk there for use.

PowerShell in the virtual machine

Get-Disk | where bustype -eq SCM | ft -AutoSize

Get-Disk | Where BusType -eq SCM | Initialize-Disk –PartitionStyle GPT

New-Partition -DiskNumber 1 -UseMaximumSize -driveletter P  | Format-Volume -NewFileSystemLabel VDISKVMPMEM01 -FileSystem NTFS -AllocationUnitSize 65536
New-Partition -DiskNumber 2 -UseMaximumSize -driveletter Q  | Format-Volume -NewFileSystemLabel VDISKVMPMEM02 -FileSystem NTFS -AllocationUnitSize 65536