Introduction

I made a rough video on how to configure Persistent Memory for Hyper-V. The server in this demo is a DELL R740. The type of persistent memory (PMEM) is NVDIMM-N. There were 6 modules in the system. In collaboration with my fellow MVPs Carsten Rachfahl and Anton Kolomyeytsev I hope to showcase storage class memory (SCM) as well (Intel Optane DC Persistent Memory, and later also NVDIMM-P). I can only hope that NRAM gets to market for real in 2020 so we can get memory class storage (MCS) out there. That would change the storage and memory world even more. If it can live up to its promise.

The video

In the video on how to configure Persistent Memory for Hyper-V, I walk you through the NVDIMM configuration in the BIOS first. Then we configure the PMEM regions on the host to get PMEM Disks.

After initializing and formatting them we create the new .vhdpmem type of virtual disk on them. The virtual machine we want to present them to requires a new type of storage controller, a virtual machine PMEM controller, to be able to add the .vhdpmem disks. Once you have done that the PMEM disks will be visible inside the virtual machine.

Inside the virtual machine, we initialize those PMEM disks and format them with NTFS. We put some large test files on them and run a diskspd test to show you how it performs. That’s it.

Enjoy the video on Vimeo here or below.

Powershell

Below you will find the PowerShell I used in the video. Adapt where needed for your setup and configuration.

PowerShell on the host

<#
Didier Van Hoye
Microsoft MVP Cloud & Datacenter Management
Twitter: @WorkigHardInIT
blog: https://blog.workinghardinit.work
#>

#Take a look at the unused PMEM regions
Get-PmemUnusedRegion

#We don't have PMEM disks yet
Get-pmemdisk

#We create PMEM disks from the unuses regions
Get-PmemUnusedRegion | New-PmemDisk -AtomicityType None
#Greate PMEM disk out of the unused regions
#New-PmemDisk -RegionId 1 -AtomicityType None

#List the PMEM disks
Get-pmemdisk


#Grab the physical disk that have Storage Class Memory (SCM) media or bus type.
Get-PhysicalDisk | Where Mediatype -eq SCM | ft -AutoSize
Get-Disk | Where BusType -eq SCM | ft -AutoSize


#Initialize the pmem disks
Get-Disk | Where BusType -eq SCM | Initialize-Disk –PartitionStyle GPT

#Take a look at the initialized disks
Get-Disk | Where BusType -eq SCM

#Create partitions and format them
New-Partition -DiskNumber 6 -UseMaximumSize -driveletter P  | Format-Volume -FileSystem NTFS -NewFileSystemLabel PMEM01 -IsDAX $True
New-Partition -DiskNumber 7 -UseMaximumSize -driveletter Q  | Format-Volume -FileSystem NTFS -NewFileSystemLabel PMEM02 -IsDAX $True

Get-Partition | fl *
Get-Volume -driveletter P,Q | Get-Partition | ft Driveletter, IsDax

#Show that the volumes are DAX
fsutil fsinfo volumeinfo P: 

#On those PMEM disk with a DAX file system we create a VHDPMEM virtual disk
New-VHD 'P:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem' -Fixed -SizeBytes 31GB
New-VHD 'Q:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem' -Fixed -SizeBytes 31GB

Get-VMPmemController PMEMVM
(Get-VMPmemController PMEMVM).Drives

#We add a virtual PMEM controller to our VM
#make sure the VM is shut down.
Add-VMPmemController PMEMVM

#Check if it is there
Get-VMPmemController PMEMVM

#We add our VHDPMEM virtual disks to the VM
Add-VMHardDiskDrive -VMName PMEMVM -ControllerType PMEM -ControllerNumber 0 -ControllerLocation 1 -Path 'P:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem'
Add-VMHardDiskDrive -VMName PMEMVM -ControllerType PMEM -ControllerNumber 0 -ControllerLocation 2 -Path 'Q:\Virtual Disks\DEMOVM-PMDisk02.vhdpmem'

#Check if the disks are there
Get-VMPmemController PMEMVM
(Get-VMPmemController PMEMVM).Drives

#That's it on the host, we now go in to the VM and initialize and format our PMEM disk there for use.

PowerShell in the virtual machine

Get-Disk | where bustype -eq SCM | ft -AutoSize

Get-Disk | Where BusType -eq SCM | Initialize-Disk –PartitionStyle GPT

New-Partition -DiskNumber 1 -UseMaximumSize -driveletter P  | Format-Volume -NewFileSystemLabel VDISKVMPMEM01 -FileSystem NTFS -AllocationUnitSize 65536
New-Partition -DiskNumber 2 -UseMaximumSize -driveletter Q  | Format-Volume -NewFileSystemLabel VDISKVMPMEM02 -FileSystem NTFS -AllocationUnitSize 65536 

Performance Demo of PMEM in Hyper-V

After a couple of successful presentations on Persistent Memory (PMEM) I created a performance demo of PMEM in Hyper-V video to showcase the performance. I am not a great video talent so this is a bit rough but it does get the message across.

The performance of PMEM is nothing short of impressive. A video might help showcase this better than a screenshot. If 1.5 million IOPS in a VM doesn’t get you excited look at the latency of 0.04 ms or less with a workload of 8K IO size, 30% write, 30% random IO.

The direct link to the video is https://vimeo.com/352583893

How to configure PMEM for Hyper-V

A follow-up video is to come soon. In that one, I configure PMEM in the BIOS, on the host and present it to a virtual machine on Windows Server 2019. I hope the results of the IO tests in this Performance Demo of PMEM in Hyper-V in this video will get you interested in how to set this up.

A bright future

Persistent Memory has a bright future all across the industry in servers, storage arrays, IoT and mobile devices, laptops and PC’s especially if MRAM and NRAM ever take off. My interest in it reminds me of the time when I started looking beyond 10/40Gbps networking and RDMA. The progress there converges excellently with persistent memory and FPGAs to show the path to composable infrastructure and help us deal with ever more data and processing needs.

Introduction

I’ll give a quick intro to NVDIMM-F in this blog post. I am doing this for completeness sake as we look at some different PMEM implementations over a series of blog posts. We have already looked at NVDIMM-N and later we’ll look at the future of PMEM with NVDIMM-P and Storage Class Memory.

A quick intro to NVDIMM-F

NVDIMM-F is basically a PMEM module that contains only flash and resides on a DIMM slot (DDR-4 bus). There is is no DRAM on an NVDIMM-F module. The main advantage is that it comes in bigger capacities than NVDIMM-N. Flash can be bigger than RAM. The DDR bus speed is excellent and allows for low latency. The most obvious drawback is that it is a lot slower than NVDIMM-N. Sure they surely leverage SLC, but still, compared with DRAM this is significantly slower.

The memory is system mapped flash. So it is slower but is persistent as the reading/writing is done Flash. RAM is not system mapped but it can be used as a cache to speed up IO The RAM cache actually is a risk during power loss as this will not be persisted.

All this means that the speed and latency (10-100us) are flash-like, so not a great. The good news is the capacity can be that large as well so from 128GB to potentially many TB.

The limitations

• NVDIMM-F provides larger flash-like capacity (128GB – xTB) but with higher latencies (10s of ms).
• It is not byte addressable only block-level access to the memory-mapped flash. There is a proprietary driver between NVDIMM-F and the cache in main memory.
• Caching in main memory (DRAM) is required to hide latencies but prevents 100% persistence.
• The copying of data between cache in system RAM and NVDIMM-F NAND puts a burden on the channel bandwidth.

The benefits

• NVDIMM-F provides large capacity NVDIMM not limited to DRAM sizes
• it allows for immediate persistence of data in the NVDIMM-F modules without the overhead of moving it to NVMe or SSD at the cost of some temporary latency. Remember that NVDIMM-F is not trying to replace or act as memory in any way.

Use cases and futures

If you have a workload where avoiding PCIe/SAS latencies is important this could be an option. But with the excellent performance of NVMe over PCIe this has been reduced. What if you wanted 4TB of high-speed storage for a Very Large Memory database some years ago? NVDIMM-F was a proposed solution.

With the advent of NVDIMM-P & other Persistent Memory / Storage Class Memory offerings and the new media (3DXP, NRAM, …), I do not think NVDIMM-F is going places anymore. Those will offer capacity and speed. While NVDIMM-N will have some use cases where the absolute lowest latency is needed and the size limitation is not a show stopper, not so for NVDIMM-F. Storage Class Memory and NVDIMM-P will take over the reins.

Diabolo Technologies that made an NVDIMM-F back in the day does not even exist anymore. Maybe that could be a sign as well.