Express Flash NVMe PCIe SSD support in VMware ESXi

This blog post is originally written by Karan Singh Gandhi and Krishnaprasad K from Dell Hypervisor Engineering team

What's Express Flash NVMe PCIe SSD?

NVMe (Non-Volatile Memory Host Controller Interface) is a specification for accessing Solid State Devices (SSDs) connected through PCI Express (PCIe) bus. The Dell PowerEdge Express Flash PCIe-SSD is based on NVMe specification and is a high-performance storage device built with enterprise-class NAND flash designed for solutions requiring low latency, high input/output operations per second (IOPS) and enterprise-class storage reliability and serviceability. The Express Flash PCIe-SSD is a PCIe Gen2/Gen3-compliant device (depending on server type) that can be configured as storage cache or as a primary storage device in demanding enterprise environments, such as enterprise blade and rack servers, video-on-demand servers, web accelerators and virtualization appliances.

Express Flash NVMe PCIe SSD support from Dell with respect to VMware ESXi

• Express Flash NVMe PCIe SSD is supported from Dell PowerEdge 12^th generation of servers onwards. The list of servers supporting Express Flash NVMe PCIe SSD are as follows:-
  • PE R620
  • PE R720
  • PE T620
  • PE R820
  • PE M620
  • PE R920
  • PE R630
  • PE R730xd
  • PE T630
  • PE M630
  • PE FC630
  • PE FC830
  • PE M820
  • PE M830

• The Express Flash NVMe PCIe SSD is supported from VMware ESXi 5.5 Update2 onwards. However the driver availability in Dell customized VMware ESXi (OR) VMware native ESXi image starts from ESXi 6.0 release onwards only.
• For enabling Express Flash NVMe PCIe SSD on VMware ESXi 5.5.x branch, ensure to install the async driver available from VMware download page.
• Refer to Express Flash NVMe PCIe SSD user’s guide which detail about the technical specifications and few aspects such as configuring and managing NVMe PCIe SSD

NVMe use cases in VMware ESXi

• NVMe PCIe SSD as a VMFS datastore
  • NVMe PCIe SSD can be formatted as a Virtual Machine File System (VMFS) datastore and the virtual machines can be placed on top of this for a better throughput
• Passthrough of NVMe device(s) to the virtual machines
  • It’s possible to passthrough the NVMe device directly to the virtual machines running on ESXi. In this case, the guest operating system should have the NVMe driver installed and loaded to the kernel to make use of the device.
  • This use case would be helpful if there is a specific read/write intensive application to be run from the guest operating system.
• NVMe PCIe SSD as Virtual Flash
  • There are two parts in virtual flash aka vFlash, NVMe PCIe SSD device can be used in any mode or both together
    • SSD as Virtual Flash Host Swap Cache: Used by hypervisor for Host Memory-Caching purposes. In this case, the hypervisor shares the SSD device amongst all the virtual machines
    • SSD as Virtual Flash Read Cache: Used by VMs for caching the virtual machines I/O requests. With this option, there is a dedicated space provided for the virtual machine’s vmdk.
• NVMe PCIe SSD hot plug
  • NVMe PCIe SSD supports hot plug. It means that the device can be removed or added on the fly even when the operating system is up and running.
  • Note that hot removing the NVMe PCIe SSD when the device is in use is NOT supported. The device has to be safely removed first prior to do physical hot removal of the device
• RDM creation from SSD device
  • Refer to VMware KB 1017530 to know more about creating RDM for the local drive which can include Express Flash NVMe PCIe SSDs as well

References

• Dell PowerEdge Express Flash Landing Page - www.dell.com/poweredge/expressflash
• Marketing datasheet for NVMe devices - http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_PowerEdge_Express_Flash_NVMe_PCIe_SSD_Spec_Sheet.pdf
• Customer User Guide - http://www.dell.com/support/home/us/en/04/product-support/product/dell-poweredge-exp-fsh-nvme-pcie-ssd/
• NVMe Home Page - http://www.nvmexpress.org/
• NVMe Specification - http://www.nvmexpress.org/wp-content/uploads/NVM_Express_1_2_Gold_20141103.pdf
• NVMe SCSI Translation Specification - http://www.nvmexpress.org/wp-content/uploads/NVM_Express_-_SCSI_Translation_Reference-1_3_20140923_Gold.pdf