July 19th, 2021

TRON

NVMe on ESXi


I am running ESXi 6.7u3 on an old PowerEdge R610 and recently purchased a no-name NVMe m.2 drive and the StarTech.com M2 PCIe SSD Adapter from Amazon which I had read on several blogs worked flawlessly with 6.7. And that part appears to have been true. Finding the right group of online people who tinker with things as I do is rarely quick or easy in the Googlesphere, as most of the official replies dismiss unsupported hardware with unsupported drivers on unsupported versions of end-of-life hardware; where my dreams generally begin. I had already bugged one certified VMWare administrator at work on something *actually* supported I needed help in figuring out, but wasn't about to approach him with this sketchy shit.

Eventually, I did find my people, and here is the unglamorous solution: Downgrade ESXi's NVMe driver from 6.7u3 to 6.7.0 GA:

# esxcli software vib install -v ht*ps://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/nvme/VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib

# esxcli software vib install -v ht*ps://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/vmware-esx-esxcli-nvme-plugin/VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib

# reboot
I am now running SSD (NVMe) Host Swap Cache as mentioned in the Performance Best Practices for VMware vSphere 6.7 document I'd read several weeks back, as well as using the NVMe drive for all my virtual machine's paging file. There seems to be enough space on the drive as well that I could theoretically build a small, fast vm as well. Just like EMC's storage at work, I too will have "tiered" storage;
SSD --> SAN --> HDD --> NAS

Here's a couple of helpful commands for sussing out NVMe-related ESXi stuffs:

esxcli storage core adapter list
esxcli nvme device list
esxcli storage nmp device list | grep Display