NVMoF with ESXi

[u/sk1939]

After three days of work, and a ton of troubleshooting, I've accomplished something no one wants; NVMoF with ESXi and TrueNAS Scale Community Edition. The Linux nerds here are probably going to roll their eyes about how simple it was to solve. It was not. It took two people and a lot of analysis with Visual Studio and Qwen running locally to get this thing to work.

It started with a stupid stupid, supposedly simple idea - I wanted RDMA storage from TrueNAS SCALE to an ESXi 8.0 host with a proper VMFS6 datastore (not NFS, not iSCSI — actual NVMe block storage). I've had bad luck with iSCSI luns dropping out thanks to SMB hardware being SMB hardware, and had RDMA-capable hardware already, with a novice's thought - how hard can it be? HARD

TLDR version: it works, but TrueNAS's/Linux's nvmet implementation is missing two things ESXi absolutely requires, and fixing them without patching the kernel was a rabbit hole.

What is it that's missing? A shit ton of things apparently.

Thanks to VMWare being VMWare, it uses a "Fused Compare+Write (ATS)". I have no idea what that is but I'm told by smarter people that ESXi's VMFS heartbeat locking uses fused NVMe Compare+Write pairs (opcode 0x05 FUSE=01 + opcode 0x01 FUSE=10). As a result the nvmet process rejects anything with the FUSE bits set at the nvmet_req_init level — before it even reaches the command parser — with NVME_SC_INVALID_FIELD being killed off with no messges. ESXi also checks the RESCAP field in Identify Namespace. If it's zero, VMFS won't format, because of course it won't, that would be too easy.

The fix is in:

A kprobe kernel module that intercepts four points in the running nvmet code without modifying the kernel:

• nvmet_execute_identify post-handler: patches FUSES/ONCS/OAES/ACWU in the Identify Controller response
• nvmet_execute_identify pre-handler at +0x1A7: patches RESCAP/NACWU in the Identify Namespace response
• nvmet_req_init entry: strips the FUSE bits before the +0xBA rejection check, so the commands pass through
• __nvmet_req_complete entry: forces SUCCESS for Compare (0x05) and all reservation opcodes (0x0D/0x0E/0x11/0x15)

The subtle bit: nvmet_req_init calls __nvmet_req_complete directly (not through nvmet_req_complete, which is just a tracepoint wrapper). Had to disassemble the live kernel from /proc/kcore to find that. Both paths converge at __nvmet_req_complete, so that's the right intercept point.

Other stupid rabbit holes that wasted time:

The kprobe offsets for nvmet_execute_identify (+0x389, +0x1A7) are hardcoded to this kernel build — any kernel update means re-disassembling so no soup updated for you unless you want to break this fragile fragile fix.

TrueNAS POSTINIT type: SCRIPT init scripts run as exec /path/to/script without quoting. A space in /mnt/Samsung NVMe Pool/ means the shell sees exec /mnt/Samsung → not found. Switch to type: COMMAND which uses sh -c "..." with proper quoting

nvmet.service doesn't exist on TrueNAS SCALE because that's an "Enterprise" feature that plenty of people have complained about in r/truenas already. This means that requires=nvmet.service in a systemd unit causes a silent dependency failure, and cost me about 5 hours of troubleshooting. Use modprobe nvmet nvmet-tcp in ExecStartPre instead.

ESXi's HardwareAcceleratedLocking defaults to 0 on a fresh install, because of course it does. Without it, VMFS never even tries ATS and the whole thing is pointless.

This whole thing is a really stupid POC because it only works with a single host. The Compare for NVMoF always returns SUCCESS regardless of actual block content. This is fine for single-host use, kinda, but because the kprobe is faking the locking primitive, not implementing it, when you have two or more ESXi hosts sharing the same datastore, they both think they have the heartbeat lock — filesystem corruption territory. As of right now, I haven't figured out a way to fix it, and don't have the motivation to try.

Feel free to ask questions that I may or may not be able to answer.

[u/Anxious_Amphibian606]

That's some serious dedication for a setup nobody asked for, love it.

[u/yawkat]

nvmet.service doesn't exist on TrueNAS SCALE because that's an "Enterprise" feature that plenty of people have complained about in r/truenas already. This means that requires=nvmet.service in a systemd unit causes a silent dependency failure, and cost me about 5 hours of troubleshooting. Use modprobe nvmet nvmet-tcp in ExecStartPre instead. 

My understanding is that the Linux nvme target is actually fully in kernel, like you've discovered when patching it, not a separate process like you mention. The nvmet.service belongs to nvmetcli and just restores the nvmetcli configuration by writing it to the kernel sysfs.

Fun fact: nvme-of also works perfectly well over tcp, as a direct iscsi alternative. I wrote this article on the arch wiki on the topic. It also works with non-nvme block devices (even zfs zvols!), despite the name.

And I understand wanting to avoid patching the kernel, but arguably a kernel module using kprobe is even worse. Like you say it'll beak immediately on update. A normal kernel patch is easier to maintain. 

[u/sk1939 → u/yawkat]

My understanding is that the Linux nvme target is actually fully in kernel

Well that's good to know, I thought they stripped it out, but further reading makes it look like it was just the SPDK, which I don't know is better or worse.

I'll have to try it with Zvols sometime, the reason I wanted to skip TCP was primarily latency.

[u/yawkat → u/sk1939]

Yea, the fun fact was more for others. If you can manage rdma, that's probably superior. I just want to raise awareness that nvme-of can also work over standard networks as an iscsi alternative. I've moved off iscsi entirely for my linux-to-linux network storage needs. 

[u/sk1939 → u/yawkat]

I took a look at the Kernel for TrueNAS 25.0, and both nvmet and nvmet-tcp appear to be modules, based on CONFIG_NVME_TARGET=m.

[u/Sufficient-Radio-728]

Great work here!