Configuring NVMe over Fabrics (NVMeoF) TCP with Proxmox and TrueNAS SCALE

Step-by-Step Configuration

Note the NQN for the namespace you created. You’ll need it to configure Proxmox.

Enable NVMeoF Service:

Go to System → Services in the TrueNAS UI.

Enable the NVMe-oF service.

Create a Zvol:

Navigate to Storage → Pools.

Create a new Zvol that will serve as the backend storage for the NVMe subsystem.

Add NVMe-oF Subsystem:

Go to Sharing → NVMe-oF Subsystems → Add.

Provide a name for the subsystem.

Under Namespace, add the Zvol you created.

In Access Settings, ensure “Allow any host to connect” is unticked. Add your Proxmox host’s NQN for restricted access.

Save the configuration.

Copy NQN:

SSH into your Proxmox server and install necessary tools:

Install Required Packages:

apt-get update
apt-get install -y nvme-cli iproute2 net-tools multipath-tools

Connect to the NVMe-oF Subsystem:

Create a configuration file on your Proxmox server. For example at /etc/nvme/nvme.env, add:

NVME_NAS_IP=<TrueNAS_IP>
NVME_NAS_PORT=4420
NVME_NAS_NQN=<Your_NQN>

Discover the NVMe Target:

Use the nvme command to discover the NVMe target:

nvme discover -t tcp -a $NVME_NAS_IP -p $NVME_NAS_PORT

Connect to the NVMe Target:

Connect to the NVMe target:

nvme connect -t tcp -n $NVME_NAS_NQN -a $NVME_NAS_IP -p $NVME_NAS_PORT

Verify Connection:

Check if the storage is connected successfully:

nvme list

Make Connections persistent:

Add connection to discovery.conf:

echo discover -t tcp -a $NVME_NAS_IP -p $NVME_NAS_PORT | tee -a /etc/nvme/discovery.conf

Reconnect to storage after reboot:

systemctl enable nvmf-autoconnect.service

Create an LVM volume group on our NVMe-oF storage device:

Use the device path from running the nvme list command:

vgcreate nvme_vg /dev/nvme0n1