Connecting Proxmox to FlashArray using NVMe-TCP

Proxmox
Audience
Public
Source Type
Documentation

This guide provides instructions on how to configure NVMe over TCP storage on Proxmox VE using native CLI tools and connect it to your FlashArray.

Note:

  • For instructions on how to connect Proxmox to the FlashArray using FC, follow this guide.

  • For instructions on how to connect Proxmox to the FlashArray using NFS, follow this guide.

  • For instructions on how to connect Proxmox to the FlashArray using iSCSI, follow this guide.

Pre-requisites:

This guide assumes you already have the following:

  • Proxmox VE 9.x or later
  • FlashArray configured with NVMe-TCP network
    • Portal IP address(es) and port
    • Subsystem NQN
  • Dedicated network interfaces for storage traffic (recommended)
  • Network connectivity between Proxmox nodes and FlashArray
  1. Configure network interfaces on all nodes:
    Note: For optimal performance and reliability, configure dedicated network interfaces for NVMe-TCP traffic.

    There are 2 options to configure network interfaces:

    • Option A: Dedicated Physical Interfaces (Recommended): Use separate physical NICs for storage traffic. No VLAN tagging, direct connection to storage network. Replace the interface names with your actual interface names. Replace the IPs with your actual host IP addresses.

      Using the GUI:

      1. Go to: Datacenter > <node> > System > Network, select the interface and click Edit.

      2. Add the IP address.

      3. Set the MTU to 9000 (or 1500 if jumbo frames are not configured on your network. This configuration is in the advanced section).

      4. Enable Autostart.

      5. Click OK.

        Note: Gateway configuration should not be necessary. It is recommended that the interfaces are on the same subnet as the storage endpoints. If this is not the case manual routing will be necessary.

      6. Repeat the above steps for each interface used for NVMe-TCP traffic.

      7. Click Apply Configuration.

      8. Repeat the above steps for each node.

      Using the CLI:

      For each Proxmox node, edit /etc/network/interfaces:

      # Storage interface 1 - dedicated physical NIC
auto <INTERFACE_NAME_1>
iface <INTERFACE_NAME_1> inet static
    address <PORTAL_IP_1>/<CIDR>
    mtu 9000

# Storage interface 2 - dedicated physical NIC
auto <INTERFACE_NAME_2>
iface <INTERFACE_NAME_2> inet static
    address <PORTAL_IP_2>/<CIDR>
    mtu 9000

    • Option B: VLAN Interfaces on Physical NICs: If you need to share physical NICs, use VLAN interfaces. This configuration can be applied directly to stand alone interfaces or to the physical interfaces in a bond. Replace the interface names with your actual interface names. Replace the IPs with your actual host IP addresses, and 100 with your actual VLAN ID(s):

      Using the GUI:

      1. Go to Datacenter > <node> > System > Network, select the interface and click Edit.

      2. If there is already an IP address assigned to the interface, leave it. This can be configured on interfaces that are part of a bond as well (these should already be active).
      3. Set the MTU to 9000 (or 1500 if jumbo frames are not configured on your network).
      4. Enable Autostart if it is not already enabled.

      5. Click OK.

      6. Repeat the above steps for each interface.

      7. Click on Create > Linux VLAN:

      8. Name the new VLAN interface in the format of <physical_interface>.<vlan_id> (ens1f0np0.100 in the example below). This should match the name of the physical interface that the VLAN is being created on.

        Note: The VLAN Tag field is not editable but shows the VLAN ID from the interface name.

      9. Add the IP address.

      10. Set the MTU to 9000 (or 1500 if jumbo frames are not supported).

      11. Enable Autostart.

      12. Click Create.

      13. Repeat for each physical interface and VLAN ID that needs to be configured.

      14. Click Apply Configuration.

      Using the CLI:

      SSH to the Proxmox node and run the following commands:

      1. Edit /etc/network/interfaces:

        # Physical interface (no IP, just up)
auto <INTERFACE_NAME_1>
iface <INTERFACE_NAME_1> inet manual
    mtu 9000

# Storage VLAN 100 on <INTERFACE_NAME_1>
auto <INTERFACE_NAME_1>.<VLAN_ID>
iface <INTERFACE_NAME_1>.<VLAN_ID> inet static
    address <PORTAL_IP_1>/<CIDR>
    vlan-raw-device <INTERFACE_NAME_1>
    mtu 9000

# Physical interface 2 (no IP, just up)
auto <INTERFACE_NAME_2>
iface <INTERFACE_NAME_2> inet manual
    mtu 9000

# Storage VLAN 101 on <INTERFACE_NAME_2>
auto <INTERFACE_NAME_2>.<VLAN_ID>
iface <INTERFACE_NAME_2>.<VLAN_ID> inet static
    address <PORTAL_IP_2>/<CIDR>
    vlan-raw-device <INTERFACE_NAME_2>
    mtu 9000

      2. Apply the configuration:

        # Apply changes
ifreload -a

# Verify interfaces are up
ip addr show <INTERFACE_NAME_1>.<VLAN_ID>
ip addr show <INTERFACE_NAME_2>.<VLAN_ID>

# Test connectivity to storage portals
ping -I <INTERFACE_NAME_1> <PORTAL_IP_1>
ping -I <INTERFACE_NAME_1> <PORTAL_IP_2>
ping -I <INTERFACE_NAME_2> <PORTAL_IP_1>
ping -I <INTERFACE_NAME_2> <PORTAL_IP_2>

      3. Repeat on each node.

    Note:

    Design Considerations:

    • Jumbo frames (MTU 9000): Recommended for storage traffic. Ensure switches and storage support jumbo frames end-to-end.
    • Same Subnet: This configuration assumes the same subnet. You can use multiple subnets if needed.
    • No bonding: Do not use a bond interface for storage. If the interfaces are part of a bond, the interface name or raw device setting will apply the configuration to the specific interface.
    • No Routing Of Storage Traffic: Ensure the interfaces are on the same subnet as the storage endpoints. If this is not the case manual routing will be necessary.
  2. Install dependencies on all Proxmox nodes by running the following commands on every node: 
    # Install nvme-cli
apt-get update
apt-get install -y nvme-cli

# Load kernel modules
modprobe nvme
modprobe nvme-tcp
modprobe nvme-core

# Make modules load on boot
cat > /etc/modules-load.d/nvme-tcp.conf << 'EOF'
nvme
nvme-tcp
nvme-core
EOF

# Make multipath persistent across reboots
echo 'options nvme_core multipath=Y' > /etc/modprobe.d/nvme-tcp.conf
  3. Generate host NQN on all nodes: 
    # Create directory
mkdir -p /etc/nvme

# Generate host NQN if it doesn't exist
if [ ! -f /etc/nvme/hostnqn ]; then
    nvme gen-hostnqn > /etc/nvme/hostnqn
fi

# Display host NQN - register this with your storage array
cat /etc/nvme/hostnqn
  4. Use the NQN generated to create host objects in the Pure Storage Purity//FA and create a volume:
    1. On each Proxmox host, run the command cat /etc/nvme/hostnqn and copy the NQN of the host so that you can map it to the host on the array:

      Example:

      [root@init20-13 ~]# cat /etc/nvme/hostnqn 
nqn.2014-08.org.nvmexpress:uuid:f3595813-be97-4277-b814-17faee1bcabc
[root@init20-13 ~]#
    2. Use this NQN to create host objects in the Pure Storage Purity//FA management interface:

      1. Go to Storage > Hosts.

      2. Click the + icon.

      3. In the pop-up window, give your host a name and under personality select none.

      4. Add the Proxmox host NQN under the Host Ports section.

    3. Create a Host Group for the Proxmox "cluster" and add the hosts to the host group.

      1. Go to Storage > Hosts.

      2. Click the + icon.

      3. Click Create Host Group. In the pop-up window, give your host group a name.

      4. Add the Proxmox hosts to the Host Group.

    4. Create a volume/LUN for your host:

      1. Go to Storage > Volumes.

      2. Click the + icon.

      3. In the pop-up window, give your volume a name, volume size and add your newly created host group to this volume/LUN.

  5. Optional: Discover targets:

    Replace the <PORTAL_IP_X> with your actual FlashArray NVMe-TCP portal addresses. Replace <INTERFACE_NAME> with your actual Proxmox interface name. Replace <INTERFACE_IP> with the Proxmox node IP address of the interface:

    # Discover available NVMe subsystems via specific interface
nvme discover -t tcp \
    -a <PORTAL_IP_1> \
    -s 8009 \
    --host-iface=<INTERFACE_NAME > \
    --host-traddr=<INTERFACE_IP>
  6. Create a udev rule on all nodes to automatically set the IO policy when NVMe subsystems are created:
    Note: The udev rule ensures the IO policy is set automatically on every boot and whenever new subsystems are connected.
    # Create udev rule for NVMe IO policy
cat > /etc/udev/rules.d/99-nvme-iopolicy.rules << 'EOF'
# Set IO policy to queue-depth for all NVMe subsystems
ACTION=="add", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="queue-depth"
EOF

# Reload udev rules
udevadm control --reload-rules
udevadm trigger

# Apply to existing subsystems (for current session)
for subsys in /sys/class/nvme-subsystem/nvme-subsys*; do
    echo "queue-depth" > "$subsys/iopolicy" 2>/dev/null || true
done
  7. Connect every Proxmox node to the NVMe-TCP Target:

    Each Proxmox interface connects to ALL FlashArray portals for maximum path redundancy. With 2 source interfaces and 4 target portals, you get 8 paths total. Replace <FLASHARRAY_IP_X> with your actual portal IP addresses, and <SUBSYSTEM_NQN> with your actual subsystem (FlashArray) NQN. Replace <INTERFACE_NAME_1> and <INTERFACE_NAME_2> with your actual interface names. Replace <INTERFACE_IP_1> and <INTERFACE_IP_2> with your actual host IP addresses. Replace 4420 with your actual port if different:

    Note:

    • The example below assumes you have 4 portals. If you have more or less, adjust accordingly.

    • Depending on your network configuration, the below commands may vary.

    # Interface 1 -> All Portals
nvme connect -t tcp -a <FLASHARRAY_IP_1> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_2> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_3> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_4> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

# Interface 2 -> All Portals
nvme connect -t tcp -a <FLASHARRAY_IP_1> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_2> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_3> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

nvme connect -t tcp -a <FLASHARRAY_IP_4> -s 4420 -n <SUBSYSTEM_NQN> \
    --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> \
    --ctrl-loss-tmo=1800 --reconnect-delay=10

    Verify the connection:

    # List subsystems and paths (should show 8 paths)
nvme list-subsys

# List NVMe devices
nvme list

# Check block devices
lsblk | grep nvme

# Verify
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
  8. Configure persistent connections on all nodes by using the native nvme-cli discovery configuration for automatic connections on boot. Each interface connects to all portals for maximum redundancy (2 interfaces × 4 portals = 8 paths):

    • Option A: Discovery Controller Configuration (Recommended):

      1. Create a discovery configuration file:

        # Create config directory
mkdir -p /etc/nvme

# Create discovery.conf - one entry per path (interface + portal combination)
# 2 interfaces x 4 portals = 8 entries
cat > /etc/nvme/discovery.conf << 'EOF'
# NVMe-TCP Discovery Configuration
# Each line defines one path: host interface -> storage portal
# Format: --transport=tcp --traddr=<portal> --trsvcid=<port> --host-iface=<iface> --host-traddr=<host-ip>

# Interface 1 -> All Portals
--transport=tcp --traddr=<PORTAL_IP_1> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1>
--transport=tcp --traddr=<PORTAL_IP_2> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1>
--transport=tcp --traddr=<PORTAL_IP_3> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1>
--transport=tcp --traddr=<PORTAL_IP_4> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1>

# Interface 2 -> All Portals
--transport=tcp --traddr=<PORTAL_IP_1> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2>
--transport=tcp --traddr=<PORTAL_IP_2> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2>
--transport=tcp --traddr=<PORTAL_IP_3> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2>
--transport=tcp --traddr=<PORTAL_IP_4> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2>
EOF

      2. Enable and start the nvme-connect service:

        # Enable automatic connection on boot
systemctl enable nvmf-autoconnect.service

# Connect all (discovers and connects to all subsystems via all paths.)
nvme connect-all

    • Option B: Per-Subsystem Configuration: For more control, create individual configuration files with explicit NQN and timeouts:

      # Create config directory for subsystems
mkdir -p /etc/nvme/config.d

# Create a config file for your subsystem
# 2 interfaces x 4 portals = 8 paths
cat > /etc/nvme/config.d/my-storage.conf << 'EOF'
# Subsystem: my-storage
# 8 paths: 2 interfaces x 4 portals

# Interface 1 -> All Portals
--transport=tcp --traddr=<PORTAL_IP_1> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_2> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_3> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_4> --trsvcid=4420 --host-iface=<INTERFACE_NAME_1> --host-traddr=<INTERFACE_IP_1> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10

# Interface 2 -> All Portals
--transport=tcp --traddr=<PORTAL_IP_1> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_2> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_3> --trsvcid=4420 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
--transport=tcp --traddr=<PORTAL_IP_4> --trsvcid=8009 --host-iface=<INTERFACE_NAME_2> --host-traddr=<INTERFACE_IP_2> --nqn=<SUBSYSTEM_NQN> --ctrl-loss-tmo=1800 --reconnect-delay=10
4420

# Enable automatic connection
systemctl enable nvmf-autoconnect.service

# Connect all
nvme connect-all

    Verify the persistent configuration:

    Warning: If you will test by rebooting your host, make sure that there are no active workloads running on the node.
    # Check if service is enabled
systemctl is-enabled nvmf-autoconnect.service

# View current connections
nvme list-subsys

# Test by rebooting
reboot

    After the reboot, verify that the connections have been restored:

    nvme list-subsys
nvme list
  9. Create an LVM volume group on one node as seen below. This guide assumes multi-node access and uses LVM thick as the only supported method for usage. See Storage - Proxmox VE for supported tolopolgies.

    Using the GUI:

    1. Go to Datacenter > <node> > Disks and make sure the NVMe device is visible. If it is not, click Reload:

    2. Go to LVM and create a new volume group with the new device:

    3. Select the Disk (NVMe device) and add a Name to the volume group. Ensure Add Storage is checked and click Create:

      The new volume group should now be visible in the storage list. The shared configuration is not yet enabled and the storage is not yet available to the other nodes.

    4. Go to: Datacenter > Storage, select the volume group and click Edit.

    5. Check the Shared box. Enable the volume on other nodes by either selecting them in the Nodes drop down or by clearing the Nodes field by clicking the x to the right of the field. Click OK:

    The storage should now be enabled on all nodes or the selected nodes.

    Tip:

    If a node has a ? overlay on the icon for the volume or the status says unknown on a node, SSH to the node and run the following command:

    pvscan --cache

    Repeat for all nodes that have this status. This should update the nodes cache of LVM volumes and the storage should now be available.

    Using the CLI:

    Run the following commands on one node only:

    # List NVMe namespaces
lsblk | grep nvme

# Create LVM physical volume (use your device)
pvcreate /dev/nvme0n1

# Create volume group
vgcreate nvme-vg /dev/nvme0n1

# Verify
vgs
pvs

    Add LVM Storage to Proxmox on one node only (configuration synchronizes via the cluster):

    # Add LVM storage
pvesm add lvm nvme-datastore \
    --vgname nvme-vg \
    --content images,rootdir \
    --shared 1

# Verify
pvesm status

    Activate the volume group by running these commands on each of the other nodes:

    pvscan --cache

    Verify the setup on all nodes:

    # Check NVMe connections
nvme list-subsys

# Check IO policy
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

# Check Proxmox storage
pvesm status

# Check LVM
vgs
lvs
Quick Reference:
Term Definition
Portal A specific network endpoint, comprising of an IP address and a TCP port number on an NVMe storage target (subsystem) where an NVMe host (initiator) connects to access storage namespaces. An example would be an interface on a controller, like the IP address eth15 (Ethernet port 15) on CT0 (Controller 0).
Command Description
nvme list List NVMe devices
nvme list-subsys List subsystems and paths
nvme discover -t tcp -a <ip> -s 8009 Discover targets
nvme connect -t tcp -a <ip> -n <nqn> Connect to target
nvme disconnect -n <nqn> Disconnect from target / subsystem
cat /sys/class/nvme-subsystem/*/iopolicy Check IO policy
nvme disconnect-all Disconnect from all targets