Configuring NVMe-TCP Persistent Connections for Proxmox

Making NVMe-TCP connections persistent ensures that storage is automatically available after system reboots without manual intervention. This page covers the complete configuration including network interfaces, IP addressing, and automatic connection setup.

1. Configure network interfaces on all nodes: Network interface configuration must be persistent and applied before NVMe connections are established.
   Important:

   Why this step is critical:

   • NVMe connections require IP connectivity before they can be established.
   • Static IPs ensure consistent addressing across reboots.
   • MTU must be configured before connections to avoid fragmentation.
   • Interfaces must be up before nvmf-autoconnect service runs.

   Option A: Dedicated Physical Interfaces

   When to use: You have dedicated NICs for storage traffic (recommended for production).

   Configuration for dedicated storage NICs on same subnet:

   # /etc/network/interfaces
   # Storage interface 1
   auto ens1f0
   iface ens1f0 inet static
       address 10.100.1.101/24  # Unique IP for this NIC
       mtu 9000                  # Jumbo frames for performance
   
   # Storage interface 2
   auto ens1f1
   iface ens1f1 inet static
       address 10.100.1.102/24  # Different IP, same subnet
       mtu 9000                  # Must match across all devices

   Note:

   Why these settings:

   • auto ens1f0: Brings interface up automatically on boot (critical for autoconnect).
   • inet static: Static IP required; DHCP would cause unpredictable addressing.
   • address 10.100.1.101/24: Each NIC gets unique IP on same subnet for multipath.
   • mtu 9000: Jumbo frames reduce CPU overhead and improve throughput.

   Apply the configuration:

   # Apply network changes
   ifreload -a
   
   # Verify interfaces are up
   ip addr show ens1f0
   ip addr show ens1f1
   
   # Test connectivity to storage portals
   ping -c 3 10.100.1.10
   ping -c 3 10.100.1.11
   ping -c 3 10.100.1.12
   ping -c 3 10.100.1.13

   Option B: VLAN Interfaces (Shared NICs)

   When to Use: You want to share physical NICs between storage, management, and VM traffic using VLANs.

   Configuration for storage VLANs on trunked interfaces:

   # /etc/network/interfaces
   # Physical NICs configured for trunking
   auto ens1f0
   iface ens1f0 inet manual    # No IP on physical interface
       bond-master bond0        # Member of bond for management/VM traffic
       mtu 9000
   
   auto ens1f1
   iface ens1f1 inet manual    # No IP on physical interface
       bond-master bond0        # Member of bond for management/VM traffic
       mtu 9000
   
   # Bond for management and VM traffic
   auto bond0
   iface bond0 inet manual     # No IP on bond itself
       bond-slaves ens1f0 ens1f1
       bond-mode active-backup  # Simple failover mode
       bond-miimon 100          # Check link every 100ms
       mtu 9000
   
   # Management VLAN
   auto bond0.10
   iface bond0.10 inet static
       address 192.168.10.101/24
       gateway 192.168.10.1     # Default gateway for management
       vlan-raw-device bond0
   
   # VM/LXC Bridge
   auto vmbr0
   iface vmbr0 inet manual
       bridge-ports bond0.20    # VLAN 20 for VM traffic
       bridge-stp off           # STP not needed in this design
       bridge-fd 0              # No forwarding delay
       bridge-vlan-aware yes
       bridge-vids 20-50        # Allow VLANs 20-50 for VMs
   
   # Storage VLAN 100 - NVMe-TCP (NO BOND - separate paths)
   auto ens1f0.100
   iface ens1f0.100 inet static
       address 10.100.1.101/24
       vlan-raw-device ens1f0   # VLAN on physical NIC, NOT bond
       mtu 9000
   
   auto ens1f1.100
   iface ens1f1.100 inet static
       address 10.100.1.102/24
       vlan-raw-device ens1f1   # VLAN on physical NIC, NOT bond
       mtu 9000

   Note:

   Why these settings:

   • Physical NICs inet manual: No IP on physical interface; VLANs will have IPs
   • bond-master bond0: Physical NICs are members of bond for management/VM traffic
   • bond-mode active-backup: Simple failover; doesn't require switch configuration
   • bond-miimon 100: Monitors link status every 100ms for fast failover detection
   • Storage VLANs on physical NICs, NOT bond: Critical - preserves separate paths for multipathing
   • vlan-raw-device ens1f0: Creates VLAN interface on physical NIC, bypassing bond

   ifupdown2 dependency

   Proxmox uses ifupdown2 for network configuration. The configuration above is for ifupdown2. If your distribution uses a different network configuration tool, the configuration will be different. The principles are the same, but the syntax will be different. Recent changes to the ifupdown code have resulted in an error when configuring the interfraces of a bridge with other configurations. Proxmox does not include the version if ifupdown2 that has this behavior. (https://lore.proxmox.com/pve-devel/20250930140948.265119-1-s.hanreich@proxmox.com/)

   Apply configuration:

   # Apply network changes
   ifreload -a
   
   # Verify storage VLAN interfaces
   ip addr show ens1f0.100
   ip addr show ens1f1.100
   
   # Test connectivity
   ping -I ens1f0.100 -c 3 10.100.1.10
   ping -I ens1f1.100 -c 3 10.100.1.10

   Configure Address Resolution Protocol (ARP) behavior (critical for same-subnet multipath):

   # Configure ARP to prevent responses on wrong interface
   cat > /etc/sysctl.d/99-nvme-tcp-arp.conf << 'EOF'
   # ARP configuration for NVMe-TCP multipath
   # Prevents ARP responses on wrong interface when multiple NICs share same subnet
   
   # For dedicated physical interfaces
   net.ipv4.conf.ens1f0.arp_ignore = 2
   net.ipv4.conf.ens1f1.arp_ignore = 2
   net.ipv4.conf.ens1f0.arp_announce = 2
   net.ipv4.conf.ens1f1.arp_announce = 2
   
   # For VLAN interfaces (uncomment if using VLANs instead of physical)
   #net.ipv4.conf.ens1f0/100.arp_ignore = 2
   #net.ipv4.conf.ens1f1/100.arp_ignore = 2
   #net.ipv4.conf.ens1f0/100.arp_announce = 2
   #net.ipv4.conf.ens1f1/100.arp_announce = 2
   
   # Global settings
   net.ipv4.conf.all.arp_ignore = 2
   net.ipv4.conf.default.arp_ignore = 2
   net.ipv4.conf.all.arp_announce = 2
   net.ipv4.conf.default.arp_announce = 2
   EOF
   
   # Apply settings immediately
   sysctl -p /etc/sysctl.d/99-nvme-tcp-arp.conf
   
   # Verify
   sysctl net.ipv4.conf.ens1f0.arp_ignore
   sysctl net.ipv4.conf.ens1f1.arp_ignore

   Important:

   Why ARP configuration is critical:

   • Multiple interfaces on same subnet can cause ARP confusion.
   • Without arp_ignore=2, interfaces may respond to ARP requests for other interfaces' IPs.
   • This breaks multipath by causing asymmetric routing.
   • See the Performance Optimization section for a more detailed explanation.
2. Load NVMe Kernel Modules on all nodes: Ensure NVMe modules load automatically on boot.
   Note:

   Why this step is critical:

   • Modules must be loaded before NVMe connections can be established.
   • Multipath must be enabled at module load time (cannot be changed later without reboot).
   • nvmf-autoconnect service depends on these modules being loaded.

   # Load modules immediately
   modprobe nvme           # Core NVMe driver
   modprobe nvme-tcp       # NVMe over TCP transport
   modprobe nvme-core      # NVMe core functionality
   
   # Make modules persistent across reboots
   cat > /etc/modules-load.d/nvme-tcp.conf << 'EOF'
   nvme
   nvme-tcp
   nvme-core
   EOF
   # Why: systemd-modules-load.service reads this file on boot
   
   # Enable multipath
   echo 'options nvme_core multipath=Y' > /etc/modprobe.d/nvme-tcp.conf
   # Why: Enables native NVMe multipathing; must be set before module loads
   
   # Verify modules are loaded
   lsmod | grep nvme
   # Expected output: nvme, nvme_tcp, nvme_core
   
   # Verify multipath is enabled
   cat /sys/module/nvme_core/parameters/multipath
   # Expected output: Y

3. Generate and register host NQN on all nodes:
   Important:

   Why this step is critical:

   • Host NQN uniquely identifies this Proxmox node to FlashArray.
   • FlashArray uses host NQN for access control (like iSCSI initiator IQN).
   • Must be persistent across reboots (stored in /etc/nvme/hostnqn).
   • Each node must have a unique host NQN.

   # Create directory
   mkdir -p /etc/nvme
   
   # Generate host NQN if it doesn't exist
   if [ ! -f /etc/nvme/hostnqn ]; then
       nvme gen-hostnqn > /etc/nvme/hostnqn
   fi
   # Why: Generates UUID-based NQN that is unique to this host
   
   # Display host NQN
   cat /etc/nvme/hostnqn
   # Example output: nqn.2014-08.org.nvmexpress:uuid:12345678-1234-1234-1234-123456789abc

   Important:

   Register this host NQN with FlashArray:

   • Why: FlashArray will reject connections from unregistered host NQNs (security).
   • How: Add this NQN to FlashArray host/initiator configuration.
   • Note: Each Proxmox node will have a different host NQN.
4. Configure persistent connections on all nodes: There are two methods for configuring persistent NVMe-TCP connections. Choose the method that best fits your environment.

   Method 1: Discovery Controller Configuration (Recommended)

   Note:

   Why this method is recommended:

   • Uses native nvme-cli discovery service (built-in to Proxmox)
   • Automatically discovers all subsystems from storage array
   • Simpler configuration (no need to specify subsystem NQN)
   • Easier to maintain when adding new subsystems

   Create discovery configuration:

   # Create config directory
   mkdir -p /etc/nvme
   
   # Create discovery.conf with all paths
   # Each line defines one path: host interface -> storage portal
   cat > /etc/nvme/discovery.conf << 'EOF'
   # NVMe-TCP Discovery Configuration
   # Format: --transport=tcp --traddr=<portal> --trsvcid=<port> --host-iface=<iface> --host-traddr=<host-ip>
   
   # Interface 1 (ens1f0 or ens1f0.100) -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101
   
   # Interface 2 (ens1f1 or ens1f1.100) -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102
   EOF

   Note:

   Why these parameters:

   • --transport=tcp: Specifies NVMe over TCP (vs RDMA or FC)
   • --traddr=10.100.1.10: Target address (storage portal IP)
   • --trsvcid=4420: Target service ID (port); 4420 is NVMe-TCP default
   • --host-iface=ens1f0: Which local interface to use for this connection (critical for multipath)
   • --host-traddr=10.100.1.101: Source IP address (must match IP configured on host-iface)

   For VLAN interfaces, use the VLAN interface names:

   cat > /etc/nvme/discovery.conf << 'EOF'
   # NVMe-TCP Discovery Configuration for VLAN interfaces
   
   # Interface 1 (ens1f0.100) -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f0.100 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f0.100 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f0.100 --host-traddr=10.100.1.101
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f0.100 --host-traddr=10.100.1.101
   
   # Interface 2 (ens1f1.100) -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f1.100 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f1.100 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f1.100 --host-traddr=10.100.1.102
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f1.100 --host-traddr=10.100.1.102
   EOF

   Enable and start the autoconnect service:

   # Enable automatic connection on boot
   systemctl enable nvmf-autoconnect.service
   # Why: Ensures nvmf-autoconnect.service runs on every boot to establish connections
   
   # Connect all paths now
   nvme connect-all
   # Why: Establishes connections immediately without waiting for reboot
   
   # Verify connections
   nvme list-subsys
   # Expected: Should show 8 paths (2 NICs × 4 portals) all in 'live' state

   Method 2: Per-Subsystem Configuration

   For more control, create individual configuration files with explicit subsystem NQN and timeout parameters:

   Note:

   When to use this method:

   • You need explicit control over timeout parameters per subsystem.
   • You want to connect to specific subsystems only (not all discovered).
   • You have multiple subsystems with different timeout requirements.
   • You need to document exact subsystem NQN in configuration.

   # Create config directory
   mkdir -p /etc/nvme/config.d
   
   # Create configuration file for your subsystem
   cat > /etc/nvme/config.d/storage-subsystem.conf << 'EOF'
   # Subsystem: storage-subsystem
   # 8 paths: 2 interfaces x 4 portals
   
   # Interface 1 -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f0 --host-traddr=10.100.1.101 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   
   # Interface 2 -> All Portals
   --transport=tcp --traddr=10.100.1.10 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.11 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.12 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   --transport=tcp --traddr=10.100.1.13 --trsvcid=4420 --host-iface=ens1f1 --host-traddr=10.100.1.102 --nqn=nqn.2024-01.com.example:storage --ctrl-loss-tmo=1800 --reconnect-delay=10
   EOF
   
   # Replace nqn.2024-01.com.example:storage with your actual subsystem NQN
   
   # Enable automatic connection
   systemctl enable nvmf-autoconnect.service
   
   # Connect all
   nvme connect-all

5. Configure IO Policy on all nodes: Set the IO policy to queue-depth for optimal performance.
   Important:

   Why this step is critical:

   • Default IO policy (numa) may not distribute load optimally.
   • Queue-depth policy provides best performance by routing to least busy path.
   • Must be configured persistently or it reverts to default on reboot.
   • Udev rule ensures policy is applied automatically when subsystem is added.

   # Create udev rule for automatic IO policy configuration
   cat > /etc/udev/rules.d/99-nvme-iopolicy.rules << 'EOF'
   # Set IO policy to queue-depth for all NVMe subsystems
   ACTION=="add|chnage", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="queue-depth"
   EOF
   # Why: Udev automatically applies this rule when NVMe subsystem is detected
   
   # Reload udev rules
   udevadm control --reload-rules
   udevadm trigger
   # Why: Makes udev re-read rules and apply to existing devices
   
   # Apply to existing subsystems
   for subsys in /sys/class/nvme-subsystem/nvme-subsys*; do
       echo "queue-depth" > "$subsys/iopolicy" 2>/dev/null || true
   done
   
   # Verify
   cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

6. Verify the persistent configuration:

   # Check if autoconnect service is enabled
   systemctl is-enabled nvmf-autoconnect.service
   
   # View current connections
   nvme list-subsys
   
   # Expected output: 8 paths (2 NICs × 4 portals)
   # nvme-subsys0 - NQN=nqn.2024-01.com.example:storage
   #  +- nvme0 tcp traddr=10.100.1.10 ... live
   #  +- nvme1 tcp traddr=10.100.1.11 ... live
   #  +- nvme2 tcp traddr=10.100.1.12 ... live
   #  +- nvme3 tcp traddr=10.100.1.13 ... live
   #  +- nvme4 tcp traddr=10.100.1.10 ... live
   #  +- nvme5 tcp traddr=10.100.1.11 ... live
   #  +- nvme6 tcp traddr=10.100.1.12 ... live
   #  +- nvme7 tcp traddr=10.100.1.13 ... live
   
   # Check IO policy
   cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy
   # Expected: queue-depth
   
   # List NVMe devices
   nvme list
   
   # Check block devices
   lsblk | grep nvme

7. Test configuration persistence with a reboot:

   # Reboot the node
   reboot
   
   # After reboot, verify everything reconnected automatically
   nvme list-subsys
   nvme list
   cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy