High Availability and Redundancy Best Practices for Using Proxmox with FlashArray over NVMe-TCP

Proxmox
Audience
Public
Source Type
Documentation

This page provides the high availability and redundancy best practices for using Proxmox with FlashArray as storage over NVMe-TCP.

Multipath Architecture

NVMe native multipathing provides automatic failover and load balancing across multiple paths. The benefits to using NVMe native are:

  • Built into the kernel: No additional software (like device-mapper multipath) required
  • Lower latency: Direct path to storage without additional layers
  • Automatic failover: Kernel handles path failures transparently
  • Load balancing: Distributes I/O across all available paths
  • Simpler configuration: Enabled with single kernel parameter

Path Redundancy Model:

Key Concepts:

Concept Benefit
2 NICs × 4 Portals = 8 Total Paths per node Provides redundancy at NIC level (2 NICs) and controller level (4 portals across 2 controllers); any single NIC or controller can fail without impact.
Each NIC connects to all storage portals Maximizes redundancy; if one NIC fails, remaining NIC still has 4 paths to storage. This ensures full connectivity.
NVMe native multipathing manages all paths as a single device Simplifies management; applications see one /dev/nvmeXnY device. The kernel handles the path selection automatically.
Automatic failover if any path fails No manual intervention is required; I/O continues on remaining paths, transparent to applications.
Load balancing across all active paths Utilizes full aggregate bandwidth, prevents single path bottleneck and improves overall performance.

IO Policy Configuration

The IO policy determines how I/O is distributed across multiple paths:

Policy Behavior Use Case When to use / not use
queue-depth Routes to path with lowest queue depth Recommended for best performance Best choice: Adapts to real-time conditions; automatically avoids congested paths; maximizes throughput; handles mixed workloads well
numa (default) Routes I/O to paths on same NUMA node CPU-local optimization

  • Good for: NUMA-aware workloads; reduces cross-NUMA traffic

  • Avoid if: Not all NUMA nodes have equal path access; can create imbalanced load
round-robin Distributes I/O evenly across all paths Even distribution

  • Good for: Simple, predictable distribution.

  • Avoid if: Paths have different performance characteristics; doesn't adapt to congestion

Why Queue Depth is Recommended:

  • Adaptive: Responds to actual path load, not static rules
  • Performance: Automatically uses fastest available path at any moment
  • Congestion avoidance: Routes around busy paths without manual intervention
  • Mixed workloads: Handles varying I/O patterns effectively

Recommended Configuration (Set via udev rule; persistent): 

cat > /etc/udev/rules.d/99-nvme-iopolicy.rules << 'EOF'
ACTION=="add|change", SUBSYSTEM=="nvme-subsystem", ATTR{iopolicy}="queue-depth"
EOF

Queue-Depth Load Balancing:

The queue-depth IO policy provides intelligent load balancing by monitoring the queue depth of each path and routing I/O requests to the path with the lowest current queue depth. This ensures optimal performance by:

  • Avoiding congested paths: Automatically routes around busy paths
  • Dynamic load distribution: Adapts in real-time to changing workload patterns
  • Maximizing throughput: Utilizes all available paths efficiently
  • Minimizing latency: Sends requests to the least busy path

How Queue-Depth Balancing Works:

In the diagram above, the NVMe multipath driver:

  1. Receives an I/O request from the application
  2. Checks the current queue depth of all 8 paths
  3. Identifies Path 4 has the lowest queue depth (1 request)
  4. Routes the new I/O request to Path 4
  5. Continuously monitors and adapts to changing conditions

Benefits Over Other Policies:

Scenario Round-Robin Queue-Depth
One path experiencing network congestion Continues sending I/O to slow path Automatically avoids congested path
Uneven workload distribution Forces equal distribution regardless of load Adapts to actual load on each path
Path recovery after failure Immediately sends full load to recovered path Gradually increases load as path proves stable
Mixed I/O sizes (small + large) Can overload paths with large I/O Balances based on actual queue depth

Verification (Check current IO policy): 

cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

Failover Behavior

Redundancy Checklist

  • Network Redundancy:

    • Minimum 2 NICs per Proxmox node: One NIC can fail without storage outage; provides load distribution.
    • Minimum 2 network switches: Eliminates switch as single point of failure; allows switch maintenance.
    • No single point of failure in network path: Any single component failure should not cause storage outage.

  • Storage Redundancy:

    • Minimum 2 storage controllers: Controller failure doesn't impact storage availability; enables active-active configuration.
    • Minimum 2 portals per controller (4 total): Provides multiple paths per controller; increases aggregate bandwidth.
    • RAID or erasure coding on storage array: Protects against drive failures; ensures data durability.

  • Path Redundancy:

    • Each NIC connects to all portals: Maximizes redundancy; if one NIC fails, other NIC maintains connectivity to all controllers.
    • Minimum 4 paths per node (2 NICs × 2 portals): Minimum for basic redundancy; allows one NIC and one controller to fail.
    • Recommended 8 paths per node (2 NICs × 4 portals): Optimal redundancy and performance; distributes load across more paths; higher aggregate bandwidth.

  • Configuration Redundancy:

    • Persistent connection configuration: Ensures storage is available after reboot without manual intervention.
    • Automatic reconnection enabled: Recovers from transient network issues automatically; reduces downtime.
    • IO policy configured via udev: Ensures optimal IO policy is applied automatically on every boot.

Connection Timeout Settings

Recommended Timeout Values: 

--ctrl-loss-tmo=1800    # 30 minutes before declaring controller lost
--reconnect-delay=10     # 10 seconds between reconnection attempts

Rationale:

  • ctrl-loss-tmo=1800 (30 minutes):

    • Why 30 minutes: Allows time for planned storage maintenance (firmware updates, controller reboots) without triggering I/O errors.
    • What happens: If a path is down for less than 30 minutes, I/O is queued and retried; after 30 minutes, path is declared dead and I/O fails.
    • Alternative values: Use -1 for infinite timeout (never give up), or lower value (e.g., 600 = 10 minutes) for faster failure detection.

  • reconnect-delay=10 (10 seconds):

    • Why 10 seconds: Balances quick recovery with avoiding connection storms that could overwhelm storage array.
    • What happens: After path failure, waits 10 seconds before attempting reconnection; prevents rapid reconnection attempts.
    • Alternative values: Lower (e.g., 5) for faster recovery; higher (e.g., 30) if storage array is sensitive to connection storms.