Performance Optimization Best Practices for Using Proxmox with FlashArray over NVMe-TCP

• Standard MTU (1500) requires more packets for same data.
• Jumbo frames (9000) reduce packet count by ~6x.
• Fewer packets mean fewer interrupts and lower CPU usage.
• Jumbo frames can improve throughput by 20-30% for large sequential I/O.

# Set MTU 9000 on all storage interfaces
ip link set dev ens1f0 mtu 9000
ip link set dev ens1f1 mtu 9000

# Verify end-to-end (tests that full 9000-byte packets can traverse path)
ping -M do -s 8972 -c 3 10.100.1.10  # 8972 + 28 bytes header = 9000
# Why -M do: Don't fragment flag; will fail if any device has MTU < 9000
# Why 8972: ICMP payload size; adds 28 bytes (20 IP + 8 ICMP) = 9000 total

• Modern servers have multiple NUMA nodes (CPU + local memory).
• Cross-NUMA memory access is slower (~2x latency).
• NICs are attached to specific NUMA nodes.
• VMs running on same NUMA node as storage NICs have lower latency.

# Check NUMA topology
numactl --hardware
# Shows: Number of NUMA nodes, CPUs per node, memory per node

# Check NIC NUMA node
cat /sys/class/net/ens1f0/device/numa_node
# Example output: 0 (NIC is on NUMA node 0)

# Pin storage-intensive VMs to same NUMA node as storage NICs
# Why: Keeps I/O path local to NUMA node; avoids cross-NUMA traffic
# How: In Proxmox VM config, set CPU affinity to CPUs on same NUMA node

• Each I/O completion generates an interrupt.
• Without balancing, all interrupts go to CPU 0 (bottleneck).
• irqbalance distributes interrupts across all CPUs.
• Prevents single CPU from becoming saturated.

# Install irqbalance
apt-get install irqbalance

# Enable and start
systemctl enable irqbalance
systemctl start irqbalance
# Why: Automatically balances IRQs across CPUs based on load

# Verify IRQ distribution
cat /proc/interrupts | grep nvme
# Should see interrupts distributed across multiple CPUs, not just CPU 0

• Queue depth limits how many I/O requests can be outstanding.
• Higher queue depth means more parallelism and better throughput.
• Having the queue depth too low may under-utilize the storage array.
• Having the queue depth too high may increase latency for individual requests.

# Check current queue depth
cat /sys/block/nvme0n1/queue/nr_requests
# Default: Usually 128 or 256

# Increase if needed (for high-throughput workloads)
echo 1024 > /sys/block/nvme0n1/queue/nr_requests
# Why 1024: Allows more concurrent I/O; useful for large sequential workloads
# When to increase: If you see queue saturation in monitoring
# When NOT to increase: Low-latency workloads (higher queue = higher latency)

• With multiple NICs on the same subnet (e.g., 10.100.1.101 and 10.100.1.102), Linux may respond to ARP requests on the wrong interface.
• This can cause asymmetric routing and connection issues
• Storage array may send traffic to one IP but receive responses from a different interface.
• Setting arp_ignore=2 ensures each interface only responds to ARP requests for its own IP address.

# Configure ARP behavior for storage interfaces
# Option 1: Temporary (for testing)
echo 2 > /proc/sys/net/ipv4/conf/ens1f0/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/ens1f1/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_ignore

# For VLAN interfaces
echo 2 > /proc/sys/net/ipv4/conf/ens1f0.100/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/ens1f1.100/arp_ignore

# Option 2: Persistent (recommended)
cat >> /etc/sysctl.d/99-nvme-tcp-arp.conf << 'EOF'
# ARP configuration for NVMe-TCP multipath
# Prevents ARP responses on wrong interface when multiple NICs share same subnet

# For dedicated physical interfaces
net.ipv4.conf.ens1f0.arp_ignore = 2
net.ipv4.conf.ens1f1.arp_ignore = 2

# For VLAN interfaces (uncomment if using VLANs)
#net.ipv4.conf.ens1f0/100.arp_ignore = 2
#net.ipv4.conf.ens1f1/100.arp_ignore = 2

# Global setting (applies to all interfaces)
net.ipv4.conf.all.arp_ignore = 2
net.ipv4.conf.default.arp_ignore = 2
EOF

# Apply settings immediately
sysctl -p /etc/sysctl.d/99-nvme-tcp-arp.conf

# Verify settings
sysctl net.ipv4.conf.ens1f0.arp_ignore
sysctl net.ipv4.conf.ens1f1.arp_ignore
# Expected output: net.ipv4.conf.ens1f0.arp_ignore = 2

# Test ARP behavior from storage array or another host on same subnet
# Send ARP request for 10.100.1.101
arping -I <interface> 10.100.1.101

# Should only get response from ens1f0 MAC address, not ens1f1

# Monitor ARP traffic
tcpdump -i ens1f0 arp
tcpdump -i ens1f1 arp
# Each interface should only respond to ARP requests for its own IP

cat >> /etc/sysctl.d/99-nvme-tcp-arp.conf << 'EOF'
# ARP announce: Use best local address for ARP requests
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.ens1f0.arp_announce = 2
net.ipv4.conf.ens1f1.arp_announce = 2
EOF

sysctl -p /etc/sysctl.d/99-nvme-tcp-arp.conf

• When sending ARP requests, use the source IP that matches the outgoing interface.
• Prevents sending ARP requests with the wrong source IP.
• Complements arp_ignore for complete ARP control.