Low Throughput While Using NVMe-TCP on RHEL/Rocky/AlmaLinux

NVMe-TCP on RHEL/Rocky/AlmaLinux

• Throughput below expected
• Not utilizing all paths
• Network bandwidth underutilized

1. FlashArray not using all paths
2. Network bottleneck
3. I/O scheduler
4. CPU bottleneck

Diagnosis:

# Test raw throughput
fio --name=test --rw=read --bs=1M --size=10G --filename=/dev/nvme0n1

# Check network utilization
iftop -i <interface>

# Check all paths are active
nvme list-subsys

# Verify IO policy
cat /sys/class/nvme-subsystem/nvme-subsys*/iopolicy

Solutions:

1. Not using all paths:
   • Verify IO policy is round-robin: echo "round-robin" > /sys/class/nvme-subsystem/nvme-subsys0/iopolicy
   • Ensure all connections are live
2. Network bottleneck:
   • Verify link speed: ethtool <interface> | grep Speed
   • Check for packet drops: ethtool -S <interface>
   • Verify flow control: ethtool -a <interface>
3. I/O scheduler:
   • Verify scheduler is ‘none’ for NVMe devices
   • Increase queue depth if needed
4. CPU bottleneck:
   • Check CPU utilization
   • Enable offload features
   • Distribute IRQs across CPUs