As mentioned earlier, the Paravirtual SCSI adapter should be leveraged for the best default performance. For virtual machines that host applications that need to push a large amount of IOPS (50,000+) to a single virtual disk, some non-default configurations are required. The PVSCSI adapter allows the default adapter queue depth limit and the per-device queue depth limit to be increased from the default of 256 and 64 (respectively) to 1024 and 256.
In general, this change is not needed and therefore not recommended for most workloads. Only increase these values if you know a virtual machine needs or will need this additional queue depth. Opening this queue for a virtual machine that does not (or should not) need it, can expose noisy neighbor performance issues. If a virtual machine has a process that unexpectedly becomes intense it can unfairly steal queue slots from other virtual machines sharing the underlying datastore on that host. This can then cause the performance of other virtual machines to suffer.
BEST PRACTICE: Leave virtual machine queue depth limits at the default unless performance requirements dictate otherwise.
If an application does need to push a high amount of IOPS to a single virtual disk these limits must be increased. See VMware KB here for information on how to configure Paravirtual SCSI adapter queue limits. The process slightly differs between Linux and Windows.
Refer to this blog post for more information:
http://www.codyhosterman.com/2017/02/understanding-vmware-esxi-queuing-and-the-flasharray/
A few general recommendations:
- Only increase these limits when needed
- If you change this limit it is required to change queue depth limits in ESXi as well, otherwise changing these values will have no tangible affect
- A good rule of knowing if you need to change these values is if you are not getting the IOPS you expect and the latency is high in the guest, but not reported as high in ESXi or on the FlashArray volume