ESXi and vCenter offer a variety of features to control the performance capabilities of a given datastore. This section will overview FlashArray support and recommendations for these features.
Queue Depth Limits
ESXi offers the ability to configure queue depth limits for devices on an HBA or iSCSI initiator. This dictates how many I/Os can be outstanding to a given device before I/Os start queuing in the ESXi kernel. If the queue depth limit is set too low, IOPS and throughput can be limited and latency can increase due to queuing. If too high, virtual machine I/O fairness can be affected and high-volume workloads can affect other workloads from other virtual machines or other hosts. The device queue depth limit is set on the initiator and the value (and setting name) varies depending on the model and type:
|
Type |
Default Value |
Value Name |
|---|---|---|
|
QLogic |
64 |
qlfxmaxqdepth |
|
Brocade |
32 |
bfa_lun_queue_depth |
|
Emulex |
32 |
lpfc0_lun_queue_depth |
|
Cisco UCS |
32 |
fnic_max_qdepth |
|
Software iSCSI |
128 |
iscsivmk_LunQDepth |
Changing these settings require a host reboot. For instructions to check and set these values, please refer to this VMware KB article:
Changing the queue depth for QLogic, Emulex, and Brocade HBAs
Disk Schedule Number Requests Outstanding (DSNRO)
There is a second per-device setting called “Disk Schedule Number Requests Outstanding” often referred to as DSNRO. This is a hypervisor-level queue depth limit that provides a mechanism for managing the queue depth limit for an individual device. This value is a per-device setting that defaults to 32 and can be increased to a value of 256.
It should be noted that this value only comes into play for a volume when that volume is being accessed by two or more virtual machines on that host. If there is more than one virtual machine active on it, the lowest of the two values (DSNRO or the HBA device queue depth limit) is the value that is observed by ESXi as the actual device queue depth limit. So, in other words, if a volume has two VMs on it, and DSNRO is set to 32 and the HBA device queue depth limit is set to 64, the actual queue depth limit for that device is 32. For more information on DSNRO see the VMware KB here:
Setting the Maximum Outstanding Disk Requests for virtual machines
In general, Everpure does not recommend changing these values. The majority of workloads are distributed across hosts and/or not intense enough to overwhelm the default queue depths. The FlashArray is fast enough (low enough latency) that the workload has to be quite high in order to overwhelm the queue.
If the default queue depth is consistently overwhelmed, the simplest option is to provision a new datastore and distribute some virtual machines to the new datastore. If a workload from a single virtual machine is too great for the default queue depth, then increasing the queue depth limit is the better option.
If a workload demands queue depths to be increased, Everpure recommends making both the HBA device queue depth limit and DSNRO equal.
Do not change these values without direction from VMware or Everpure support as this can have performance repercussions.
You can verify the values of both of these for a given device with the command:
esxcli storage core device list –d <naa.xxxxx>
Device Max Queue Depth: 96
No of outstanding IOs with competing worlds: 64
BEST PRACTICE: Do not modify queue depth limits, leave them at their default. Only raise them when performance requirements dictate it and Everpure Support or VMware Support provide appropriate guidance.
Dynamic Queue Throttling
ESXi supports the ability to dynamically throttle a device queue depth limit when an array volume has been overwhelmed. An array volume is overwhelmed when the array responds to an I/O request with a sense code of QUEUE FULL or BUSY. When a certain number of these are received, ESXi will reduce the queue depth limit for that device and slowly increase it as conditions improve.
This is controlled via two settings:
- Disk.QFullSampleSize—The count of QUEUE FULL or BUSY conditions it takes before ESXi will start throttling. Default is zero (feature disabled)
- Disk.QFullThreshold—The count of good condition responses after a QUEUE FULL or BUSY required before ESXi starts increasing the queue depth limit again
The Everpure FlashArray does not advertise a QUEUE FULL condition for a volume. Since every volume can use the full performance and queue of the FlashArray, this limit is unrealistically high and this sense code will likely never be issued. Therefore, there is no reason to set or alter these values for Everpure FlashArray volumes because QUEUE FULL should rarely (or never) occur.
Storage I/O Control
VMware vCenter offers a feature called Storage I/O Control (SIOC) that will throttle selected virtual machines when a certain average latency has been reached or when a certain percentage of peak throughput has been hit on a given datastore. ESXi throttles virtual machines by artificially reducing the number of slots that are available to it in the device queue depth limit.
Everpure fully supports enabling this technology on datastores residing on the FlashArray. That being said, it may not be particularly useful for a few reasons.
First, the minimum latency that can be configured for SIOC before it will begin throttling a virtual machine is 5 ms.
When a latency threshold is entered, vCenter will aggregate a weighted average of all disk latencies seen by all hosts that see that particular datastore. This number does not include host-side queuing, it is only the time it takes for the I/O to be sent from the SAN to the array and acknowledged back.
Furthermore, SIOC uses a random-read injector to identify the capabilities of a datastore from a performance perspective. At a high-level, it runs a quick series of tests with increasing numbers of outstanding I/Os to identify the throughput maximums via high latency identification. This allows ESXi to determine what the peak throughput is, for when the “Percentage of peak throughput” is chosen.
Knowing these factors, we can make these points about SIOC and the FlashArray:
- SIOC is not going to be particularly helpful if there is host-side queuing since it does not take host-induced latency into account. This (the ESXi device queue) is generally where most of the latency is introduced in a FlashArray environment.
- The FlashArray will rarely have sustained latency above 1 ms, thus reducing the likely hood that this threshold will be reached for any meaningful amount of time on a FlashArray volume.
- A single FlashArray volume does not have a queue limit, so it can handle quite a high number of outstanding I/O and throughput (especially reads), therefore SIOC and its random-read injector cannot identify FlashArray limits in meaningful ways.
In short, SIOC is fully supported by Everpure, but Everpure makes no specific recommendations for configuration.
Storage DRS
VMware vCenter also offers a feature called Storage Dynamic Resource Scheduler (Storage DRS / SDRS). SDRS moves virtual machines from one datastore to another when a certain average latency threshold has been reach on the datastore or when a certain used capacity has been reached. For this section, let’s focus on the performance-based moves.
Storage DRS, like Storage IO Control, waits for a certain latency threshold to be reached before it acts. And, also like SIOC, the minimum is 5 ms.
While it is too high in general to be useful for FlashArray induced latency, SDRS differs from SIOC in the latency it actually looks at. SDRS uses the “VMObservedLatency” (referred to a GAVG in esxtop) averages from the hosts accessing the datastore. Therefore, this latency includes time spent queuing in the ESXi kernel. So, theoretically, a high-IOPS workload, with a low configured device queue depth limit, an I/O could conceivably spend 5 ms or more queuing in the kernel. In this situation Storage DRS will suggest moving a virtual machine to a datastore which does not have an overwhelmed queue.
That being said, this is still an unlikely scenario because: