The first VMware feature we want to discuss today is called “Adaptive Queueing”, also known as the “Adaptive Queue Depth Algorithm”. Throughout this article we will refer to this feature as AQDA.
AQDA was first introduced in ESXi 3.5U4 (over 12 years ago in 2008!). This technology was released to help overloaded storage arrays (or storage ports) by throttling how much I/O is allowed to be sent at once. AQDA accomplishes this task by dynamically changing (reducing) the queue depth on a per datastore level for any device that is reporting congestion.
By reducing the queue on the impacted datastores, this limits the amount of outstanding I/O that can be sent to the storage array. In theory, by slowing down incoming I/O request, this should give the array time to “catch up” to the workload. If nothing else, it will hopefully at least create a more consistent bandwidth / throughput / latency. This means that there “should be” less impact to the environment. The “should be” is in quotes because there are obviously some situations in which it may not help or may not help enough. You hopefully get the general idea of its purpose by this point in time though.
One of the most important takeaways with AQDA is that the congestion has to be reported to the ESXi host from the storage array before it kicks in. It is not simply based off of latency. The storage array informs the ESXi host of congestion by sending specific SCSI Status Codes back to the host as I/O requests are received.
The specific SCSI codes required to trigger AQDA are:
- TASK_SET_FULL (0x28)
- BUSY (0x8)
As the ESXi host monitors the I/O responses it will determine which datastore is reporting the congestion and keep a count of how many of those requests are tagged as BUSY or FULL. Once a specific amount of these congestions responses have been received (we’ll say 16) then the ESXi host will begin to reduce the queue depth and throttle the I/O being sent to the impacted datastore.
AQDA does this by cutting the queue in halves as congestion is continually reported (e.g 64 > 32 > 16 > 8 > 4 > 2 > 1). We have confirmed through testing that queue depth can indeed drop all the way down to 1 with AQDA. This only happened in the most extreme of circumstances though when QoS was set very low on the volume and a very high workload was triggered.
As time goes on, and the storage array is able to successfully complete I/O requests, the ESXi host will keep track of these completions (0x00). Again, once a specific amount of successful completions are received (we’ll say 8) from storage it will slowly start to increase the queue depth back to its original value before the congestion event was reported. These queue slots are returned one at a time and do not double like it does when the queue is being reduced. So if the queue dropped from 64 down to 32 then AQDA would need a total of 256 successful I/O completions to get back to the original queue depth value of 64 (32 * 8).
Important Note: If TASK_SET_FULL or BUSY responses are received while the queue is being replenished then the queue slots will stop filling until there are 8 successful completions in a row. If the threshold for BUSY or TASK_SET_FULL responses are met, then not only will the queue stop increasing but it will begin to decrease again to slow the flow of I/O.