Storage I/O Control (referred to as SIOC moving forward) is a feature created by VMware that is also triggered during times of increased congestion. This is actually quite a powerful feature (if used properly) and we could spend quite a long time explaining every detail. Since this is more about helping you understand what it does and what conditions trigger this feature, we won’t give you all the details, but will provide links to places you can go further into detail if you so desire.
Some people may ask, “If we already have AQDA, why create another feature for congestion?”. This is a fair and reasonable question, one that is relatively simple in concept but a little trickier under the hood.
We know that AQDA is engaged only when the storage array says, at a SCSI level,
“Hey, I am a little overloaded over here… mind slowing down your I/O requests so I can catch up?”
Once that signal is received the ESXi host begins to reduce the queue and slow things down until the signal received from storage says,
“Okay, all caught up”,
at which point the queue slots are slowly replenished.
This is a great feature, right? But what about times the storage array isn’t reporting problems because it isn’t responsible for the congestion? Or maybe the storage array simply doesn’t send those notifications? What if congestion is happening within the environment due to other limitations? I know we all love to blame storage, but storage isn’t always the problem! Other things that could cause congestion are oversaturated network, failing switches and/or paths, a rouge VM (or VMs), and the list goes on…. and this is where SIOC comes into play and really shines.
Unlike AQDA, SIOC is engaged at times of increased latency on individual datastores rather than waiting for SCSI codes from an array. This means that SIOC doesn’t have to wait for storage to say there is a problem before it is able to kick in. It can engage and start trying to mitigate (or lessen the impact) of the problem without that signal.
In current versions of ESXi there are two options available for determining when SIOC should engage:
– Percentage of Peak Throughput – This option indicates the estimated latency threshold when the datastore is using that percentage of its estimated peak throughput. (Default is 90%)
– Manual – This allows the end-user to manually determine what the latency should be before SIOC engages (Default: 30ms)
Once engaged, SIOC works very similarly to adaptive queueing (with slight differences), in the sense that it slows the rate of I/O by adaptively changing (reducing) the queue depth of the impacted datastore. As soon as latency subsides then the queue depth will be increased to allow for full rate of exchange between the hosts and storage.
Okay great, but if AQDA and SIOC both work by reducing queues then what more is SIOC providing outside of simply engaging during increased latency rather then SCSI triggered events?
In a single (important) word: fairness . The best part about SIOC is that it takes into account all of the VMs residing on the impacted datastore and is cluster aware , which means it doesn’t matter which ESXi hosts are housing the VMs running on the impacted datastore. This means that when I/O throttling does need to be enforced, SIOC will ensure that every VM has the same opportunity to send I/O to the storage array. This will ensure that no single VM can “steal” all of the storage queues during the time of lessened storage resources (unless you specifically configure it to do so).
This is something that AQDA does not offer and is one of the big differentiators between the two.
Several other key distinctions about SIOC should be noted:
- SIOC statistics are evaluated every 4 seconds to determine what action should be taken based on the latency and incoming I/O.
- SIOC uses the average latency of a datastore across all connected hosts. This means that if a single host for some reason is seeing higher latency, but all others don’t, SIOC will not engage… not unless their average still exceeds the latency threshold.
- SIOC will not lower the queue depth below a value of 4.
As stated previously, this is a somewhat simplistic view of what SIOC does under the covers, and there is more complexity around ensuring proper shares are set for more critical VMs, how it can be tuned, etc and that can be found in the resources listed below (and many more I am sure).
- Storage I/O Control, the basics – Duncan Epping Blog Post
- vSphere 6.5 what’s new – Storage IO Control – Duncan Epping Blog Post
- Managing I/O Resources – VMware Official Documentation