Now that we have a basic understanding of how iSCSI liveness is determined, let’s put this in a way that is easier to understand and why when a single cable pull is performed, a backend storage array is rebooted, a switch is taken down, etc. results in this lengthy pause.
Conceptually, this isn’t all too difficult to understand if we think about it logically for a moment based off of what we know. If we take a moment to review the information we have been given we can do some pretty simple math. If we pull a cable and an iSCSI session is unexpectedly lost the following events transpire:
- NOP-Out request is sent to the target port from the initiator (up to 15 seconds, depending on several factors).
- Initiator waits for a response from the target (10 seconds).
- No response is received from the target so session recovery kicks in (10 seconds).
- Session recovery fails I/O is sent to the SCSI layer to be failed and sent down existing healthy paths.
So we have: 15 + 10 + 10 = Up to 35 seconds of I/O time paused.
I have found that on average with ESXi the paused I/O time is roughly 25-28 seconds in time. But have seen on a few occasions where 30+ seconds was experienced and caused unwanted behavior to applications.
So why does all I/O pause for that amount of time when a single path is lost? Well, since applications are often reliant on previous I/O completing the initiator pauses all I/O to the target, down all paths, until the state of the suspect path(s) and all pending I/Os are determined. Since there is a chance it could recover and I/O is just completing slower than normal it doesn’t want to prematurely fail or retry until it is certain things are down. After all, recovery of an environment, especially prematurely, can cause just as many problems. That is what timers are for after all to dictate safe zones for specific actions to be taken!