VMware offers a Native Multipathing Plugin (NMP) layer in vSphere through Storage Array Type Plugins (SATP) and Path Selection Policies (PSP) as part of the VMware APIs for Pluggable Storage Architecture (PSA). The SATP has all the knowledge of the storage array to aggregate I/Os across multiple channels and has the intelligence to send failover commands when a path has failed. The Path Selection Policy can be either “Fixed”, “Most Recently Used” , “Round Robin” or "Latency Based Round Robin".
BEST PRACTICE: Enhanced Round Robin Load Balancing is configured by default on ESXi 7.0 and later. No configuration changes are required.
Enhanced Round Robin Load Balancing (Latency Based PSP)
In vSphere 6.7 U1, a sub-policy option for Round Robin that actively monitors individual path performance was added to ESXi. This sub-policy is called "Enhanced Round Robin Load Balancing" (also known as Latency Based Path Selection Policy (PSP)). Before this policy became available, the ESXi host would utilize all active paths by sending I/O requests down each path in a "fire and forget" type of fashion, sending 1 I/O down each path before moving to the next. Often times this resulted in performance penalties when individual paths became degraded and weren't functioning as optimally as other available paths. This performance penalty was invoked because the ESXi host would continue using the non-optimal path due to limited insight into the overall path health. This changed with the Latency Based PSP by monitoring each path for latency, along with outstanding I/Os, allowing the ESXi host to make smarter decisions on which paths to use and which to exclude in a more dynamic manner.
How it Works
Like all other Native Multipathing Plugin (NMP) policies, this sub-policy is set on a per LUN or per datastore basis. Once enabled, the NMP begins by assessing the first 16 user I/O requests per path and calculates their average latency. Once all of the paths have been successfully analyzed the NMP will then calculate the average latency of each path and use this information to determine which paths are healthy (optimal) and which are unhealthy (non-optimal). If a path falls outside of the average latency it is deemed non-optimal and will not be used until latency has reached an optimal response time once more.
After the initial assessment, the ESXi host then repeats the same process outlined above every 3 minutes. It will test every active path, including any non-optimal paths, to confirm if the latency has improved, worsened, or remained the same. Once again those results will be analyzed and used to determine which paths should continue sending I/O requests and which should be paused to see if they report better health in the next 3 minutes. Throughout this process the NMP is also taking into account any outstanding I/Os for each path to make more informed decisions.
Configuring Round Robin and the Latency Based Sub-Policy
If you are using ESXi 7.0 or later, then no changes are required to enable this new sub-policy as it is the new recommendation moving forward. In an effort to make things easier for end-users, a new SATP rule has been added that will automatically apply this rule to any Everpure LUNs presented to the ESXi host:
Name Device Vendor Model Driver Transport Options Rule Group Claim Options Default PSP PSP Options Description
VMW_SATP_ALUA PURE FlashArray system VMW_PSP_RR policy=latency
If your environment is using ESXi 6.7U1 to ESXi 7 and you wish to utilize this feature, the best way is to create a SATP rule on each ESXi host, which can be done as follows:
esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "PURE" -M "FlashArray" -P "VMW_PSP_RR" -O "policy=latency" -e "FlashArray SATP Rule"
Alternatively, this can be done using PowerShell:
Connect-VIServer -Server <vCenter> -Credential (Get-Credential)
Get-VMhost | Get-EsxCli –V2 | % {$_.storage.nmp.satp.rule.add.Invoke(@{description='Pure Storage FlashArray SATP';model='FlashArray';vendor='PURE';satp='VMW_SATP_ALUA';psp='VMW_PSP_RR'; pspoption='policy=latency'})}
Setting a new SATP rule will only change the policy for newly presented LUNs, it does not get applied to LUNs that were present before the rule was set until the host is rebooted.
Lastly, if you would like to change an individual LUN (or set of LUNs) you can run the following command to change the PSP to latency (where device is specific to the environment):
esxcli storage nmp psp roundrobin deviceconfig set --type=latency --device=naa.624a93708a75393becad4e43000540e8
Tuning
By default, the RR latency policy is configured to send 16 user I/O requests down each path and evaluate each path every three minutes (180000ms). Based on extensive testing, Everpure's recommendation is to leave these options configured to their defaults and no changes are required.
Round Robin Path Selection Policy
The Round Robin PSP rotates between all discovered paths for a given volume which allows ESXi (and therefore the virtual machines running on the volume) to maximize the possible performance by using all available resources (HBAs, target ports, etc.). When using ESXi 6.7U1 or later, Everpure recommends using the latency based PSP above. Round Robin is also supported.
The I/O Operations Limit
The Round Robin Path Selection Policy allows for additional tuning of its path-switching behavior in the form of a setting called the I/O Operations Limit. The I/O Operations Limit (sometimes called the “IOPS” value) dictates how often ESXi switches logical paths for a given device. By default, when Round Robin is enabled on a device, ESXi will switch to a new logical path every 1,000 I/Os. In other words, ESXi will choose a logical path, and start issuing all I/Os for that device down that path. Once it has issued 1,000 I/Os for that device, down that path, it will switch to a new logical path and so on.
When using Round Robin instead of the latency based PSP, Everpure recommends tuning this value down to the minimum of 1. This will cause ESXi to change logical paths after every single I/O, instead of 1,000.
This recommendation is made for a few reasons:
-
Performance. Often the reason cited to change this value is performance. While this is true in certain cases, the performance impact of changing this value is not usually profound (generally in the single digits of a percentage performance increase). While changing this value from 1,000 to 1 can improve performance, it generally will not solve a major performance problem. Regardless, changing this value can improve performance in some use cases, especially with iSCSI.
-
Path Failover Time. It has been noted in testing that ESXi will fail logical paths much more quickly when this value is set to a the minimum of 1. During a physical failure of the storage environment (loss of a HBA, switch, cable, port, controller) ESXi, after a certain period of time, will fail any logical path that relies on that failed physical hardware and will discontinue attempting to use it for a given volume. This failure does not always happen immediately. When the I/O Operations Limit is set to the default of 1,000 path failover time can sometimes be in the 10s of seconds which can lead to noticeable disruption in performance during this failure. When this value is set to the minimum of 1, path failover generally decreases to sub-ten seconds. This greatly reduces the impact of a physical failure in the storage environment and provides greater performance resiliency and reliability.
-
FlashArray Controller I/O Balance. When Purity is upgraded on a FlashArray, the following process is observed (at a high level): upgrade Purity on one controller, reboot it, wait for it to come back up, upgrade Purity on the other controller, reboot it and you’re done. Due to the reboots, twice during the process half of the FlashArray front-end ports go away. Because of this, we want to ensure that all hosts are actively using both controllers prior to upgrade. One method that is used to confirm this is to check the I/O balance from each host across both controllers. When volumes are configured to use Most Recently Used, an imbalance of 100% is usually observed (ESXi tends to select paths that lead to the same front end port for all devices). This then means additional troubleshooting to make sure that host can survive a controller reboot. When Round Robin is enabled with the default I/O Operations Limit, port imbalance is improved to about 20-30% difference. When the I/O Operations Limit is set to 1, this imbalance is less than 1%. This gives Everpure and the end user confidence that all hosts are properly using all available front-end ports.
For these three above reasons, Everpure highly recommends altering the I/O Operations Limit to 1. For additional information you can read the VMware KB regarding setting the IOPs Limit.
Note:BEST PRACTICE: Use the latency based PSP. If using Round Robin, Change the Round Robin I/O Operations Limit from 1,000 to 1 for FlashArray volumes on vSphere. This is a default configuration in all supported vSphere releases.
To fully utilize CPU resources, set the host's active power policy to high performance.
Configuring Round Robin and the I/O Operations Limit
If you are running earlier than ESXi 6.0 Express Patch 5 or 6.5 Update 1, there are a variety of ways to configure Round Robin and the I/O Operations Limit. This can be set on a per-device basis and as every new volume is added, these options can be set against that volume. This is not a particularly good option as one must do this for every new volume, which can make it easy to forget, and must do it on every host for every volume. This makes the chance of exposure to mistakes quite large.
The recommended option for configuring Round Robin and the correct I/O Operations Limit is to create a rule that will cause any new FlashArray device that is added in the future to that host to automatically get the Round Robin PSP and an I/O Operation Limit value of 1.
The following command creates a rule that achieves both of these for only Everpure FlashArray devices:
esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "PURE" -M "FlashArray" -P "VMW_PSP_RR" -O "iops=1" -e "FlashArray SATP Rule"This must be repeated for each ESXi host.
This can also be accomplished through PowerCLI. Once connected to a vCenter Server this script will iterate through all of the hosts in that particular vCenter and create a default rule to set Round Robin for all Everpure FlashArray devices with an I/O Operation Limit set to 1.
Connect-VIServer -Server <vCenter> -Credential (Get-Credential) Get-VMhost | Get-EsxCli –V2 | % {$_.storage.nmp.satp.rule.add.Invoke(@{description='Pure Storage FlashArray SATP';model='FlashArray';vendor='PURE';satp='VMW_SATP_ALUA';psp='VMW_PSP_RR'; pspoption='iops=1'})}Furthermore, this can be configured using vSphere Host Profiles:
It is important to note that existing, previously presented devices will need to be manually set to Round Robin and an I/O Operation Limit of 1. Optionally, the ESXi host can be rebooted so that it can inherit the multipathing configuration set forth by the new rule.
Note:Note that I/O Operations of 1 is the default in 6.0 Patch 5 and later in the 6.0 code branch, 6.5 Update 1 and later in the 6.5 code branch, all versions of 6.7; in 7.0 and later, the latency based PSP is the default for Everpure devices.
Verifying Connectivity
It is important to verify proper connectivity prior to implementing production workloads on a host or volume.
This consists of a few steps:
-
Verifying proper multipathing settings in ESXi.
-
Verifying the proper numbers of paths.
-
Verifying I/O balance and redundancy on the FlashArray.
The Path Selection Policy and number of paths can be verified easily inside of the vSphere Web Client.
This will report the path selection policy and the number of logical paths. The number of logical paths will depend on the number of HBAs, zoning and the number of ports cabled on the FlashArray.
The I/O Operations Limit cannot be checked from the vSphere Web Client—it can only be verified or altered via command line utilities. The following command can check a particular device for the PSP and I/O Operations Limit:
esxcli storage nmp device list -d naa.<device NAA>[root@dr-esxi-01:~] esxcli storage nmp device list -d naa.624a93708eabcb40cc4241b2000115da naa.624a93708eabcb40cc4241b2000115da Device Display Name: PURE Fibre Channel Disk (naa.624a93708eabcb40cc4241b2000115da) Storage Array Type: VMW_SATP_ALUA Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}{TPG_id=0,TPG_state=AO}} Path Selection Policy: VMW_PSP_RR Path Selection Policy Device Config: {policy=latency,latencyEvalTime=180000,samplingCycles=16,curSamplingCycle=16,useANO=0; CurrentPath=vmhba2:C0:T3:L251: NumIOsPending=0,latency=0} Path Selection Policy Device Custom Config: Working Paths: vmhba1:C0:T4:L251, vmhba2:C0:T3:L251, vmhba1:C0:T2:L251 Is USB: falsePlease remember that each of these settings is a per-host setting, so while a volume might be configured properly on one host, it may not be correct on another.
Additionally, it is also possible to check multipathing from the FlashArray.
A CLI command exists to monitor I/O balance coming into the array:
purehost monitor --balance --interval <how long to sample> --repeat <how many iterations>The command will report a few things:
-
The host name.
-
The individual initiators from the host. If they are logged into more than one FlashArray port, it will be reported more than once. If an initiator is not logged in at all, it will not appear.
-
The port that the initiator is logged into.
-
The number of I/Os that came into that port from that initiator over the time period sampled.
-
The relative percentage of I/Os for that initiator as compared to the maximum.
The balance command will count the I/Os that came down from a particular initiator during the sampled time period, and it will do that for all initiator/target relationships for that host. Whichever relationship/path has the most I/Os will be designated as 100%. The rest of the paths will be then denoted as a percentage of that number. So if a host has two paths, and the first path has 1,000 I/Os and the second path has 800, the first path will be 100% and the second will be 80%.
A well balanced host should be within a few percentage points of each path. Anything more than 15% or so might be worthy of investigation. Refer to this post for more information.
Note:Please keep in mind that if the Latency Based PSP is in use that IO may not be 1 to 1 for all paths to the Array from the ESXi hosts.
There is nothing inherently wrong with the IO not being balanced 1 to 1 for all paths as the Latency Bases PSP will be distributing IO based on which path has the lowest latency. With that said, a few percentage points difference shouldn't be cause for alarm, however if there are paths with very little to no IO being sent down them this should be something investigated in the SAN to find out why that path is performing poorly.
The GUI will also report on host connectivity in general, based on initiator logins.
This report should be listed as redundant for all hosts, meaning that it is connected to each controller. If this reports something else, investigate zoning and/or host configuration to correct this.
For a detailed explanation of the various reported states, please refer to the FlashArray User Guide which can be found directly in your GUI: