A common question when first provisioning storage on the FlashArray is what capacity should I be using for each volume? VMware VMFS supports up to a maximum size of 64 TB. The FlashArray supports far larger than that, but for ESXi, volumes should not be made larger than 64 TB due to the filesystem limit of VMFS.
Using a smaller number of large volumes is generally a better idea today. In the past a recommendation to use a larger number of smaller volumes was made for performance limitations that no longer exist.
This limit traditionally was due to two reasons:
- VMFS scalability issues due to locking
- Per-volume queue limitations on the underlying array. VMware resolved the first issue with the introduction of Atomic Test and Set (ATS), also called Hardware Assisted Locking. Prior to the introduction of VAAI ATS, VMFS used LUN-level locking via full SCSI-2 reservations to acquire exclusive metadata control for a VMFS volume. In a cluster with multiple nodes, all metadata operations were serialized and hosts had to wait until whichever host, currently holding a lock, released that lock. This behavior not only caused metadata lock queues but also prevented standard I/O to a volume from VMs on other ESXi hosts which were not currently holding the lock. With VAAI ATS, the lock granularity is reduced to a much smaller level of control (specific metadata segments, not an entire volume) for the VMFS that a given host needs to access. This behavior makes the metadata change process not only very efficient, but more importantly provides a mechanism for parallel metadata access while still maintaining data integrity and availability. ATS allows for ESXi hosts to no longer queue metadata change requests, which consequently speeds up operations that previously had to wait for a lock. Therefore, situations with large amounts of simultaneous virtual machine provisioning operations will see the most benefit. The standard use cases benefiting the most from ATS include:
- High virtual machine to VMFS density.
- Extremely dynamic environments—numerous provisioning and de-provisioning of VMs (e.g. VDI using non-persistent linked-clones).
- High intensity virtual machine operations such as boot storms, or virtual disk growth.
The introduction of ATS removed scaling limits via the removal of lock contention; thus, moving the bottleneck down to the storage, where many traditional arrays had per-volume I/O queue limits. This limited what a single volume could do from a performance perspective as compared to what the array could do in aggregate. This is not the case with the FlashArray.
A FlashArray volume is not limited by an artificial performance limit or an individual queue. A single FlashArray volume can offer the full performance of an entire FlashArray, so provisioning ten volumes instead of one, is not going to empty the HBAs out any faster. From a FlashArray perspective, there is no immediate performance benefit to using more than one volume for your virtual machines.
The main point is that there is always a bottleneck somewhere, and when you fix that bottleneck, it is transferred somewhere in the storage stack. ESXi was once the bottleneck due to its locking mechanism, then it fixed that with ATS. This, in turn, moved the bottleneck down to the array volume queue depth limit. The FlashArray doesn’t have a volume queue depth limit, so now that bottleneck has been moved back to ESXi and its internal queues.
Altering VMware queue limits is not generally needed with the exception of extraordinarily intense workloads. For high-performance configuration, refer to the section of this document on Datastore Management