Managing the capacity usage of your VMFS datastores is an important part of regular care in your virtual infrastructure. There are a variety of mechanisms inside of ESXi and vCenter to monitor capacity. Frequently, the concept of data reduction on the FlashArray is seen as a complicating factor, when in reality it is a simplifying factor, or at worse, a non-issue.
Letās overview some concepts on how to best manage VMFS datastores from a capacity perspective.
VMFS Usage vs. FlashArray Volume Capacity
VMFS reports how much is currently allocated in the filesystem on that volume. Depending on the type of virtual disk (thin or thick), dictates how much is consumed upon creation of the virtual machine (or virtual disk specifically). Thin disks only allocate what the guest has actually written to, and therefore VMFS only records what the virtual machine has written in its space usage. Thick type virtual disks allocate the full virtual disk immediately, so VMFS records much more space as being used than is actually used by the virtual machines.
This is one of the reasons thin virtual disks are preferredāyou get better insight into how much space the guests are actually using.
Regardless of what type you choose, ESXi is going to take the sum total of the allocated space of your virtual disks and compare that to the total capacity of the filesystem of the volume. The used space is the sum of those virtual disks allocations. This number increases as virtual disks grow or new ones are added, and can decrease as old ones are deleted, moved, or even shrunk.
Compare this to what the FlashArray reports for capacity. What the FlashArray reports for volume usage is NOT the amount used for that volume. What the FlashArray reports is the unique footprint of the volume on that array.
In the example below we can see that we are using a 5 TB FlashArray volume and VMFS datastorea. The example confirms that the VMFS datastore reports a total of 720.72 GB of used space on the 5TB filesystem. This tell us that there is a combined total of 720.72 GB of allocated virtual disks on this filesystem:
Now letās look at the FlashArray volume.
The FlashArray volume shows that 50.33 GB is being used. Does this mean that VMFS is incorrect? No. VMFS is always the source of truth. The āVolumesā metric on the FlashArray simply represents the amount of physical capacity that has been written to the volume after data reduction that no other volume shares.
This metric can change at any time as the data set changes on that volume or any other volume on the FlashArray. If, for instance, some other host writes 2 GB to another volume (letās call it āvolume2ā), and that 2 GB happens to be identical to 2 GB of that 50.33 GB GB on āsn1-m20-e05-28-prod-dsā, then āsn1-m20-e05-28-prod-dsā would no longer have 50.33 GB of unique space. It would drop down to 48.33 GB, even though nothing changed on āsn1-m20-e05-28-prod-dsā itself. Instead, another application just happened to write similar data, making the footprint of āsn1-m20-e05-28-prod-dsā less unique.
For a more detailed conversation around this, refer to this blog post:
http://www.codyhosterman.com/2017/01/vmfs-capacity-monitoring-in-a-data-reducing-world/
Why doesnāt VMFS report the same used capacity as the FlashArray for the underlying volume?
Well, because they mean different things. VMware reports what is allocated on the VMFS and the FlashArray reports what is unique to the underlying volume. The FlashArray value can change constantly. The FlashArray metric is only meant to show how reducible the data on that volume is internal to the volume and against the entire array. Conversely, VMFS capacity usage is based solely on how much capacity is allocated to it by virtual machines. The FlashArray volume space metric, on the other hand, actually relates to what is also being used on other volumes. In other words, VMFS usage is only affected by data on the VMFS volume itself. The FlashArray volume space metric is affected by the data on all of the volumes. So the two values should not be conflated.
For capacity tracking, you should refer to the VMFS usage. How do we best track VMFS usage? What do we do when it is full?
Monitoring and Managing VMFS Capacity Usage
As virtual machines grow and as new ones are added, the VMFS volume they sit on will slowly fill up. How to respond and to manage this is a common question.
In general, using a product like VCF Operations with the Everpure FlashArray Management Pack for VMware Cloud Foundation (VCF) Operations is a great option here. But for the purposes of this document we will focus on what can be done inside of vCenter alone.
You need to decide on a few things:
- At what percentage full of my VMFS volume do I become concerned?
- When that happens what should I do?
- What capacity value should I monitor on the FlashArray?
The first question is the easiest to answer. Choose either a percentage full or at a certain capacity free. Do you want to do something when, for example, a VMFS volume hits 75% full or when there is less than 50 GB free? Choose what makes sense to you.
vCenter alerts are a great way to monitor VMFS capacity automatically. There is a default alert for datastore capacity, but it does not do anything other than tag the datastore object with the alarm state. Everpure recommends creating an additional alarm for capacity that executes some type of additional action when the alarm is triggered.
Configuring a script to run, an email to be issued, or a notification trap to be sent greatly diminishes the chance of a datastore running out of space unnoticed.
BEST PRACTICE: Configure capacity alerts to send a message or initiate an action.
The next step is to decide what happens when a capacity warning occurs.
There are a few options:
-
Add a new volume
Your solution may be one of these options or a mix of all three. Letās quickly walk through the options.
Option 1: Increase the capacity of the volume
This is the simplest option. If capacity has crossed the threshold you have specified, increase the volume capacity to clear the threshold.
The process is:
1. Increase the FlashArray volume capacity.
2. Rescan the hosts that use the datastore.
3. Increase the VMFS to use the new capacity.
4. Choose āUse āFree space xxx GB/TBā to expand the datastoreā .
Note:There should be a note that the datastore already occupies space on this volume. If this note does not appear, you have selected the wrong device to expand. Everpure highly recommends that you do not create VMFS datastores that span multiple volumesāa VMFS should have a one to one relationship to a FlashArray volume.
5. This will clear the alarm and add additional capacity.
Option 2: Move virtual machine off of the volume
Another option is to move one or more virtual machines from a more-full datastore to a less-full datastore. While this can be manually achieved through case-by-case Storage vMotion, Everpure recommends leveraging Storage DRS to automate this. Storage DRS provides, in addition to the performance-based moves discussed earlier in this document, the ability to automatically Storage vMotion VMs based on capacity usage of VMFS datastores. If a datastore reaches a certain percent full, SDRS can automatically move, or make recommendations for, virtual machines to be moved to balance out space usage across volumes.
1. SDRS is enabled on a datastore cluster.
2. When a datastore cluster is created you can enable SDRS and choose capacity threshold settings, which can either be a percentage or a capacity amount.
Everpure has no specific recommendations for these values and can be decided upon based on your own environment. Everpure does have a few recommendations for datastore cluster configuration in general:
- Only include datastores on the same FlashArray in a given datastore cluster. This will allow Storage vMotion to use the VAAI XCOPY offload to accelerate the migration process of virtual machines and greatly reduce the footprint of the migration workload.
- Include datastores with similar configurations in a datastore cluster. For example, if a datastore is replicated on the FlashArray, only include datastores that are replicated in the same FlashArray protection group so that a SDRS migration does not violate required protection for a virtual machine.
Option 3: Create a new VMFS volume
The last option is to create an entirely new VMFS volume. You might decide to do this for a few reasons:
- The current VMFS volumes have maxed out possible capacity (64 TB each).
- The current VMFS volumes have overloaded the queue depth inside of every ESXi server using it. Therefore, they can be grown in capacity, but cannot provide any more performance due to ESXi limits.
In this situation follow the standard VMFS provisioning steps for a new datastore. Once the creation of volumes and hosts/host groups and the volume connection is complete, the volumes will be accessible to the ESXi host(s). Using the vSphere Web Client, initiate a āRescan Storageā¦ā to make the newly-connected Everpure volume(s) fully-visible to the ESXi servers in the cluster. One can then use the āAdd Storageā wizard to format the newly added volume.
Shrinking a Volume
Note:While it is possible to shrink a FlashArray volume non-disruptively, vSphere does not have the ability to shrink a VMFS partition. Therefore, do not shrink FlashArray volumes that contain VMFS datastores as doing so could incur data loss.
If you have mistakenly increased the size of a datastore, or a larger datastore is simply no longer required, the right steps to take would be creating a new datastore at the required size and then migrating the VMs from the old datstore to the new. Once the migration has been completed you can destroy the old datastore and remove the volume from the FlashArray.
Mounting a Snapshot Volume
The Everpure FlashArray provides the ability to take local or remote point-in-time snapshots of volumes which can then be used for backup/restore and/or test/dev. When a snapshot is taken of a volume containing VMFS, there are a few additional steps from both the FlashArray and vSphere sides to be able to access the snapshot point-in-time data.
When a FlashArray snapshot is taken, a new volume is not createdāessentially it is a metadata point-in-time reference to data blocks on the array that reflect that momentās version of the data. This snapshot is immutable and cannot be directly mounted. Instead, the metadata of a snapshot has to be ācopiedā to an actual volume which then allows the point-in-time, which was preserved by the snapshot metadata, to be presented to a host. This behavior allows the snapshot to be re-used again and again without changing the data in that snapshot. If a snapshot is not needed more than one time an alternative option is to create a direct snap copy from one volume to anotherāmerging the snapshot creation step with the association step.
When a volume hosting a VMFS datastore is copied via array-based snapshots, the copied VMFS datastore is now on a volume that has a different serial number than the original source volume. Therefore, the VMFS will be reported as having an invalid signature since the VMFS datastore signature is a hash partially based on the serial of the hosting device. Consequently, the device will not be automatically mounted upon rescanāinstead the new datastore wizard needs to be run to find the device and resignature the VMFS datastore. Everpure recommends resignaturing copied volumes rather than mounting them with an existing signatures (referred to as force mounting).
Note:BEST PRACTICE: "Assign a new signature" to copied VMFS volumes and do not force mount them.
For additional details on resignaturing and snapshot management, please refer to the following blog posts: