Actions Needed for "Pools >75% full" Alert

Recommended Actions for Pools Exceeding 75% Utilization 

Sample alert is given below

affected nodes: [{'node_name': '0e366e49-6e99-4254-9ce8-8f8ea1f4b7be', 'pool_id': 'c62d2025-9bde-47d7-85f6-75fd8f658437', 'approx time': '2026-01-05 20:40:02', 'number of alerts': 1, 'message': 'Free disk space (110.0 GiB available of 550.0 GiB) is below recommended level Min(Percent 0.25 (137.5 GiB), AbsValue 200.0 GiB) on node 0e366e49-6e99-4254-9ce8-8f8ea1f4b7be (XX.YY.ZZ.YZ) for pool c62d2025-9bde-47d7-85f6-75fd8f658437'}]

This warning is triggered when any storage pool in the cluster exceeds 75% of its total capacity. It is intended to notify users to take corrective action before the pool reaches full capacity.

Listing the actions we can perform based on different scenarios

• Check whether storage pools are unevenly utilized by running the following command:

pxctl cluster provision-status

• Identify any pools that are significantly under-utilized. If under-utilized pools are detected, initiate a one-time pool rebalance using below command

pxctl service pool rebalance submit

• To automate pool rebalancing, Portworx Autopilot can be configured as described in the following documentation:

https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/autopilot/use-cases/rebalance-pool

• Volume replicas can be manually moved using the pxctl volume ha-update command

• To identify snapshots residing in a specific pool or node, use the following command:

pxctl v l -s --pool-uid <Pool ID>
pxctl v l -s --node <Node ID>

• The following command displays snapshots along with their creation dates:

export POOL=<Pool UUID>
for i in $(pxctl v l -s --pool-uid $POOL -s|tail -n +2|awk '{print $1}'); do echo $i|tr -d "\n";pxctl v i $i|grep Creation;done

• Review the snapshot list and delete any snapshots that are no longer required with following command

pxctl volume delete <Snapshot ID>

• Identify the volumes residing in a specific pool or node to determine if any are unnecessary. The following command can be used to list

pxctl v l -v --pool-uid <Pool ID>
pxctl v l -v --node <Node ID>

• The following command displays each volume along with its creation date and usage for review. Some volumes may still exist even if their corresponding PVC has already been deleted from the k8s layer.

export NODE="<node id>"
for i in $(pxctl v l -v --node $NODE|tail -n +2|awk '{print $1}');do echo $i|tr -d "\n";pxctl v i $i|egrep "Size|Bytes used|State|Last Attached"|tr -d  "\n";echo "";done

If everything looks good as per the above steps, pool expansion can be performed

https://docs.portworx.com/portworx-enterprise/operations/scale-portworx-cluster/expand-storage-pool