Steps to convert Ephemeral volume for PX-Prometheus DB to Persistent with retention customisation

As of PX-Ent 3.3.1.3 and PX-CSI 25.6, if PX-Monitoring is enabled, by default PX creates prometheus with emptydir{} volume for its DB.

Questions:

1. How to Convert this Ephemeral volume for PX-Prometheus DB to Persistent volume?
2. How to customise the retention days and retention size?

• For new installation with px-monitoring enabled, add below volumeClaimTemplate in STC spec once it is generated before applying
• Customise the 
  • StorageClassName
  • retention
  • retentionSize
• Setting "spec.monitoring.prometheus.retention" propogates the retention time period as argument like --storage.tsdb.retention.time=<time-value>. This argument sets the time-based retention for the Time Series Database (TSDB). Any data older than this will be automatically deleted during compaction.
• Prometheus supports  size-based retention also and this can be set by updating "spec.monitoring.prometheus.retentionSize" which propogates --storage.tsdb.retention.size=<size_value>. If the total size of the data exceeds this limit, Prometheus will remove the oldest data, regardless of the time-based retention setting, to stay under the size cap.
• If retention is set to 90d and retentionSize is set to 51200MB, the
  • If prometheus data grows very fast, reaching the 50 GB size limit in less than 90 days, Prometheus will start deleting old data to stay within that limit.
  • If prometheus data grows slowly and doesn't reach the 50 GB limit, Prometheus will still delete data once it becomes older than 90 days.
  • Ensure the retentionSize is less than PVC size - "spec.monitoring.prometheus.storage.volumeClaimTemplate.spec.resources.requests.storage"

• For existing installation with px-monitoring, validate the type of volume being used. If it emptydir{}, follow below steps to convert it

Steps to convert this Ephemeral volume for PX-Prometheus DB to Persistent volume:

1. Take a safe copy of current stc by executing "kubectl -n <px-namespace> get stc -o yaml"
2. Edit STC by adding storage under "spec.monitoring.prometheus.storage.volumeClaimTemplate" with above Template
3. Once you save this, prometheus-px-prometheus-0 pod gets restarted and a new pod will create with persistent volume
4. Comparison of STC "without persistent volume" vs "with persistent volume" below

Validation:

1. Verification of persistent volume in portworx namespace for prometheus
2. Verification of volume type in prometheus-px-prometheus-0 pod

• Category : Software
• Subcategory : Software

Enhancement requested in have option in spec generator