Portworx managed prometheus pod is continuously restarting

Audience
Public
Product
Portworx
Content Type
Source Type
Knowledge Base

Portworx managed prometheus pod is continuously restarting

KB0022601
Pure Product
Portworx
Environment
PX Enterprise
PX CSI
Question

What could be the reason of portworx managed prometheus pod is continuously restarting?

Answer

 

Reason for Restart:

  • As part of the default installation, Portworx can deploy a managed Prometheus by setting spec.monitoring.prometheus.enabled to true. This deploys both the Prometheus Operator and a Prometheus instance within the namespace where Portworx is installed.
  • If another Prometheus Operator is running in the cluster and managing cluster-wide Prometheus objects, it can interfere with the Portworx-managed Prometheus instance. This may modify the running StatefulSet, causing continuous restarts and conflicts between multiple operators.

Impact:

  • This does not impact the overall functioning of the cluster.
  • When Autopilot is enabled on the PX Enterprise cluster and uses the Portworx-managed Prometheus endpoint, its functionality can be affected.

Resolution

  • If a central Prometheus is running in the cluster, the Portworx-managed Prometheus can be disabled by setting spec.monitoring.prometheus.enabled to false in the StorageCluster.
  • When Autopilot rules are enabled in the PX Enterprise cluster, the cluster can be configured to use a central Prometheus instance once the Portworx-managed Prometheus is disabled. Below example shows pointing autopilot to rancher prometheus
spec:
  autopilot:
    enabled: true
    providers:
    - name: default
      type: prometheus
      params:
      url: http://rancher-monitoring-prometheus.cattle-monitoring-system:9090

 

  • When the additional Prometheus instance in the cluster is unnecessary, it can be disabled to ensure the Portworx-managed Prometheus operates correctly without conflicts.