RWX volumes that takes long time to get mounted into Pods.

Sometimes, my RWX volumes on Kubernetes using Portworx when having lot of files, takes long time to get mounted each time that I have to scale up a new Pod.

Is a known issue caused by Kubernetes way to deal with permissions on shared volumes.

It got fixed on Kubernetes 1.20 but you require to configure your Pods/Deployments/Statefulsets.

https://github.com/kubernetes/kubernetes/issues/69699

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods 

https://kubernetes.io/blog/2020/12/14/kubernetes-release-1.20-fsgroupchangepolicy-fsgrouppolicy/

In particular, the solution is to set the securityContext on your Pods/Deployments/Statefulsets to define the same fsGroup for all the consumers of your PVC and also set the fsGroupChangePolicy to OnRootMismatch 

      securityContext:
        runAsUser: 2000
        runAsGroup: 3000
        fsGroup: 3000
        fsGroupChangePolicy: "OnRootMismatch"

This is the whole example.

Notice how two deployments are consuming the same shared volumes, but sharing the same fsGroup

# This will use NFS 4.0 Sharedv4 Service with failover mechanism
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: portworx-sharedv4-service
provisioner: pxd.portworx.com
parameters:
  repl: "2"
  io_profile: "auto"
  priority_io: "high"
  nodiscard: "true"
  autofstrim: "true"
  disable_io_profile_protection: "1"
  sharedv4: "true"
  sharedv4_svc_type: "ClusterIP"
reclaimPolicy: Retain
allowVolumeExpansion: true
---
# Defining sharedv4_svc_type: "" will enforce NFS 3.0, but we are moving away from NFS 3.0 on Portworx.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: portworx-sharedv4-normal
provisioner: pxd.portworx.com
parameters:
  repl: "2"
  io_profile: "auto"
  priority_io: "high"
  nodiscard: "true"
  autofstrim: "true"
  disable_io_profile_protection: "1"
  sharedv4_svc_type: ""
  sharedv4: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
---
#ConfigMap to set the default umask of the containers to 0002 (u=rwx,g=rwx,o=rx)
kind: ConfigMap
apiVersion: v1
metadata:
  name: profile
data:
  profile: "umask 0002\n"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rwx-volume
  annotations:
    volume.beta.kubernetes.io/storage-class: portworx-sharedv4-service
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: rwx-volume2
  annotations:
    volume.beta.kubernetes.io/storage-class: portworx-sharedv4-normal
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: px-busybox-test
  name: px-busybox-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: px-busybox-test
  template:
    metadata:
      labels:
        app: px-busybox-test
    spec:
      schedulerName: stork
      securityContext:
        runAsUser: 2000
        runAsGroup: 3000
        fsGroup: 3000
        fsGroupChangePolicy: "OnRootMismatch"
      containers:
      - image: gcr.io/google_containers/busybox
        name: px-busybox-test
        command: ["/bin/sh", "-c"]
        args: [ "tail -f /dev/null" ]
        env:
          - name: "ENV"
            value: "/etc/profile"
          - name: "TEST_VAR"
            value: "This is a test var"
        volumeMounts:
          - name: rwx-volume
            mountPath: /opt/rwxvolume
          - name: rwx-volume2
            mountPath: /opt/rwxvolume2
          - name: profile-volume
            mountPath: /etc/profile
            subPath: profile 
      #serviceAccountName: my-app-sa
      volumes:
        - name: rwx-volume
          persistentVolumeClaim:
            claimName: rwx-volume
        - name: rwx-volume2
          persistentVolumeClaim:
            claimName: rwx-volume2
        - name: profile-volume
          configMap:
            name: profile
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: px-busybox-test-2
  name: px-busybox-test-2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: px-busybox-test-2
  template:
    metadata:
      labels:
        app: px-busybox-test-2
      annotations:
        openshift.io/scc: log-collector-scc
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: "app"
                  operator: In
                  values:
                  - px-busybox-test
            topologyKey: "kubernetes.io/hostname"
      schedulerName: stork
      securityContext:
        runAsUser: 4000
        runAsGroup: 3000
        fsGroup: 3000
        fsGroupChangePolicy: "OnRootMismatch"
      containers:
      - image: gcr.io/google_containers/busybox
        name: px-busybox-test
        command: ["/bin/sh", "-c"]
        args: [ "tail -f /dev/null" ]
        env:
          - name: "ENV"
            value: "/etc/profile"
          - name: "TEST_VAR"
            value: "This is a test var"
        volumeMounts:
          - name: rwx-volume
            mountPath: /opt/rwxvolume
          - name: rwx-volume2
            mountPath: /opt/rwxvolume2
          - name: profile-volume
            mountPath: /etc/profile
            subPath: profile 
      #serviceAccountName: my-app-sa
      volumes:
        - name: rwx-volume
          persistentVolumeClaim:
            claimName: rwx-volume
        - name: rwx-volume2
          persistentVolumeClaim:
            claimName: rwx-volume2
        - name: profile-volume
          configMap:
            name: profile