Kubernetes Statefulset Not Finding PVC

In Kubernetes if your statefulset does not start up and after describing it you get the error that the persisted volume claim (PVC) was not found it is likely due to the volumeClaimTemplates name being different from the name used by at least one volume mount in the volumeMounts section of the statefulset spec.

I will modify the statefulset example from the official docs to demonstrate. The below modified example will cause the PVC not found error:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: 'nginx'
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: bar
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: foo
      spec:
        accessModes: ['ReadWriteOnce']
        storageClassName: 'my-storage-class'
        resources:
          requests:
            storage: 1Gi

In the above you can see that volumeClaimTemplates[0].metadata.name is set to foo. Now if you look further up under spec.volumeClaimTemplates[0].metadata.name we see that is set to foo. These 2 values need to be the same which is not 100% clear. I eventually found this spelled out clearly in the API reference documentation over here under the volumeClaimTemplates description:

volumeClaimTemplates is a list of claims that pods are allowed to reference. The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template, with the same name.

To fix the above YAML we would update either spec.volumeClaimTemplates[0].metadata.name or volumeClaimTemplates[0].metadata.name to be the same or add another volume mount with the same name spec.volumeClaimTemplates[0].metadata.name:

Option 1 - Using the same name

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: 'nginx'
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: foo
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: foo
      spec:
        accessModes: ['ReadWriteOnce']
        storageClassName: 'my-storage-class'
        resources:
          requests:
            storage: 1Gi

Option 2 - Adding another volumeMount with the same name

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: 'nginx'
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
              name: web
          volumeMounts:
            - name: bar #this mount path has to be unique
              mountPath: /usr/bar/nginx/html
            - name: foo
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: foo
      spec:
        accessModes: ['ReadWriteOnce']
        storageClassName: 'my-storage-class'
        resources:
          requests:
            storage: 1Gi