Score:0

Kubernetes volumeMount: can create/delete but can't read/write files

cn flag

We've faced weird issue with one of volumeMounts of one of our deployments. Here are relevant manifests' snippets:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: xxx-media-pv-resource-claim
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: cephfs
...
apiVersion: extensions/v1beta1
kind: Deployment
...
        volumeMounts:
        - mountPath: /opt/media/logs
          name: xxx-media-log-storage
        - mountPath: /shared-media-files
          name: xxx-media-resource-storage
...
      volumes:
      - name: xxx-media-log-storage
        persistentVolumeClaim:
          claimName: xxx-media-pv-log-claim
      - name: xxx-media-resource-storage
        persistentVolumeClaim:
          claimName: xxx-media-pv-resource-claim
...

We get read/write error: "Operation not permitted" when trying to, e.g. cat/echo from/to files in /shared-media-files although: a) can successfully touch/mkdir/chmod/rm there; b) can do everything in /opt/media/logs which is backed up by identical PVC (except name). Cephfs storageclass and corresponding Ceph cluster is a storage for whole k8s cluster and all k8s entities that use it except xxx-media-resource-storage behave as expected.

k8s master: v1.14.1
k8s workers: v1.16.0
cephfs-provisioner: v2.1.0-k8s1.11
Ceph: 14.2.1

We checked everything obvious like Ceph cluster health, free storage space, file permissions and ACLs, mount mode, PV/PVC descriptions, logs and didn't find anything special. As xxx-media-pv-resource-claim PVC accessMode is RWX, we tried to use it with another test pod and got same errors. We tried to google the issue as well but with no luck.

Please help to troubleshoot the issue.

Score:1
no flag

This is not supported:

k8s master: v1.14.1 
k8s workers: v1.16.0

https://kubernetes.io/releases/version-skew-policy/

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.