Score:0

nfs (or ceph, or k8s) works strange: ls/tree not working, but i can read files

es flag

I have some problems and i want to know where i have to check.

  • my env: ceph nfs mount to a k8s pod

  • problem:

    • at mounted path (on k8s node and inside the pod too), ls/tree not working, but i can read/write files

    • because i can read files and only can't see in filesystem, so i thought inode is weird.

    • mounted volume has negative IUsed and increasing Inodes when i create/modify files, which i don't know why.

    • here is the command/result and my mount option(on the machine)

      sh-4.2# ls -al
      total 0
      drwx--x--x   1  root root 7 6 02:45 .
      drwxrwxrwx 216  root root 7 6 02:45 ..
      sh-4.2# 
      sh-4.2# cd 03
      sh-4.2# 
      sh-4.2# ls -al
      total 0
      drwxr-xr-x   1  root root 7 6 02:45 .
      drwx--x--x 216  root root 7 6 02:45 ..
      sh-4.2# 
      sh-4.2# cat test
      cat: test: No such file or directory
      sh-4.2# 
      sh-4.2# echo "FILE EXISTS" > test
      sh-4.2# 
      sh-4.2# ls -al
      total 0
      drwxr-xr-x   1  root root 7 6 02:45 .
      drwx--x--x 216  root root 7 6 02:45 ..
      sh-4.2# 
      sh-4.2# cat test
      FILE EXISTS
      sh-4.2# 
      sh-4.2# df -iH --type=nfs4
      Filysystem                  Inodes  IUsed  IFree  IUse%  Mounted on 
      some.ceph.url.com/path1        657  -1.7G   1.7G      -  /data0/kubelet/pods/**/volumes/kubernetes.io~nfs/path1
      sh-4.2#
      sh-4.2# cat /etc/mtab | grep nfs4
      some.ceph.url.com/path1 /data0/kubelet/pods/**/volumes/kubernetes.io~nfs/**** 
      nfs4 rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,
      timeo=600,retrans=2,sec=sys,clientaddr=172.30.*.*,local_lock=none,addr=172.30.*.* 0 0
      

Thank you in advance. where can i find the root cause or searching keyword?

Vit avatar
cn flag
Vit
the way and place you installed kubernetes?
whojes avatar
es flag
centos 7 on premise
Vit avatar
cn flag
Vit
seems like kind of FS corruption. Have you run fsck tool to check logs and issues?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.