Score:0

How to enable read permission on Kubernetes mount path

vn flag

I have installed FluentD on Kind-Kubernetes cluster on CentOS VM running on my laptop. I'm having issues getting FluentD to read logs as it is throwing the following error.

2021-08-29 08:26:31 +0000 [warn]: #0 [tail_container_logs] /var/log/containers/myapp-77df6bfff9-jcnwc_default_myapp-4d82556157a94e991f011bac956d182e941a122b40b3d53fc67dfd6f39aef5d4.log unreadable. It is excluded and would be examined next time.

Below is my fluent.conf file.

<system>
  @log_level debug
</system>

<source>
  @type tail
  @id tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/fluent/log/containers.log.pos
  #time_format %Y-%m-%dT%H:%M:%S
  tag kubernetes.*
  format json_in_json
  read_from_head true
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.**>
  @type file
  @id   output1
  path         /var/fluent/log/log/data.*.log
  append       true
  #time_slice_format %Y%m%d
  #time_slice_wait   10m
  #time_format       %Y%m%dT%H%M%S%z
</match>

In the deployment YAML file I have the following configuration.

spec:
  containers:
  - name: my-fluentd
    image: custom-fluentd
    imagePullPolicy: Never
    resources:
      limits:
        memory: 800Mi
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlogpods
      mountPath: /var/log/pods
      readOnly: true
    - name: varfluent
      mountPath: /var/fluent/log
  terminationGracePeriodSeconds: 30
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlogpods
    hostPath:
      path: /var/log/pods
  - name: varfluent
    hostPath:
      path: /var/fluent/logs

I have read in several threads related to similar issue but none of them seems to be particularly solving the issue I got.

When I had a look at the log files in /var/log/containers on the FluentD pod, I can see they are symlink to /var/log/pods folder. Here is an sample output of the directory listing within '/var/log/containers' folder.

lrwxrwxrwx    1 root     root            93 Aug 29 05:02 myapp-77df6bfff9-jcnwc_default_myapp-4d82556157a94e991f011bac956d182e941a122b40b3d53fc67dfd6f39aef5d4.log -> /var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp/3.log

When looking at the file permission of the log file in FluentD pod, this is what I can see;

/var/log/pods # ls -ltr
total 0
drwxr-xr-x    3 root     root            28 Aug 27 15:29 kube-system_kube-scheduler-kind-control-plane_69dd939498054a211c3461b2a9cc8d26
drwxr-xr-x    3 root     root            37 Aug 27 15:29 kube-system_kube-controller-manager-kind-control-plane_46dac9a538838115821dfd9559149484
drwxr-xr-x    3 root     root            28 Aug 27 15:29 kube-system_kube-apiserver-kind-control-plane_bd1c21fe1f0ef615e0b5e41299f1be61
drwxr-xr-x    3 root     root            18 Aug 27 15:29 kube-system_etcd-kind-control-plane_24ba8551bcc724a32d591bb02c423d92
drwxr-xr-x    3 root     root            24 Aug 27 15:30 kube-system_kube-proxy-hqdkp_6ea129ba-1b2e-425e-a77b-ad75dacc4cda
drwxr-xr-x    3 root     root            25 Aug 27 15:30 kube-system_kindnet-dsjr6_32cc1367-ce01-481d-b626-58c00ea9aa6c
drwxr-xr-x    3 root     root            36 Aug 27 15:30 local-path-storage_local-path-provisioner-547f784dff-5rj88_fc2f27cf-3779-4cfb-a517-00f4bf12ee0c
drwxr-xr-x    3 root     root            21 Aug 27 15:30 kube-system_coredns-558bd4d5db-lrzf4_6c526c83-f140-4e14-839c-ce00fa275890
drwxr-xr-x    3 root     root            21 Aug 27 15:30 kube-system_coredns-558bd4d5db-bxp22_5b9836f4-66e0-42e4-a999-2428e910a557
drwxr-xr-x    3 root     root            19 Aug 27 15:34 default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a
drwxr-xr-x    3 root     root            24 Aug 29 23:47 default_my-fluentd-t4bq6_08921212-3c02-478c-83bd-4f9aad37b9cf
/var/log/pods # 
/var/log/pods # 
/var/log/pods # cd default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a # ls -ltr
total 0
drwxr-xr-x    2 root     root            32 Aug 29 05:02 myapp
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a # cd myapp/
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp # ls -ltr
total 772
-rw-r-----    1 root     root          2974 Aug 29 05:02 2.log
-rw-r-----    1 root     root        729669 Aug 29 23:49 3.log
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp

As you can see the log files i.e. 2.log, 3.log has no read permission for others.

This is inherited from the host which the files are created. Therefore it seems to be an issue with permissions set by Docker logging driver when creating files.

I would like to know how to enable read permission for files created by the containers. Appreciate if you could assist.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.