I have installed FluentD on Kind-Kubernetes cluster on CentOS VM running on my laptop. I'm having issues getting FluentD to read logs as it is throwing the following error.
2021-08-29 08:26:31 +0000 [warn]: #0 [tail_container_logs] /var/log/containers/myapp-77df6bfff9-jcnwc_default_myapp-4d82556157a94e991f011bac956d182e941a122b40b3d53fc67dfd6f39aef5d4.log unreadable. It is excluded and would be examined next time.
Below is my fluent.conf file.
<system>
@log_level debug
</system>
<source>
@type tail
@id tail_container_logs
path /var/log/containers/*.log
pos_file /var/fluent/log/containers.log.pos
#time_format %Y-%m-%dT%H:%M:%S
tag kubernetes.*
format json_in_json
read_from_head true
</source>
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<match kubernetes.**>
@type file
@id output1
path /var/fluent/log/log/data.*.log
append true
#time_slice_format %Y%m%d
#time_slice_wait 10m
#time_format %Y%m%dT%H%M%S%z
</match>
In the deployment YAML file I have the following configuration.
spec:
containers:
- name: my-fluentd
image: custom-fluentd
imagePullPolicy: Never
resources:
limits:
memory: 800Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
- name: varfluent
mountPath: /var/fluent/log
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varfluent
hostPath:
path: /var/fluent/logs
I have read in several threads related to similar issue but none of them seems to be particularly solving the issue I got.
When I had a look at the log files in /var/log/containers
on the FluentD pod, I can see they are symlink to /var/log/pods
folder. Here is an sample output of the directory listing within '/var/log/containers' folder.
lrwxrwxrwx 1 root root 93 Aug 29 05:02 myapp-77df6bfff9-jcnwc_default_myapp-4d82556157a94e991f011bac956d182e941a122b40b3d53fc67dfd6f39aef5d4.log -> /var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp/3.log
When looking at the file permission of the log file in FluentD pod, this is what I can see;
/var/log/pods # ls -ltr
total 0
drwxr-xr-x 3 root root 28 Aug 27 15:29 kube-system_kube-scheduler-kind-control-plane_69dd939498054a211c3461b2a9cc8d26
drwxr-xr-x 3 root root 37 Aug 27 15:29 kube-system_kube-controller-manager-kind-control-plane_46dac9a538838115821dfd9559149484
drwxr-xr-x 3 root root 28 Aug 27 15:29 kube-system_kube-apiserver-kind-control-plane_bd1c21fe1f0ef615e0b5e41299f1be61
drwxr-xr-x 3 root root 18 Aug 27 15:29 kube-system_etcd-kind-control-plane_24ba8551bcc724a32d591bb02c423d92
drwxr-xr-x 3 root root 24 Aug 27 15:30 kube-system_kube-proxy-hqdkp_6ea129ba-1b2e-425e-a77b-ad75dacc4cda
drwxr-xr-x 3 root root 25 Aug 27 15:30 kube-system_kindnet-dsjr6_32cc1367-ce01-481d-b626-58c00ea9aa6c
drwxr-xr-x 3 root root 36 Aug 27 15:30 local-path-storage_local-path-provisioner-547f784dff-5rj88_fc2f27cf-3779-4cfb-a517-00f4bf12ee0c
drwxr-xr-x 3 root root 21 Aug 27 15:30 kube-system_coredns-558bd4d5db-lrzf4_6c526c83-f140-4e14-839c-ce00fa275890
drwxr-xr-x 3 root root 21 Aug 27 15:30 kube-system_coredns-558bd4d5db-bxp22_5b9836f4-66e0-42e4-a999-2428e910a557
drwxr-xr-x 3 root root 19 Aug 27 15:34 default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a
drwxr-xr-x 3 root root 24 Aug 29 23:47 default_my-fluentd-t4bq6_08921212-3c02-478c-83bd-4f9aad37b9cf
/var/log/pods #
/var/log/pods #
/var/log/pods # cd default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a # ls -ltr
total 0
drwxr-xr-x 2 root root 32 Aug 29 05:02 myapp
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a # cd myapp/
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp # ls -ltr
total 772
-rw-r----- 1 root root 2974 Aug 29 05:02 2.log
-rw-r----- 1 root root 729669 Aug 29 23:49 3.log
/var/log/pods/default_myapp-77df6bfff9-jcnwc_23e368b2-0df8-49bd-b7d9-9af96a57626a/myapp
As you can see the log files i.e. 2.log, 3.log
has no read permission for others.
This is inherited from the host which the files are created. Therefore it seems to be an issue with permissions set by Docker logging driver when creating files.
I would like to know how to enable read permission for files created by the containers.
Appreciate if you could assist.