I have a number of boxes that run docker containers. To keep a record of all the logs, we have 1 instance of logspout on every box that grabs all container logs on that box, and routes it to a syslog-ng instance (docker container) on a central host which stores it to a specific path.
I'm trying to extend this setup to use grafana/loki. Problem is that the __syslog_connection_hostname label always evaluates to "[local host fdqn].,[hostname],localhost.localdomain." no matter where the logs come from. I know syslog-ng knows where the actual hosts are, because the route that we store the logs are is something like: date_underscore/hostname/containername.log
I've tried setting keep-hostname(yes)
in syslog-ng, no dice. Not sure what I'm missing.
syslog-ng conf
@version: 3.37
@include "scl.conf"
options {
dir-perm(0755);
keep-hostname(yes);
};
source s_network {
default-network-drivers();
};
destination d_local {
file("/logs/${YEAR}_${MONTH}_${DAY}/${HOST_FROM}/${PROGRAM}.log" perm(0755) create_dirs(yes));
};
destination d_loki {
syslog("localhost" transport("tcp") port(1514));
};
log {
source(s_network);
destination(d_local);
destination(d_loki);
};
promtail conf snippet
- job_name: syslog
syslog:
listen_address: 0.0.0.0:1514
idle_timeout: 60s
label_structured_data: yes
labels:
job: "syslog"
relabel_configs:
- source_labels: ['__syslog_message_hostname']
target_label: 'container_name'
- source_labels: ['__syslog_connection_hostname']
target_label: 'hostname'
- source_labels: ['__syslog_connection_ip_address']
target_label: 'ip'
- source_labels: ['__syslog_message_severity']
target_label: 'severity'