I have a client machine which forwards its log files to a central rsyslog server. I can see the logs on the central syslog server here:
root@server:/var/log# ls -al /var/log/client/
total 1952
drwxr-xr-x 2 syslog syslog 4096 Jun 19 07:06 .
drwxrwxr-x 14 root syslog 4096 Jun 19 10:26 ..
-rw-r----- 1 syslog adm 324895 Jun 19 16:02 auth.log
-rw-r----- 1 syslog adm 31787 Jun 19 15:17 authpriv.log
-rw-r----- 1 syslog adm 3859 Jun 19 15:17 cron.log
-rw-r----- 1 syslog adm 96770 Jun 19 15:38 daemon.log
-rw-r----- 1 syslog adm 1475044 Jun 19 16:03 kern.log
-rw-r----- 1 syslog adm 11019 Jun 19 12:50 local0.log
-rw-r----- 1 syslog adm 15883 Jun 19 12:49 syslog.log
-rw-r----- 1 syslog adm 1688 Jun 19 07:16 user.log
I am most interested in local0.log.
I have configured the central rsyslog server to forward the logs to logstash:
root@server:/etc/rsyslog.d# cat /etc/rsyslog.d/70-output.conf
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
*.* action(type="omfwd" target="localhost" port="10514" protocol="udp" template="json-template")
Logstash is configured to forward the logs to elasticsearch:
root@server:/etc/rsyslog.d# cat /etc/logstash/conf.d/logstash.conf
input {
tcp {
host => "127.0.0.1"
port => 10514
codec => "json"
type => "rsyslog"
}
}
filter { }
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}
}
I can see elasticsearch now has logs:
root@server:/etc/rsyslog.d# curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'
{
"took" : 4,
"timed_out" : false,
...
"hits" : [
{
"_index" : "logstash-2021.06.19-000001",
"_type" : "_doc",
"_id" : "3YxQJHoBrZHyFIavQWQZ",
"_score" : 1.0,
"_source" : {
"message" : " [20468.211706] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=15690 PROTO=TCP SPT=43024 DPT=6780 WINDOW=1024 RES=0x00 SYN URGP=0 ",
...
However, it seems only some of the log files are being forwarded to logstash (or logstash isn't forwarding them to elasticsearch).
Looking at all the logs in elasticsearch (or within kibana) I can see /var/log/client/local0.log
is being completely ignored.
Is there a way to force this log file to be included? How can I troubleshoot this?
Thanks!