Score:0

Rsyslog only forwarding some log files to logstash

br flag

I have a client machine which forwards its log files to a central rsyslog server. I can see the logs on the central syslog server here:

root@server:/var/log# ls -al /var/log/client/
total 1952
drwxr-xr-x  2 syslog syslog    4096 Jun 19 07:06 .
drwxrwxr-x 14 root   syslog    4096 Jun 19 10:26 ..
-rw-r-----  1 syslog adm     324895 Jun 19 16:02 auth.log
-rw-r-----  1 syslog adm      31787 Jun 19 15:17 authpriv.log
-rw-r-----  1 syslog adm       3859 Jun 19 15:17 cron.log
-rw-r-----  1 syslog adm      96770 Jun 19 15:38 daemon.log
-rw-r-----  1 syslog adm    1475044 Jun 19 16:03 kern.log
-rw-r-----  1 syslog adm      11019 Jun 19 12:50 local0.log
-rw-r-----  1 syslog adm      15883 Jun 19 12:49 syslog.log
-rw-r-----  1 syslog adm       1688 Jun 19 07:16 user.log

I am most interested in local0.log.

I have configured the central rsyslog server to forward the logs to logstash:

root@server:/etc/rsyslog.d# cat /etc/rsyslog.d/70-output.conf 

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

*.* action(type="omfwd" target="localhost" port="10514" protocol="udp" template="json-template")

Logstash is configured to forward the logs to elasticsearch:

root@server:/etc/rsyslog.d# cat /etc/logstash/conf.d/logstash.conf 
input {                                                                                      
  tcp {                                                                                      
    host => "127.0.0.1"                                                                      
    port => 10514                                                                            
    codec => "json"                                                                          
    type => "rsyslog"                                                                        
  }                                                                                          
}                                                                                            
                                                                                         
filter { }                                                                                   
                                                                                                                                                                                                     
output {                                                                                     
  if [type] == "rsyslog" {                                                                   
    elasticsearch {                                                                          
      hosts => [ "127.0.0.1:9200" ]                                                          
    }                                                                                        
  }                                                                                          
}  

I can see elasticsearch now has logs:

root@server:/etc/rsyslog.d# curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'

{
  "took" : 4,
  "timed_out" : false,
  ...
    "hits" : [
      {
        "_index" : "logstash-2021.06.19-000001",
        "_type" : "_doc",
        "_id" : "3YxQJHoBrZHyFIavQWQZ",
        "_score" : 1.0,
        "_source" : {
          "message" : " [20468.211706] [UFW BLOCK] IN=eth0 OUT= MAC=<snip> SRC=<snip> DST=<snip> LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=15690 PROTO=TCP SPT=43024 DPT=6780 WINDOW=1024 RES=0x00 SYN URGP=0 ",
        ...
      

However, it seems only some of the log files are being forwarded to logstash (or logstash isn't forwarding them to elasticsearch).

Looking at all the logs in elasticsearch (or within kibana) I can see /var/log/client/local0.log is being completely ignored.

Is there a way to force this log file to be included? How can I troubleshoot this?

Thanks!

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.