Join Log in
Score:2
Croviajo avatar Server

rsyslog is not forwarding logs to elasticsearch

in flag
Croviajo

I'm trying to configure rsyslog to send logs to logstash and then forward them to elasticsearch.

I have create a config file /etc/rsyslog.d/60-output.conf with the following content:

*.* @localhost:10514;json-template

and a template file /etc/rsyslog.d/01-json-template.conf with the following content:

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

Then I restarted rsyslog service. And for logstash I created a config file /etc/logstash/conf.d/logstash.conf with the following content:

input {
  udp {
    port => 10514
    codec => "json"
    type => "rsyslog"
  }
}
filter { }
output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "localhost:9200" ]
    }
  }
}

Then I restarted logstash.

When I run sudo netstat -tulpn | grep 10514 I get this:

user@rsyslog-server:/var/log$ sudo netstat -tulpn | grep 10514
udp        0      0 0.0.0.0:10514           0.0.0.0:*                           5327/java

so Logstash is listening on port 10514.

To verify the elasticsearch input I run curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty' but this doesn't return any results:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

I'm using:

  • rsyslogd 8.2208.0 (aka 2022.08)
  • logstash 7.17.8
  • elastisearch 7.17.8

How can I solve this ?

Edit:

I run logstash as following to see the logs:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --verbose

and this is the output:

sing bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-01-01 18:50:48.417 [main] runner - DEPRECATION WARNING: The flag ["--verbose"] has been deprecated, please use "--log.level=info" instead.
[INFO ] 2023-01-01 18:50:48.423 [main] runner - Starting Logstash {"logstash.version"=>"7.17.8", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.17+8 on 11.0.17+8 +indy +jit [linux-x86_64]"}
[INFO ] 2023-01-01 18:50:48.426 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2023-01-01 18:50:48.689 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2023-01-01 18:50:49.682 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2023-01-01 18:50:50.739 [Converge PipelineAction::Create<main>] Reflections - Reflections took 50 ms to scan 1 urls, producing 119 keys and 419 values 
[WARN ] 2023-01-01 18:50:51.350 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.396 [Converge PipelineAction::Create<main>] udp - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.438 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:51.479 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:51.658 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[INFO ] 2023-01-01 18:50:51.953 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2023-01-01 18:50:52.174 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2023-01-01 18:50:52.199 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.17.8) {:es_version=>7}
[WARN ] 2023-01-01 18:50:52.200 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2023-01-01 18:50:52.276 [[main]-pipeline-manager] elasticsearch - Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[INFO ] 2023-01-01 18:50:52.362 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x332001c0 run>"}
[INFO ] 2023-01-01 18:50:52.380 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2023-01-01 18:50:53.026 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.66}
[INFO ] 2023-01-01 18:50:53.093 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[WARN ] 2023-01-01 18:50:53.158 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2023-01-01 18:50:53.186 [[main]<udp] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2023-01-01 18:50:53.188 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2023-01-01 18:50:53.206 [[main]<udp] udp - Starting UDP listener {:address=>"0.0.0.0:10514"}
[INFO ] 2023-01-01 18:50:53.244 [[main]<udp] udp - UDP listener started {:address=>"0.0.0.0:10514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
^C[WARN ] 2023-01-01 19:02:34.721 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2023-01-01 19:02:35.721 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2023-01-01 19:02:35.798 [Converge PipelineAction::StopAndDelete<main>] pipelinesregistry - Removed pipeline from registry successfully {:pipeline_id=>:main}
[INFO ] 2023-01-01 19:02:35.873 [LogStash::Runner] runner - Logstash shut down.

sudo journalctl -u logstash -f :

janv. 01 18:50:32 rsyslog-server systemd[1]: logstash.service: Deactivated successfully.
janv. 01 18:50:32 rsyslog-server systemd[1]: Stopped logstash.
janv. 01 18:50:32 rsyslog-server systemd[1]: logstash.service: Consumed 1min 31.847s CPU time.
janv. 01 19:04:10 rsyslog-server systemd[1]: Started logstash.
janv. 01 19:04:10 rsyslog-server logstash[16111]: Using bundled JDK: /usr/share/logstash/jdk
janv. 01 19:04:10 rsyslog-server logstash[16111]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
janv. 01 19:04:22 rsyslog-server logstash[16111]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,731][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,740][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.8", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.17+8 on 11.0.17+8 +indy +jit [linux-x86_64]"}
janv. 01 19:04:22 rsyslog-server logstash[16111]: [2023-01-01T19:04:22,741][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
janv. 01 19:04:24 rsyslog-server logstash[16111]: [2023-01-01T19:04:24,002][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
janv. 01 19:04:24 rsyslog-server logstash[16111]: [2023-01-01T19:04:24,940][INFO ][org.reflections.Reflections] Reflections took 56 ms to scan 1 urls, producing 119 keys and 419 values
janv. 01 19:04:25 rsyslog-server logstash[16111]: [2023-01-01T19:04:25,731][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
janv. 01 19:04:25 rsyslog-server logstash[16111]: [2023-01-01T19:04:25,965][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,138][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,148][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.8) {:es_version=>7}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,153][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,206][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,256][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,276][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x53d2107b run>"}
janv. 01 19:04:26 rsyslog-server logstash[16111]: [2023-01-01T19:04:26,466][INFO ][logstash.outputs.elasticsearch][main] Created rollover alias {:name=>"<logstash-{now/d}-000001>"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,008][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.73}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,069][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,127][INFO ][logstash.inputs.udp      ][main][9e1b2cf1672d2dd22a032525fa695bdf0d4d43163b1fd297f3b99f01f316ae38] Starting UDP listener {:address=>"0.0.0.0:10514"}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,157][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
janv. 01 19:04:27 rsyslog-server logstash[16111]: [2023-01-01T19:04:27,160][INFO ][logstash.inputs.udp      ][main][9e1b2cf1672d2dd22a032525fa695bdf0d4d43163b1fd297f3b99f01f316ae38] UDP listener started {:address=>"0.0.0.0:10514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
^C
490
1 + 2
asktyagi avatar
in flag
asktyagi
Please add logs as well.
0
Reply
Croviajo avatar
in flag
Croviajo
@asktyagi please check my updates
0
Reply
Score:0
user10653633 avatar Server
tw flag
user10653633

what was the issue here? I have an identical problem but I'm using TCP instead of UDP. Thanks!

+ 3
ph flag
Dave M
This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://serverfault.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://serverfault.com/help/whats-reputation), you can also [add a bounty](https://serverfault.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/low-quality-posts/555176)
0
Reply
Quantim avatar
in flag
Quantim
This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://serverfault.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://serverfault.com/help/whats-reputation), you can also [add a bounty](https://serverfault.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/555175)
0
Reply
us flag
Roy
If you have a new question, please ask it by clicking the [Ask Question](https://serverfault.com/questions/ask) button. Include a link to this question if it helps provide context. - [From Review](/review/low-quality-posts/555176)
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.