Score:0

Fluentd unable to parse Json logs generated from node pm2 server

us flag

I am using fluentd, Elasticsearch and Grafana stack for logging and monitoring setup. I am tailing the logs(in JSON format) coming from NodeJS container/pod deployed in EKS cluster and dumping into Elasticsearch.

Problem The log format coming in stdout from node(deployed using pm2 web server) is not accurate json for example:

10:37:01 0|server | {"timestamp":"*","level":"*","label":"*","user":"*","message":"*","RequestMethod":"*","RequestPath":"*","StatusCode":*,"MachineName":"V","EnvironmentName":"*","Appplication":"*","CorrelationId":"*"}

As you can see above that unnecessary data 10:37:01 0|server | is coming outside json {} so as a result while parsing the logs using filter directive its throwing the error and saying "pattern not matched " error

Goal My logs should be fetched from stdout via fluentd and dumped into Elasticsearch hence I need to fix the logs structure just to make sure my logs should look like below:

{"timestamp":"*","level":"*","label":"*","user":"*","message":"*","RequestMethod":"*","RequestPath":"*","StatusCode":*,"MachineName":"V","EnvironmentName":"*","Appplication":"*","CorrelationId":"*"}

Which means no data 10:37:01 0|server | coming outside Json {} body. FYI I also checked at the pm2 server whether if we can remove this entry in the logs but couldn't find anything on it. Any methods that could fix my problem will be great.

yash avatar
us flag
Anyone please help ?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.