Stop filebeat sending copius metadata

Damian Games

6/22/24, 4:35 PM

I am sending data from local log files with filebeat to graylog and I am getting a 20x storage overhead compared to the original files. There are a large amount of metadata fields however I can't seem to get rid of them. I have tried many variations of removing fields such as:

processors:
  - drop_fields:
      fields: ["ecs.version", "agent.version", "agent.type", "agent.id", "agent.hostname", "input.type"]

Do any of you have any recommendation of how to strip everthing except the timestamp and raw log that has been sent? I do not need anything like id or agent type because they are all coming from the same place.

1 + 1

metadata

elk

filebeat

graylog

HBruijn

6/22/24, 5:59 PM

It looks like your trying to remove those fields in filebeat/journalbeat ; rather than getting the agent to go against its intended behaviour the trick is probably to drop unwanted fields in Graylog instead like https://go2docs.graylog.org/5-0/making_sense_of_your_log_data/functions_descriptions.html#removefield and/or the old message https://community.graylog.org/t/how-to-add-remove-additional-fields-in-graylog/9445

Score:1

Server

Willman

8/10/24, 8:53 PM

There are 2 metadata fields required by Graylog in order for Sidecar to function properly: collector_node_id and node.name, so you can't remove those.

Filebeat itself prevents removal of the @timestamp field.

And the @metadata_* fields are set by Logstash and cannot be removed.

Other than this, you can remove all other Filebeat fields. Here's a config snippet you can use to do so in your filebeat.yml:

processors:
  - drop_fields:
      fields: [
          "agent.ephemeral_id",
          "agent.id",
          "agent.type",
          "agent.version",
          "ecs.version",
          "host.name",
          "input.type",
          "log.file.path",
          "log.offset"
      ]

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Stop filebeat sending copius metadata

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.