systemd file not pickuping environment values

Systemd file for filebeat doesn't pickup env variables and throw as below

ExecStart=/usr/share/filebeat/bin/filebeat -environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=2)

Below is my systemd for filebeat service

 [Unit]
Description=Filebeat sends log files to Logstash or directly to Elasticsearch.
Documentation=https://www.elastic.co/products/beats/filebeat
Wants=network-online.target
After=network-online.target

[Service]

Environment="BEAT_LOG_OPTS="
Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
Environment="BEAT_PATH_OPTS=-path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"
ExecStart=/usr/share/filebeat/bin/filebeat -environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always

[Install]
WantedBy=multi-user.target

As mentioned in filebeat documentation, -environment is not used for specifying environment variables, but to specify the environment the Filebeat is running. Moreover, the BEAT_PATH_OPTS is available to filebeat process, ant not available to systemd, where you are trying to use with $BEAT_LOG_OPTS. The systemd service file is not a shell script. See man systemd.service for details.

-environment

For logging purposes, specifies the environment that Filebeat is running in. This setting is used to select a default log output when no log output is configured. Supported values are: systemd, container, macos_service, and windows_service. If systemd or container is specified, Filebeat will log to stdout and stderr by default.

Those values are not shell environment variables, but they may be read from the service unit's supplementary files from the "drop-in" directory. This is essentially a directory with a .d suffix, so if your service unit is /etc/systemd/system/filebeat.service, then you can have a file like /etc/systemd/system/filebeat.service.d/filebeat.service.conf (the filename can be anything, just keep the .conf extension).

Then your filebeat.service.conf would look like so:

[Service]
Environment="BEAT_LOG_OPTS="
Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
Environment="BEAT_PATH_OPTS=-path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"

So in a nutshell, you can move certain stuff out from the unit file, so it can remain unchanged, while the .conf files can be changed as needed.

And that's about it. systemd units aren't supposed to read environment variables. Those variables are limited to your shell.

Of course, you can set up all kinds of scripting to adjust these .conf files as needed, if you want, just be sure to call systemctl daemon-reload first. Until then, systemd will not re-read those values. Certain services uses other quirks that read variables from files in under /etc/sysconfig or other dirs, but the concept is the same. It's just a separation of the variables (i.e. the configuration) and the unit itself into different files.