%HOSTNAME% doesn't get injected inside of all logs that are received. They are coming from the same IP address and most of them get marked. However, some of them just don't get a host name in front of them. In this environment there's no DNS from the interface these logs are coming from. To deal with this, I've inserted the name and IP of the sender into the hosts file of the OS and this works most of the time. Below is an example of a log with a host name and without.
2022-02-16T11:11:36.093115-07:00 IQM_TYPE_SERVER 10.0.0.41
2022-02-16T11:11:36.094688-07:00 lxdenp1bigip01.example.com iqmgmt_ssl_connect: SSL error: SSL connect (1)
Here's the custom additions to the config, everything else is default.
Rsyslogd version is 8.2001.0
module(load="imudp")
input(type="imudp" port="514" ruleset="datadog")
ruleset(name="datadog"){
action(type="omfile" file="/var/log/dd_stage")
$template DatadogFormat,"redacted <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% %msg%\n"
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.logs.datadoghq.com
*.* @@intake.logs.datadoghq.com:443;DatadogFormat
}
$ModLoad immark
$MarkMessagePeriod 20