Overview
I have a server that receives Syslog data from external clients, and I do not have administrative access to these clients. The goal is to move the existing configuration to data in transit encryption by implementing TLS for Syslog on TCP port 6514. All networking, firewall rules, and SELinux configurations have been verified and I have successfully tested receiving non-encrypted Syslog messages over TCP port 6514. I have also downloaded rsyslog-gnutls as the TLS driver.
Syslog Server Details
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
$ rsyslogd -v
rsyslogd 8.24.0-57.el7_9.3, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
yum list installed | grep rsyslog-gnutls
rsyslog-gnutls.x86_64 8.24.0-57.el7_9.3
Base Syslog Config (/etc/rsyslog.conf)
This remains untouched from the previously working non-encrypted configuration over port 514/tcp, but included here to be complete.
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
$FileCreateMode 0640
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none /var/log/messages
*.* /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log all user comands
local3.* /var/log/userCommands.log
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
IncludeConfig (/etc/rsyslog.d/syslog-tls.conf)
This is the file I created with custom configurations. The config below shows the current working state without encryption on port 6514/tcp. This is the only .conf file present in /etc/rsyslog.d so no other configs should be getting imported. There were three other .conf files present that I renamed to .conf.bak so my assumption is rsyslogd will not read those due to the added extension.
########################################################################
#
# This file is included from /etc/rsyslog.conf as long as it is located
# in /etc/rsyslog.d/.
#
########################################################################
$umask 0000 #not supported in global() for rsyslog versions < 8.26
global(
preserveFQDN="on"
parser.escapeControlCharactersOnReceive="off" #Prevent escaping of new lines
#defaultNetstreamDriver="gtls"
#defaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/myCertAuthCertificate.pem"
#defaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/mySyslogUfServerChainedCertificate.pem"
#defaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/mySyslogUfServerPrivKey.key"
)
module(
load="imtcp"
maxSessions="500"
disableLFDelimiter="off"
#streamDriver.Name="gtls"
#streamDriver.Mode="1"
#streamDriver.Authmode="x509/certvalid"
#streamDriver.PermittedPeers="172.16.32.155"
)
module(
load="builtin:omfile"
dirCreateMode="0750"
dirOwner="splunk"
dirGroup="splunk"
fileCreateMode="0640"
fileOwner="splunk"
fileGroup="splunk"
)
template(name="foo_test" type="string" string="/var/log/foo_test/%HOSTNAME%/%$year%_%$month%_%$day%.log")
ruleset(name="foo"){
if ($fromhost-ip != '127.0.0.1') then
action(type="omfile" dynaFile="foo_test")
stop
}
input(type="imtcp" port="6514")
Issue
If I uncomment the TLS related configuration parameters (DefaultNetstreamDriver and StreamDriver) in syslog-tcp.conf and restart rsyslog, port 6514/tcp does not show up under netstat anymore. This tells me that something is wrong with the configuration parameters chosen and/or the order in which I have defined them.
Running syntax validation checks on the config files shows no issues, but from my research this only indicates syntax is good and there could still be issues at runtime.
$ sudo systemctl restart rsyslog
$ sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-08-09 18:46:52 GMT; 3s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 22327 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─22327 /usr/sbin/rsyslogd -n
Aug 09 18:46:52 testbox01 systemd[1]: Starting System Logging Service...
Aug 09 18:46:52 testbox01 systemd[1]: Started System Logging Service.
$ rsyslogd -f /etc/rsyslog.conf -N1
rsyslogd: version 8.24.0-57.el7_9.3, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
$ rsyslogd -f /etc/rsyslog.d/syslog-tls.conf -N3
rsyslogd: version 8.24.0-57.el7_9.3, config validation run (level 3), master config /etc/rsyslog.d/syslog-tls.conf
rsyslogd: End of config validation run. Bye.
This is my first stab at rsyslog configuration with TLS, and prior to this I did not have much rsyslog experience in general. I'm hoping someone more versed in rsyslog can show me the errors of my ways :) Thanks in advance!
UPDATES
8/9/23: I ran rsyslogd in debug mode and got a lot of output. Truncated version of output is provided below with messages that I thought were relevant to the issue. I can provide more output if needed. I noticed that port 6514/tcp came up as a listening port when rsyslogd is run in foreground debug mode. As soon as I kill debug mode and restart rsyslogd in normal fashion port 6514/tcp stays down, i.e. does not show up in LISTEN state in netstat output.
3144.701090845:main thread : imtcp: trying to add port *:6514
3144.701096552:main thread : ratelimit:tcperver:new ratelimiter:bReduceRepeatMsgs 0
3144.701103144:main thread : caller requested object 'nsd_gtls', not found (iRet -3003)
3144.701107289:main thread : Requested to load module 'lmnsd_gtls'
3144.701112377:main thread : loading module '/usr/lib64/rsyslog/lmnsd_gtls.so'
3144.704593252:main thread : source file nsd_gtls.c requested reference for module 'lmnet', reference count now 6
3144.704601962:main thread : caller requested object 'nsd_ptcp', not found (iRet -3003)
3144.704606370:main thread : Requested to load module 'lmnsd_ptcp'
3144.704613121:main thread : loading module '/usr/lib64/rsyslog/lmnsd_ptcp.so'
3144.704740556:main thread : source file nsd_ptcp.c requested reference for module 'lmnetstrms', reference count now 4
3144.704750825:main thread : module lmnsd_ptcp of type 2 being loaded (keepType=0).
3144.704754868:main thread : entry point 'isCompatibleWithFeature' not present in module
3144.704758844:main thread : entry point 'setModCnf' not present in module
3144.704762675:main thread : entry point 'getModCnfName' not present in module
3144.704766304:main thread : entry point 'beginCnfLoad' not present in module
3144.704770744:main thread : source file nsd_gtls.c requested reference for module 'lmnsd_ptcp', reference count now 1
3144.704785499:main thread : GTLS CA file: '/etc/rsyslog.d/certs/myCertAuthCertificate.pem'
3144.705337400:main thread : source file nsdsel_gtls.c requested reference for module 'lmnsd_ptcp', reference count now 2
3144.705344715:main thread : module lmnsd_gtls of type 2 being loaded (keepType=1).
3144.705348930:main thread : entry point 'isCompatibleWithFeature' not present in module
3144.705352660:main thread : entry point 'setModCnf' not present in module
3144.705356421:main thread : entry point 'getModCnfName' not present in module
3144.705360038:main thread : entry point 'beginCnfLoad' not present in module
3144.705365118:main thread : source file netstrms.c requested reference for module 'lmnsd_gtls', reference count now 1
3144.705373376:main thread : GTLS certificate file: '/etc/rsyslog.d/certs/mySyslogUfServerChainedCertificate.pem'
3144.705377299:main thread : GTLS key file: '/etc/rsyslog.d/certs/mySyslogUfServerPrivKey.key'
3144.716630033:main thread : creating tcp listen socket on port 6514
3144.723178541:main thread : We could initialize 1 TCP listen sockets out of 2 we received - this may or may not be an error indication.
3144.723190317:main thread : Allocating buffer for 500 TCP sessions.
3144.723201696:main thread : telling modules to activate config 0x55b2e2beaa20
3144.723206050:main thread : activating config 0x55b2e2beaa20 for module builtin:omfile
3144.723210213:main thread : activating config 0x55b2e2beaa20 for module builtin:ompipe
3144.723214794:main thread : activating config 0x55b2e2beaa20 for module builtin:omfwd
3144.723219318:main thread : activating config 0x55b2e2beaa20 for module imuxsock
3144.723223556:main thread : activating config 0x55b2e2beaa20 for module imjournal
3144.723227739:main thread : activating config 0x55b2e2beaa20 for module imtcp
3144.724453266:main thread : Allowed TCP Senders:
3144.724459171:main thread : No restrictions set.
3144.724465166:main thread : iterateAllActions calling into action 0x55b2e2c03ec0
3144.724472897:main thread : action 1 queue: starting queue
8/10/23: I added the SYSLOGD_OPTIONS="-d" option to /etc/sysconfig/rsyslog as shown below. Any issues with this being deprecated since v3 and not running in the compatibility mode it specifies in the file comments?
# Options for rsyslogd
# Syslogd options are deprecated since rsyslog v3.
# If you want to use them, switch to compatibility mode 2 by "-c 2"
# See rsyslogd(8) for more details
SYSLOGD_OPTIONS="-d"
I am now seeing error messages upon startup of rsyslog as shown below.
$ sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-08-10 13:21:36 GMT; 6s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 9362 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─9362 /usr/sbin/rsyslogd -n -d
Aug 10 13:21:36 testbox01 systemd[1]: Starting System Logging Service...
Aug 10 13:21:36 testbox01 rsyslogd[9362]: [origin software="rsyslogd" swVersion="8.24.0-57.el7_9.3" x-pid="9362" x-info="http://www.rsyslog.com"] start
Aug 10 13:21:36 testbox01 rsyslogd[9362]: error reading certificate file '/etc/rsyslog.d/certs/myCertAuthCertificate.pem' - a common cause is that the file does not ex...com/e/2078 ]
Aug 10 13:21:36 testbox01 rsyslogd[9362]: could not load module '/usr/lib64/rsyslog/lmnsd_gtls.so', rsyslog error -2078 [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2068 ]
Aug 10 13:21:36 testbox01 systemd[1]: Started System Logging Service.
Aug 10 13:21:36 testbox01 rsyslogd[9362]: tcpsrv could not create listener (inputname: 'imtcp') [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2068 ]
Aug 10 13:21:36 testbox01 rsyslogd[9362]: activation of module imtcp failed [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2068 ]
Hint: Some lines were ellipsized, use -l to show in full.
Setting SELinux to Permissive mode allowed rsyslog to start successfully without the errors present. Looking into possible resolution so that SELinux can remain enforced.
sudo setenforce 0
