TLDR;
I'm getting significantly lower throughput when forwarding syslog messages with rsyslog on a Redhat server with 32 cores and 128Gi RAM to a remote server using TCP instead of UDP syslog in the omfwd action.
How can I use TCP for forwarding messages while keeping up with inbound message volume?
Note1: pps = packets per second
Note2: Note the difference in Rx vs Tx stats
When using UDP to forward incoming messages:
[~]$ ./check_network_stats.bash eth0
Rx 10088 pps Tx 10092 pps |--| Rx 7 Mbps Tx 7 Mbps
Rx 11858 pps Tx 11857 pps |--| Rx 8 Mbps Tx 8 Mbps
Rx 11503 pps Tx 11502 pps |--| Rx 8 Mbps Tx 8 Mbps
Rx 11423 pps Tx 11321 pps |--| Rx 8 Mbps Tx 8 Mbps
When using TCP to forward incoming messages:
[~]$ source check_network_stats.bash eth0
Rx 10318 pps Tx 87 pps |--| Rx 7 Mbps Tx 0 Mbps
Rx 12150 pps Tx 162 pps |--| Rx 8 Mbps Tx 0 Mbps
Rx 9504 pps Tx 139 pps |--| Rx 7 Mbps Tx 0 Mbps
Rx 9774 pps Tx 67 pps |--| Rx 6 Mbps Tx 0 Mbps
Rx 12894 pps Tx 159 pps |--| Rx 9 Mbps Tx 0 Mbps
rsyslog.conf:
# rsyslog configuration file
#################
#### MODULES ####
#################
module(load="imjournal" StateFile="imjournal.state")
module(load="imklog") # reads kernel messages (the same are read from journald)
module(load="immark" interval="300") # provides --MARK-- message capability
module(load="imudp" threads="4" batchSize ="128") # Provides UDP syslog reception
module(load="imptcp" threads="10") # Provides TCP syslog reception
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") # Use default timestamp format
###############
#### RULES ####
###############
ruleset(name="sendToLogstash") {
action(type="omfwd"
name="action_ls"
Target="10.10.10.10"
Port="514"
Protocol="udp" # when this changed to TCP, throughput drops
queue.type="FixedArray"
)
}
###########################
#### LISTENERS ####
###########################
input(type="imudp" port="514" ruleset="sendToLogstash")
input(type="imptcp" port="514" ruleset="sendToLogstash")
###########################
#### GLOBAL DIRECTIVES ####
###########################
mark.* /var/log/messages
# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")
