We currently using rsyslog for gathering logs from ~1M IoT devices, we used to use plain-text UDP syslog and our server handling those just fine with minimal system load, but now because we need to send some sensitive data inside the syslog so we decided to enable TLS but after tried it with just 20k devices, our server keeps dropping logs because all the CPU cores are loaded at 100%.
It looks like that TCP TLS isn't very suitable for this large amount of requests because the syslog client just sent a message then disconnect, forcing it to redo the TLS handshake process every time it wants to send a log. Is there any other encryption mechanism included with rsyslog that I could've used? I was thinking of some sort of asymmetric encryption with a static key so it doesn't have to do the handshake every time?
I've thought about VPN but OpenVPN will create an even bigger load on the VPN server meanwhile our client hardware doesn't have a recent enough kernel for a lightweight Wireguard setup.
Our current rsyslog server is having 40 cores (80 HT), 48GB RAM so if we have to buy more server for setup some sort of load balancing system then there will be a significant cost constrain.
