Proxy www-out reached process FD limit (maxsock=4168). Please check 'ulimit-n' and restart

I've compiled haproxy to apply special LUA filters and now haproxy keeps running into the max file open limit. It runs happily along and then suddenly the logs show the following message:

Proxy www-out reached process FD limit (maxsock=4026). Please check 'ulimit-n' and restart.

I have tried increasing this limit by creating a service limits file for haproxy.

cat /etc/systemd/system/haproxy.service.d/limits.conf 
[Service]
LimitNOFILE=600000

When starting haproxy the limits file is loaded.

● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/haproxy.service.d
           └─limits.conf
   Active: active (running) since Wed 2021-08-11 15:07:08 CEST; 8s ago
     Docs: man:haproxy(1)
           file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 25865 ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
 Main PID: 25867 (haproxy)

However that seems to only change the hard limit and not the soft limit of haproxy, looking at the process.

root     25867  0.1  0.0 141136 18264 ?        Ss   15:07   0:00 /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy  25869  0.1  0.0 1173536 12860 ?       Sl   15:07   0:00  \_ /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
test@test ~> cat /proc/25867/limits | grep "open"
Max open files            4168                 600000               files  

Finally just some info on haproxy

Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1
  DEBUG   = 

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=36).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.3
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0

I usually handle this kind of limitations under /etc/security/limits/limits.d where one can tweak both per-user hard and soft limits. (Check the comments inside /etc/security/limits/limits.conf for config details). I have no idea how that interacts with directives set within systemd.