Edit: It would appear the culprit is libcrypto.so.3;
Updated question:
- Is LibSSL3 slower in general?
- Is LibSS33 just slower for legacy providers? or
- Is this likely a PHP interface issue?
We have had to add "legacy" support for older encryption (upgrade process in progress, but we still need old support). Removal is unfortunately not an option at this stage.
We upgraded from Ubuntu 20.04.1 to ubuntu 22.04. CPU use is 100% worse on AWS / EC2.
See the attached graph
.
- Orange/green/blue: Ubuntu 22.04.1, running at 40% CPU (while each server handling 25% of the load)
- Purple: Ubuntu 20.04, running under 20% (also handling 25% of the load)
- Red: Adding a 5th server, experimenting with other solutions available - see below.
top confirms that it's just CPU that's doubled. (Buffered / Cached memory can be accounted for because of uptime) - the main difference is si (softirq)
Server setup identical (scripted) in all cases. Any configuration is identical.
Installed software are
- apache,
- php 8.1 (through fast CGI, include mongoDB, memcache, maxmindDB and redis libraries; FastCGI reports similar number of processes)
- CloudWatch services.
EC2 spec is c5.xlarge, g3 volume. AWS auth through Role credentials (cached). Nothing special.
Volume (disk) throughput identical (according to AWS monitoring); Network traffic identical on all servers (according to AWS monitoring and nload).
Where else can I look to resolve? Is there additional config for 22 that is required over 20? What are softirqs - or is that a by product of the issue?
WARNING: If you do intel_iommu=off as suggested in other solution, CPU goes to 100%, load reached meltdown - see other image. Be very careful with this "fix".
