From the last two to three days I noticed that my PHP8.1-FPM process crashes after around 12:00 AM-12:30 AM at night.
My server has highest traffic in morning and lowest in night. But FPM process runs fine for whole day and crashes on night. This is very strange.
Here is what I found in logs:
DAY 1
[05-Dec-2022 00:19:47] NOTICE: [pool www] child 10868 started
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 9967 exited with code 0 after 7446.963850 seconds from start
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 10871 started
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 9969 exited with code 0 after 7433.300005 seconds from start
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 10872 started
[05-Dec-2022 00:32:41] WARNING: [pool www] child 10194 exited on signal 9 (SIGKILL) after 5713.957769 seconds from start
[05-Dec-2022 00:32:41] NOTICE: [pool www] child 10876 started
[05-Dec-2022 00:32:41] NOTICE: Terminating ...
[05-Dec-2022 00:32:41] NOTICE: exiting, bye-bye!
DAY 2
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23687 started
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 21662 exited with code 0 after 11118.073861 seconds from start
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23688 started
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 21663 exited with code 0 after 11071.941161 seconds from start
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 23689 started
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 21664 exited with code 0 after 11046.438863 seconds from start
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 23690 started
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 22058 exited with code 0 after 11163.770384 seconds from start
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 24124 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23076 exited on signal 9 (SIGKILL) after 8637.063701 seconds from start
[06-Dec-2022 00:09:32] NOTICE: [pool www] child 24580 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23080 exited on signal 9 (SIGKILL) after 8607.986722 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24581 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23089 exited on signal 9 (SIGKILL) after 8507.937877 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24582 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23380 exited on signal 9 (SIGKILL) after 6752.142496 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24583 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23403 exited on signal 9 (SIGKILL) after 6643.858307 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24584 started
[06-Dec-2022 00:09:33] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 147 total children
[06-Dec-2022 00:09:33] NOTICE: Terminating ...
[06-Dec-2022 00:09:33] NOTICE: exiting, bye-bye!
[06-Dec-2022 00:23:49] NOTICE: fpm is running, pid 24611
[06-Dec-2022 00:23:49] NOTICE: ready to handle connections
[06-Dec-2022 00:23:49] NOTICE: systemd monitor interval set to 10000ms
[06-Dec-2022 00:23:53] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 59 idle, and 63 total children
Server Configuration
Digitalocean 4 Core AMD CPU 8GB RAM $54 USD basic droplet.
PHP8.1 FPM Settings
pm = dynamic
pm.max_children = 241
pm.start_servers = 60
pm.min_spare_servers = 60
pm.max_spare_servers = 180
pm.max_requests = 500
-- UPDATE
This application is a image server. It serves dynamically resized images from object storage or images from another server. It has very small php script which resizes images and serve them. Nginx is used for caching processed images.
-- UPDATE
dmesg output
[147949.035475] 0 pages in swap cache
[147949.035475] Swap cache stats: add 0, delete 0, find 0/0
[147949.035476] Free swap = 0kB
[147949.035477] Total swap = 0kB
[147949.035478] 2097014 pages RAM
[147949.035478] 0 pages HighMem/MovableOnly
[147949.035479] 61947 pages reserved
[147949.035480] 0 pages hwpoisoned
[147949.035480] Tasks state (memory values in pages):
[147949.035481] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[147949.035485] [ 380] 0 380 9935 864 106496 0 -250 systemd-journal
[147949.035489] [ 417] 0 417 72328 6775 114688 0 -1000 multipathd
[147949.035496] [ 539] 103 539 22338 415 73728 0 0 systemd-timesyn
[147949.035498] [ 606] 100 606 4029 626 73728 0 0 systemd-network
[147949.035500] [ 625] 101 625 6378 1681 94208 0 0 systemd-resolve
[147949.035502] [ 650] 0 650 2880 1060 69632 0 -1000 systemd-udevd
[147949.035505] [ 747] 0 747 1821 618 57344 0 0 cron
[147949.035506] [ 749] 102 749 2191 891 57344 0 -900 dbus-daemon
[147949.035508] [ 752] 998 752 345261 1828 208896 0 -900 do-agent
[147949.035511] [ 757] 0 757 20699 669 61440 0 0 irqbalance
[147949.035513] [ 758] 0 758 8249 2870 102400 0 0 networkd-dispat
[147949.035515] [ 767] 0 767 58621 410 81920 0 0 polkitd
[147949.035517] [ 768] 104 768 55600 1093 77824 0 0 rsyslogd
[147949.035522] [ 771] 0 771 292597 4332 307200 0 -900 snapd
[147949.035524] [ 776] 0 776 3873 574 69632 0 0 systemd-logind
[147949.035526] [ 778] 0 778 98136 1305 122880 0 0 udisksd
[147949.035529] [ 786] 0 786 79236 954 118784 0 0 ModemManager
[147949.035531] [ 787] 0 787 1554 192 49152 0 0 agetty
[147949.035534] [ 798] 0 798 1543 214 53248 0 0 agetty
[147949.035535] [ 826] 0 826 3855 1169 73728 0 -1000 sshd
[147949.035538] [ 863] 0 863 27523 2822 114688 0 0 unattended-upgr
[147949.035540] [ 904] 0 904 40076 1619 110592 0 0 nginx
[147949.035541] [ 1373] 0 1373 73985 1009 163840 0 0 packagekitd
[147949.035544] [ 1712] 0 1712 343392 174 180224 0 -900 droplet-agent
[147949.035547] [ 12216] 0 12216 53067 2645 151552 0 0 php-fpm8.1
[147949.035549] [ 12743] 33 12743 50015 15472 225280 0 0 nginx
[147949.035552] [ 12744] 33 12744 51002 16135 233472 0 0 nginx
[147949.035554] [ 12745] 33 12745 50614 16019 233472 0 0 nginx
[147949.035556] [ 12746] 33 12746 50765 16062 233472 0 0 nginx
[147949.035558] [ 12747] 33 12747 40214 5403 147456 0 0 nginx
[147949.035560] [ 23025] 33 23025 81250 26391 356352 0 0 php-fpm8.1
[147949.035562] [ 23027] 33 23027 63171 13406 245760 0 0 php-fpm8.1
[147949.035564] [ 23028] 33 23028 66013 15875 266240 0 0 php-fpm8.1
[147949.035566] [ 23029] 33 23029 81755 23040 335872 0 0 php-fpm8.1
[147949.035568] [ 23030] 33 23030 81289 25643 352256 0 0 php-fpm8.1
[147949.035570] [ 23031] 33 23031 65772 15814 266240 0 0 php-fpm8.1
[147949.035572] [ 23032] 33 23032 60680 10956 225280 0 0 php-fpm8.1
[147949.035574] [ 23033] 33 23033 81290 22731 327680 0 0 php-fpm8.1
[147949.035576] [ 23034] 33 23034 79989 22259 327680 0 0 php-fpm8.1
[147949.035578] [ 23035] 33 23035 62802 12404 241664 0 0 php-fpm8.1
[147949.035579] [ 23036] 33 23036 81250 22808 327680 0 0 php-fpm8.1
[147949.035581] [ 23037] 33 23037 81760 23549 331776 0 0 php-fpm8.1
[147949.035584] [ 23038] 33 23038 63141 12398 237568 0 0 php-fpm8.1
[147949.035585] [ 23040] 33 23040 65876 16118 266240 0 0 php-fpm8.1
[147949.035588] [ 23041] 33 23041 54628 5099 176128 0 0 php-fpm8.1
[147949.035593] [ 23042] 33 23042 80737 22410 323584 0 0 php-fpm8.1
[147949.035595] [ 23043] 33 23043 63388 12852 245760 0 0 php-fpm8.1
[147949.035596] [ 23044] 33 23044 79976 26488 364544 0 0 php-fpm8.1
[147949.035599] [ 23045] 33 23045 81353 25677 352256 0 0 php-fpm8.1
[147949.035601] [ 23046] 33 23046 64263 14482 253952 0 0 php-fpm8.1
[147949.035603] [ 23048] 33 23048 81247 28102 368640 0 0 php-fpm8.1
[147949.035604] [ 23049] 33 23049 81763 17517 286720 0 0 php-fpm8.1
[147949.035606] [ 23050] 33 23050 54633 5260 176128 0 0 php-fpm8.1
[147949.035608] [ 23051] 33 23051 81745 27413 368640 0 0 php-fpm8.1
[147949.035610] [ 23052] 33 23052 81250 20292 307200 0 0 php-fpm8.1
[147949.035612] [ 23053] 33 23053 62492 12051 237568 0 0 php-fpm8.1
[147949.035614] [ 23054] 33 23054 79479 19786 311296 0 0 php-fpm8.1
[147949.035616] [ 23055] 33 23055 82274 24380 339968 0 0 php-fpm8.1
[147949.035618] [ 23056] 33 23056 81762 23765 335872 0 0 php-fpm8.1
[147949.035619] [ 23057] 33 23057 60234 10616 221184 0 0 php-fpm8.1
[147949.035624] [ 23058] 33 23058 62266 12114 237568 0 0 php-fpm8.1
[147949.035626] [ 23059] 33 23059 64558 15277 253952 0 0 php-fpm8.1
[147949.035628] [ 23060] 33 23060 81670 17767 286720 0 0 php-fpm8.1
[147949.035630] [ 23061] 33 23061 81740 23569 335872 0 0 php-fpm8.1
[147949.035632] [ 23062] 33 23062 81765 27157 364544 0 0 php-fpm8.1
[147949.035634] [ 23063] 33 23063 55149 5538 180224 0 0 php-fpm8.1
[147949.035636] [ 23064] 33 23064 53607 4475 167936 0 0 php-fpm8.1
[147949.035638] [ 23065] 33 23065 81251 26365 356352 0 0 php-fpm8.1
[147949.035640] [ 23066] 33 23066 79991 19315 303104 0 0 php-fpm8.1
[147949.035642] [ 23067] 33 23067 79803 20677 315392 0 0 php-fpm8.1
[147949.035643] [ 23068] 33 23068 65398 16096 262144 0 0 php-fpm8.1
[147949.035645] [ 23069] 33 23069 62633 12246 233472 0 0 php-fpm8.1
[147949.035647] [ 23070] 33 23070 79262 19484 299008 0 0 php-fpm8.1
[147949.035649] [ 23071] 33 23071 79788 30080 376832 0 0 php-fpm8.1
[147949.035651] [ 23072] 33 23072 62276 12389 237568 0 0 php-fpm8.1
[147949.035653] [ 23073] 33 23073 79974 21148 323584 0 0 php-fpm8.1
[147949.035655] [ 23074] 33 23074 81250 20299 307200 0 0 php-fpm8.1
[147949.035656] [ 23075] 33 23075 63702 13680 249856 0 0 php-fpm8.1
[147949.035658] [ 23077] 33 23077 80732 17341 282624 0 0 php-fpm8.1
[147949.035660] [ 23078] 33 23078 79772 20834 303104 0 0 php-fpm8.1
[147949.035662] [ 23079] 33 23079 65346 15686 262144 0 0 php-fpm8.1
[147949.035664] [ 23081] 33 23081 63206 13763 245760 0 0 php-fpm8.1
[147949.035666] [ 23082] 33 23082 81761 24195 339968 0 0 php-fpm8.1
[147949.035667] [ 23083] 33 23083 64690 15490 258048 0 0 php-fpm8.1
[147949.035669] [ 23084] 33 23084 54811 5039 176128 0 0 php-fpm8.1
[147949.035674] [ 23085] 33 23085 63144 12510 237568 0 0 php-fpm8.1
[147949.035676] [ 23086] 33 23086 62481 12562 237568 0 0 php-fpm8.1
[147949.035680] [ 23087] 33 23087 65662 15683 262144 0 0 php-fpm8.1
[147949.035682] [ 23088] 33 23088 81757 27232 364544 0 0 php-fpm8.1
[147949.035684] [ 23090] 33 23090 81762 18209 290816 0 0 php-fpm8.1
[147949.035686] [ 23091] 33 23091 65152 15280 258048 0 0 php-fpm8.1
[147949.035688] [ 23092] 33 23092 79988 21290 327680 0 0 php-fpm8.1
[147949.035690] [ 23171] 33 23171 81760 18679 294912 0 0 php-fpm8.1
[147949.035692] [ 23173] 33 23173 79410 17029 286720 0 0 php-fpm8.1
[147949.035694] [ 23175] 33 23175 65824 16263 266240 0 0 php-fpm8.1
[147949.035696] [ 23185] 33 23185 65365 16180 262144 0 0 php-fpm8.1
[147949.035698] [ 23186] 33 23186 65231 15066 262144 0 0 php-fpm8.1
[147949.035699] [ 23187] 33 23187 79988 20002 315392 0 0 php-fpm8.1
[147949.035701] [ 23375] 33 23375 68209 18499 282624 0 0 php-fpm8.1
[147949.035703] [ 23378] 33 23378 60998 11722 225280 0 0 php-fpm8.1
[147949.035705] [ 23379] 33 23379 79435 18467 286720 0 0 php-fpm8.1
[147949.035707] [ 23388] 33 23388 79258 21188 319488 0 0 php-fpm8.1
[147949.035709] [ 23389] 33 23389 79453 20039 311296 0 0 php-fpm8.1
[147949.035711] [ 23390] 33 23390 61517 11708 229376 0 0 php-fpm8.1
[147949.035713] [ 23391] 33 23391 55516 5670 184320 0 0 php-fpm8.1
[147949.035715] [ 23392] 33 23392 63335 12141 237568 0 0 php-fpm8.1
[147949.035717] [ 23393] 33 23393 81245 19769 303104 0 0 php-fpm8.1
[147949.035721] [ 23394] 33 23394 80731 18069 290816 0 0 php-fpm8.1
[147949.035723] [ 23395] 33 23395 81229 17634 290816 0 0 php-fpm8.1
[147949.035725] [ 23396] 33 23396 62102 11782 233472 0 0 php-fpm8.1
[147949.035727] [ 23397] 33 23397 61251 11551 229376 0 0 php-fpm8.1
[147949.035729] [ 23398] 33 23398 65025 15476 258048 0 0 php-fpm8.1
[147949.035731] [ 23399] 33 23399 56609 6542 192512 0 0 php-fpm8.1
[147949.035733] [ 23400] 33 23400 61918 11956 233472 0 0 php-fpm8.1
[147949.035735] [ 23401] 33 23401 78978 25135 352256 0 0 php-fpm8.1
[147949.035736] [ 23402] 33 23402 61182 11420 229376 0 0 php-fpm8.1
[147949.035738] [ 23404] 33 23404 65174 15748 262144 0 0 php-fpm8.1
[147949.035740] [ 23405] 33 23405 81247 22004 319488 0 0 php-fpm8.1
[147949.035742] [ 23406] 33 23406 53598 4420 167936 0 0 php-fpm8.1
[147949.035744] [ 23407] 33 23407 61984 11988 233472 0 0 php-fpm8.1
[147949.035746] [ 23408] 33 23408 63977 14485 249856 0 0 php-fpm8.1
[147949.035748] [ 23409] 33 23409 81247 25246 348160 0 0 php-fpm8.1
[147949.035749] [ 23410] 33 23410 64153 14631 253952 0 0 php-fpm8.1
[147949.035751] [ 23411] 33 23411 81279 23444 331776 0 0 php-fpm8.1
[147949.035753] [ 23412] 33 23412 65221 15610 262144 0 0 php-fpm8.1
[147949.035755] [ 23413] 33 23413 81753 27004 360448 0 0 php-fpm8.1
[147949.035757] [ 23414] 33 23414 63471 14123 245760 0 0 php-fpm8.1
[147949.035759] [ 23415] 33 23415 63396 14222 245760 0 0 php-fpm8.1
[147949.035761] [ 23467] 33 23467 80729 15421 266240 0 0 php-fpm8.1
[147949.035763] [ 23469] 33 23469 65609 15655 262144 0 0 php-fpm8.1
[147949.035767] [ 23470] 33 23470 63897 14733 249856 0 0 php-fpm8.1
[147949.035769] [ 23472] 33 23472 81239 22790 327680 0 0 php-fpm8.1
[147949.035771] [ 23473] 33 23473 62872 13170 241664 0 0 php-fpm8.1
[147949.035773] [ 23474] 33 23474 64835 15560 258048 0 0 php-fpm8.1
[147949.035775] [ 23475] 33 23475 65215 15739 262144 0 0 php-fpm8.1
[147949.035777] [ 23476] 33 23476 63137 12595 237568 0 0 php-fpm8.1
[147949.035778] [ 23477] 33 23477 78749 17139 282624 0 0 php-fpm8.1
[147949.035780] [ 23478] 33 23478 81236 22468 323584 0 0 php-fpm8.1
[147949.035783] [ 23479] 33 23479 81238 22144 319488 0 0 php-fpm8.1
[147949.035784] [ 23480] 33 23480 61533 12097 229376 0 0 php-fpm8.1
[147949.035786] [ 23481] 33 23481 61742 12461 233472 0 0 php-fpm8.1
[147949.035788] [ 23668] 33 23668 81120 27387 364544 0 0 php-fpm8.1
[147949.035790] [ 23669] 33 23669 81281 25762 352256 0 0 php-fpm8.1
[147949.035792] [ 23670] 33 23670 80711 22075 323584 0 0 php-fpm8.1
[147949.035794] [ 23671] 33 23671 81629 21215 311296 0 0 php-fpm8.1
[147949.035796] [ 23672] 33 23672 54106 4783 172032 0 0 php-fpm8.1
[147949.035798] [ 23673] 33 23673 80732 20166 303104 0 0 php-fpm8.1
[147949.035800] [ 23674] 33 23674 62566 13115 237568 0 0 php-fpm8.1
[147949.035802] [ 23675] 33 23675 62266 11933 237568 0 0 php-fpm8.1
[147949.035804] [ 23676] 33 23676 62778 12260 237568 0 0 php-fpm8.1
[147949.035805] [ 23677] 33 23677 93296 27104 364544 0 0 php-fpm8.1
[147949.035807] [ 23678] 33 23678 79960 26685 364544 0 0 php-fpm8.1
[147949.035809] [ 23679] 33 23679 79235 17993 286720 0 0 php-fpm8.1
[147949.035814] [ 23680] 33 23680 63334 13892 245760 0 0 php-fpm8.1
[147949.035816] [ 23681] 33 23681 82264 22326 323584 0 0 php-fpm8.1
[147949.035817] [ 23682] 33 23682 62045 12131 233472 0 0 php-fpm8.1
[147949.035819] [ 23683] 33 23683 81649 24578 344064 0 0 php-fpm8.1
[147949.035821] [ 23684] 33 23684 81755 25202 348160 0 0 php-fpm8.1
[147949.035823] [ 23685] 33 23685 56617 6875 192512 0 0 php-fpm8.1
[147949.035825] [ 23686] 33 23686 55403 5763 180224 0 0 php-fpm8.1
[147949.035827] [ 23687] 33 23687 78956 21007 319488 0 0 php-fpm8.1
[147949.035829] [ 23688] 33 23688 58896 9008 208896 0 0 php-fpm8.1
[147949.035830] [ 23689] 33 23689 61751 11976 233472 0 0 php-fpm8.1
[147949.035832] [ 23690] 33 23690 79253 29982 372736 0 0 php-fpm8.1
[147949.035834] [ 23803] 0 23803 4285 846 73728 0 0 sshd
[147949.035836] [ 23806] 0 23806 4260 971 73728 0 0 systemd
[147949.035838] [ 23807] 0 23807 42651 1255 90112 0 0 (sd-pam)
[147949.035840] [ 23863] 0 23863 2287 943 53248 0 0 bash
[147949.035842] [ 23878] 0 23878 2152 868 53248 0 0 htop
[147949.035843] [ 24124] 33 24124 63362 13980 245760 0 0 php-fpm8.1
[147949.035846] [ 24573] 33 24573 54409 4318 176128 0 0 php-fpm8.1
[147949.035848] [ 24574] 0 24574 53067 1478 126976 0 0 php-fpm8.1
[147949.035850] [ 24575] 0 24575 53150 1663 126976 0 0 php-fpm8.1
[147949.035854] [ 24576] 0 24576 53067 1235 122880 0 0 php-fpm8.1
[147949.035856] [ 24577] 0 24577 53067 1589 122880 0 0 php-fpm8.1
[147949.035857] [ 24578] 0 24578 53067 1591 122880 0 0 php-fpm8.1
[147949.035859] [ 24579] 0 24579 53067 1572 122880 0 0 php-fpm8.1
[147949.035861] [ 24580] 0 24580 53067 1615 122880 0 0 php-fpm8.1
[147949.035863] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-resolved.service,mems_allowed=0,global_oom,task_memcg=/system.slice/php8.1-fpm.service,task=php-fpm8.1,pid=23071,uid=33
[147949.035882] Out of memory: Killed process 23071 (php-fpm8.1) total-vm:319152kB, anon-rss:112388kB, file-rss:1824kB, shmem-rss:6108kB, UID:33 pgtables:368kB oom_score_adj:0
