Score:0

PHP8.1 FPM Crashes everyday after 12AM

cl flag

From the last two to three days I noticed that my PHP8.1-FPM process crashes after around 12:00 AM-12:30 AM at night.

My server has highest traffic in morning and lowest in night. But FPM process runs fine for whole day and crashes on night. This is very strange.

Here is what I found in logs:

DAY 1

[05-Dec-2022 00:19:47] NOTICE: [pool www] child 10868 started
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 9967 exited with code 0 after 7446.963850 seconds from start
[05-Dec-2022 00:25:02] NOTICE: [pool www] child 10871 started
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 9969 exited with code 0 after 7433.300005 seconds from start
[05-Dec-2022 00:27:37] NOTICE: [pool www] child 10872 started
[05-Dec-2022 00:32:41] WARNING: [pool www] child 10194 exited on signal 9 (SIGKILL) after 5713.957769 seconds from start
[05-Dec-2022 00:32:41] NOTICE: [pool www] child 10876 started
[05-Dec-2022 00:32:41] NOTICE: Terminating ...
[05-Dec-2022 00:32:41] NOTICE: exiting, bye-bye!

DAY 2

[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23687 started
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 21662 exited with code 0 after 11118.073861 seconds from start
[05-Dec-2022 22:42:26] NOTICE: [pool www] child 23688 started
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 21663 exited with code 0 after 11071.941161 seconds from start
[05-Dec-2022 22:42:31] NOTICE: [pool www] child 23689 started
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 21664 exited with code 0 after 11046.438863 seconds from start
[05-Dec-2022 22:42:41] NOTICE: [pool www] child 23690 started
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 22058 exited with code 0 after 11163.770384 seconds from start
[05-Dec-2022 23:31:34] NOTICE: [pool www] child 24124 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23076 exited on signal 9 (SIGKILL) after 8637.063701 seconds from start
[06-Dec-2022 00:09:32] NOTICE: [pool www] child 24580 started
[06-Dec-2022 00:09:32] WARNING: [pool www] child 23080 exited on signal 9 (SIGKILL) after 8607.986722 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24581 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23089 exited on signal 9 (SIGKILL) after 8507.937877 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24582 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23380 exited on signal 9 (SIGKILL) after 6752.142496 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24583 started
[06-Dec-2022 00:09:33] WARNING: [pool www] child 23403 exited on signal 9 (SIGKILL) after 6643.858307 seconds from start
[06-Dec-2022 00:09:33] NOTICE: [pool www] child 24584 started
[06-Dec-2022 00:09:33] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 147 total children
[06-Dec-2022 00:09:33] NOTICE: Terminating ...
[06-Dec-2022 00:09:33] NOTICE: exiting, bye-bye!
[06-Dec-2022 00:23:49] NOTICE: fpm is running, pid 24611
[06-Dec-2022 00:23:49] NOTICE: ready to handle connections
[06-Dec-2022 00:23:49] NOTICE: systemd monitor interval set to 10000ms
[06-Dec-2022 00:23:53] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 59 idle, and 63 total children

Server Configuration Digitalocean 4 Core AMD CPU 8GB RAM $54 USD basic droplet. PHP8.1 FPM Settings

pm = dynamic 
pm.max_children = 241 
pm.start_servers = 60
pm.min_spare_servers = 60 
pm.max_spare_servers = 180 
pm.max_requests = 500

-- UPDATE This application is a image server. It serves dynamically resized images from object storage or images from another server. It has very small php script which resizes images and serve them. Nginx is used for caching processed images.

-- UPDATE dmesg output

[147949.035475] 0 pages in swap cache
[147949.035475] Swap cache stats: add 0, delete 0, find 0/0
[147949.035476] Free swap  = 0kB
[147949.035477] Total swap = 0kB
[147949.035478] 2097014 pages RAM
[147949.035478] 0 pages HighMem/MovableOnly
[147949.035479] 61947 pages reserved
[147949.035480] 0 pages hwpoisoned
[147949.035480] Tasks state (memory values in pages):
[147949.035481] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[147949.035485] [    380]     0   380     9935      864   106496        0          -250 systemd-journal
[147949.035489] [    417]     0   417    72328     6775   114688        0         -1000 multipathd
[147949.035496] [    539]   103   539    22338      415    73728        0             0 systemd-timesyn
[147949.035498] [    606]   100   606     4029      626    73728        0             0 systemd-network
[147949.035500] [    625]   101   625     6378     1681    94208        0             0 systemd-resolve
[147949.035502] [    650]     0   650     2880     1060    69632        0         -1000 systemd-udevd
[147949.035505] [    747]     0   747     1821      618    57344        0             0 cron
[147949.035506] [    749]   102   749     2191      891    57344        0          -900 dbus-daemon
[147949.035508] [    752]   998   752   345261     1828   208896        0          -900 do-agent
[147949.035511] [    757]     0   757    20699      669    61440        0             0 irqbalance
[147949.035513] [    758]     0   758     8249     2870   102400        0             0 networkd-dispat
[147949.035515] [    767]     0   767    58621      410    81920        0             0 polkitd
[147949.035517] [    768]   104   768    55600     1093    77824        0             0 rsyslogd
[147949.035522] [    771]     0   771   292597     4332   307200        0          -900 snapd
[147949.035524] [    776]     0   776     3873      574    69632        0             0 systemd-logind
[147949.035526] [    778]     0   778    98136     1305   122880        0             0 udisksd
[147949.035529] [    786]     0   786    79236      954   118784        0             0 ModemManager
[147949.035531] [    787]     0   787     1554      192    49152        0             0 agetty
[147949.035534] [    798]     0   798     1543      214    53248        0             0 agetty
[147949.035535] [    826]     0   826     3855     1169    73728        0         -1000 sshd
[147949.035538] [    863]     0   863    27523     2822   114688        0             0 unattended-upgr
[147949.035540] [    904]     0   904    40076     1619   110592        0             0 nginx
[147949.035541] [   1373]     0  1373    73985     1009   163840        0             0 packagekitd
[147949.035544] [   1712]     0  1712   343392      174   180224        0          -900 droplet-agent
[147949.035547] [  12216]     0 12216    53067     2645   151552        0             0 php-fpm8.1
[147949.035549] [  12743]    33 12743    50015    15472   225280        0             0 nginx
[147949.035552] [  12744]    33 12744    51002    16135   233472        0             0 nginx
[147949.035554] [  12745]    33 12745    50614    16019   233472        0             0 nginx
[147949.035556] [  12746]    33 12746    50765    16062   233472        0             0 nginx
[147949.035558] [  12747]    33 12747    40214     5403   147456        0             0 nginx
[147949.035560] [  23025]    33 23025    81250    26391   356352        0             0 php-fpm8.1
[147949.035562] [  23027]    33 23027    63171    13406   245760        0             0 php-fpm8.1
[147949.035564] [  23028]    33 23028    66013    15875   266240        0             0 php-fpm8.1
[147949.035566] [  23029]    33 23029    81755    23040   335872        0             0 php-fpm8.1
[147949.035568] [  23030]    33 23030    81289    25643   352256        0             0 php-fpm8.1
[147949.035570] [  23031]    33 23031    65772    15814   266240        0             0 php-fpm8.1
[147949.035572] [  23032]    33 23032    60680    10956   225280        0             0 php-fpm8.1
[147949.035574] [  23033]    33 23033    81290    22731   327680        0             0 php-fpm8.1
[147949.035576] [  23034]    33 23034    79989    22259   327680        0             0 php-fpm8.1
[147949.035578] [  23035]    33 23035    62802    12404   241664        0             0 php-fpm8.1
[147949.035579] [  23036]    33 23036    81250    22808   327680        0             0 php-fpm8.1
[147949.035581] [  23037]    33 23037    81760    23549   331776        0             0 php-fpm8.1
[147949.035584] [  23038]    33 23038    63141    12398   237568        0             0 php-fpm8.1
[147949.035585] [  23040]    33 23040    65876    16118   266240        0             0 php-fpm8.1
[147949.035588] [  23041]    33 23041    54628     5099   176128        0             0 php-fpm8.1
[147949.035593] [  23042]    33 23042    80737    22410   323584        0             0 php-fpm8.1
[147949.035595] [  23043]    33 23043    63388    12852   245760        0             0 php-fpm8.1
[147949.035596] [  23044]    33 23044    79976    26488   364544        0             0 php-fpm8.1
[147949.035599] [  23045]    33 23045    81353    25677   352256        0             0 php-fpm8.1
[147949.035601] [  23046]    33 23046    64263    14482   253952        0             0 php-fpm8.1
[147949.035603] [  23048]    33 23048    81247    28102   368640        0             0 php-fpm8.1
[147949.035604] [  23049]    33 23049    81763    17517   286720        0             0 php-fpm8.1
[147949.035606] [  23050]    33 23050    54633     5260   176128        0             0 php-fpm8.1
[147949.035608] [  23051]    33 23051    81745    27413   368640        0             0 php-fpm8.1
[147949.035610] [  23052]    33 23052    81250    20292   307200        0             0 php-fpm8.1
[147949.035612] [  23053]    33 23053    62492    12051   237568        0             0 php-fpm8.1
[147949.035614] [  23054]    33 23054    79479    19786   311296        0             0 php-fpm8.1
[147949.035616] [  23055]    33 23055    82274    24380   339968        0             0 php-fpm8.1
[147949.035618] [  23056]    33 23056    81762    23765   335872        0             0 php-fpm8.1
[147949.035619] [  23057]    33 23057    60234    10616   221184        0             0 php-fpm8.1
[147949.035624] [  23058]    33 23058    62266    12114   237568        0             0 php-fpm8.1
[147949.035626] [  23059]    33 23059    64558    15277   253952        0             0 php-fpm8.1
[147949.035628] [  23060]    33 23060    81670    17767   286720        0             0 php-fpm8.1
[147949.035630] [  23061]    33 23061    81740    23569   335872        0             0 php-fpm8.1
[147949.035632] [  23062]    33 23062    81765    27157   364544        0             0 php-fpm8.1
[147949.035634] [  23063]    33 23063    55149     5538   180224        0             0 php-fpm8.1
[147949.035636] [  23064]    33 23064    53607     4475   167936        0             0 php-fpm8.1
[147949.035638] [  23065]    33 23065    81251    26365   356352        0             0 php-fpm8.1
[147949.035640] [  23066]    33 23066    79991    19315   303104        0             0 php-fpm8.1
[147949.035642] [  23067]    33 23067    79803    20677   315392        0             0 php-fpm8.1
[147949.035643] [  23068]    33 23068    65398    16096   262144        0             0 php-fpm8.1
[147949.035645] [  23069]    33 23069    62633    12246   233472        0             0 php-fpm8.1
[147949.035647] [  23070]    33 23070    79262    19484   299008        0             0 php-fpm8.1
[147949.035649] [  23071]    33 23071    79788    30080   376832        0             0 php-fpm8.1
[147949.035651] [  23072]    33 23072    62276    12389   237568        0             0 php-fpm8.1
[147949.035653] [  23073]    33 23073    79974    21148   323584        0             0 php-fpm8.1
[147949.035655] [  23074]    33 23074    81250    20299   307200        0             0 php-fpm8.1
[147949.035656] [  23075]    33 23075    63702    13680   249856        0             0 php-fpm8.1
[147949.035658] [  23077]    33 23077    80732    17341   282624        0             0 php-fpm8.1
[147949.035660] [  23078]    33 23078    79772    20834   303104        0             0 php-fpm8.1
[147949.035662] [  23079]    33 23079    65346    15686   262144        0             0 php-fpm8.1
[147949.035664] [  23081]    33 23081    63206    13763   245760        0             0 php-fpm8.1
[147949.035666] [  23082]    33 23082    81761    24195   339968        0             0 php-fpm8.1
[147949.035667] [  23083]    33 23083    64690    15490   258048        0             0 php-fpm8.1
[147949.035669] [  23084]    33 23084    54811     5039   176128        0             0 php-fpm8.1
[147949.035674] [  23085]    33 23085    63144    12510   237568        0             0 php-fpm8.1
[147949.035676] [  23086]    33 23086    62481    12562   237568        0             0 php-fpm8.1
[147949.035680] [  23087]    33 23087    65662    15683   262144        0             0 php-fpm8.1
[147949.035682] [  23088]    33 23088    81757    27232   364544        0             0 php-fpm8.1
[147949.035684] [  23090]    33 23090    81762    18209   290816        0             0 php-fpm8.1
[147949.035686] [  23091]    33 23091    65152    15280   258048        0             0 php-fpm8.1
[147949.035688] [  23092]    33 23092    79988    21290   327680        0             0 php-fpm8.1
[147949.035690] [  23171]    33 23171    81760    18679   294912        0             0 php-fpm8.1
[147949.035692] [  23173]    33 23173    79410    17029   286720        0             0 php-fpm8.1
[147949.035694] [  23175]    33 23175    65824    16263   266240        0             0 php-fpm8.1
[147949.035696] [  23185]    33 23185    65365    16180   262144        0             0 php-fpm8.1
[147949.035698] [  23186]    33 23186    65231    15066   262144        0             0 php-fpm8.1
[147949.035699] [  23187]    33 23187    79988    20002   315392        0             0 php-fpm8.1
[147949.035701] [  23375]    33 23375    68209    18499   282624        0             0 php-fpm8.1
[147949.035703] [  23378]    33 23378    60998    11722   225280        0             0 php-fpm8.1
[147949.035705] [  23379]    33 23379    79435    18467   286720        0             0 php-fpm8.1
[147949.035707] [  23388]    33 23388    79258    21188   319488        0             0 php-fpm8.1
[147949.035709] [  23389]    33 23389    79453    20039   311296        0             0 php-fpm8.1
[147949.035711] [  23390]    33 23390    61517    11708   229376        0             0 php-fpm8.1
[147949.035713] [  23391]    33 23391    55516     5670   184320        0             0 php-fpm8.1
[147949.035715] [  23392]    33 23392    63335    12141   237568        0             0 php-fpm8.1
[147949.035717] [  23393]    33 23393    81245    19769   303104        0             0 php-fpm8.1
[147949.035721] [  23394]    33 23394    80731    18069   290816        0             0 php-fpm8.1
[147949.035723] [  23395]    33 23395    81229    17634   290816        0             0 php-fpm8.1
[147949.035725] [  23396]    33 23396    62102    11782   233472        0             0 php-fpm8.1
[147949.035727] [  23397]    33 23397    61251    11551   229376        0             0 php-fpm8.1
[147949.035729] [  23398]    33 23398    65025    15476   258048        0             0 php-fpm8.1
[147949.035731] [  23399]    33 23399    56609     6542   192512        0             0 php-fpm8.1
[147949.035733] [  23400]    33 23400    61918    11956   233472        0             0 php-fpm8.1
[147949.035735] [  23401]    33 23401    78978    25135   352256        0             0 php-fpm8.1
[147949.035736] [  23402]    33 23402    61182    11420   229376        0             0 php-fpm8.1
[147949.035738] [  23404]    33 23404    65174    15748   262144        0             0 php-fpm8.1
[147949.035740] [  23405]    33 23405    81247    22004   319488        0             0 php-fpm8.1
[147949.035742] [  23406]    33 23406    53598     4420   167936        0             0 php-fpm8.1
[147949.035744] [  23407]    33 23407    61984    11988   233472        0             0 php-fpm8.1
[147949.035746] [  23408]    33 23408    63977    14485   249856        0             0 php-fpm8.1
[147949.035748] [  23409]    33 23409    81247    25246   348160        0             0 php-fpm8.1
[147949.035749] [  23410]    33 23410    64153    14631   253952        0             0 php-fpm8.1
[147949.035751] [  23411]    33 23411    81279    23444   331776        0             0 php-fpm8.1
[147949.035753] [  23412]    33 23412    65221    15610   262144        0             0 php-fpm8.1
[147949.035755] [  23413]    33 23413    81753    27004   360448        0             0 php-fpm8.1
[147949.035757] [  23414]    33 23414    63471    14123   245760        0             0 php-fpm8.1
[147949.035759] [  23415]    33 23415    63396    14222   245760        0             0 php-fpm8.1
[147949.035761] [  23467]    33 23467    80729    15421   266240        0             0 php-fpm8.1
[147949.035763] [  23469]    33 23469    65609    15655   262144        0             0 php-fpm8.1
[147949.035767] [  23470]    33 23470    63897    14733   249856        0             0 php-fpm8.1
[147949.035769] [  23472]    33 23472    81239    22790   327680        0             0 php-fpm8.1
[147949.035771] [  23473]    33 23473    62872    13170   241664        0             0 php-fpm8.1
[147949.035773] [  23474]    33 23474    64835    15560   258048        0             0 php-fpm8.1
[147949.035775] [  23475]    33 23475    65215    15739   262144        0             0 php-fpm8.1
[147949.035777] [  23476]    33 23476    63137    12595   237568        0             0 php-fpm8.1
[147949.035778] [  23477]    33 23477    78749    17139   282624        0             0 php-fpm8.1
[147949.035780] [  23478]    33 23478    81236    22468   323584        0             0 php-fpm8.1
[147949.035783] [  23479]    33 23479    81238    22144   319488        0             0 php-fpm8.1
[147949.035784] [  23480]    33 23480    61533    12097   229376        0             0 php-fpm8.1
[147949.035786] [  23481]    33 23481    61742    12461   233472        0             0 php-fpm8.1
[147949.035788] [  23668]    33 23668    81120    27387   364544        0             0 php-fpm8.1
[147949.035790] [  23669]    33 23669    81281    25762   352256        0             0 php-fpm8.1
[147949.035792] [  23670]    33 23670    80711    22075   323584        0             0 php-fpm8.1
[147949.035794] [  23671]    33 23671    81629    21215   311296        0             0 php-fpm8.1
[147949.035796] [  23672]    33 23672    54106     4783   172032        0             0 php-fpm8.1
[147949.035798] [  23673]    33 23673    80732    20166   303104        0             0 php-fpm8.1
[147949.035800] [  23674]    33 23674    62566    13115   237568        0             0 php-fpm8.1
[147949.035802] [  23675]    33 23675    62266    11933   237568        0             0 php-fpm8.1
[147949.035804] [  23676]    33 23676    62778    12260   237568        0             0 php-fpm8.1
[147949.035805] [  23677]    33 23677    93296    27104   364544        0             0 php-fpm8.1
[147949.035807] [  23678]    33 23678    79960    26685   364544        0             0 php-fpm8.1
[147949.035809] [  23679]    33 23679    79235    17993   286720        0             0 php-fpm8.1
[147949.035814] [  23680]    33 23680    63334    13892   245760        0             0 php-fpm8.1
[147949.035816] [  23681]    33 23681    82264    22326   323584        0             0 php-fpm8.1
[147949.035817] [  23682]    33 23682    62045    12131   233472        0             0 php-fpm8.1
[147949.035819] [  23683]    33 23683    81649    24578   344064        0             0 php-fpm8.1
[147949.035821] [  23684]    33 23684    81755    25202   348160        0             0 php-fpm8.1
[147949.035823] [  23685]    33 23685    56617     6875   192512        0             0 php-fpm8.1
[147949.035825] [  23686]    33 23686    55403     5763   180224        0             0 php-fpm8.1
[147949.035827] [  23687]    33 23687    78956    21007   319488        0             0 php-fpm8.1
[147949.035829] [  23688]    33 23688    58896     9008   208896        0             0 php-fpm8.1
[147949.035830] [  23689]    33 23689    61751    11976   233472        0             0 php-fpm8.1
[147949.035832] [  23690]    33 23690    79253    29982   372736        0             0 php-fpm8.1
[147949.035834] [  23803]     0 23803     4285      846    73728        0             0 sshd
[147949.035836] [  23806]     0 23806     4260      971    73728        0             0 systemd
[147949.035838] [  23807]     0 23807    42651     1255    90112        0             0 (sd-pam)
[147949.035840] [  23863]     0 23863     2287      943    53248        0             0 bash
[147949.035842] [  23878]     0 23878     2152      868    53248        0             0 htop
[147949.035843] [  24124]    33 24124    63362    13980   245760        0             0 php-fpm8.1
[147949.035846] [  24573]    33 24573    54409     4318   176128        0             0 php-fpm8.1
[147949.035848] [  24574]     0 24574    53067     1478   126976        0             0 php-fpm8.1
[147949.035850] [  24575]     0 24575    53150     1663   126976        0             0 php-fpm8.1
[147949.035854] [  24576]     0 24576    53067     1235   122880        0             0 php-fpm8.1
[147949.035856] [  24577]     0 24577    53067     1589   122880        0             0 php-fpm8.1
[147949.035857] [  24578]     0 24578    53067     1591   122880        0             0 php-fpm8.1
[147949.035859] [  24579]     0 24579    53067     1572   122880        0             0 php-fpm8.1
[147949.035861] [  24580]     0 24580    53067     1615   122880        0             0 php-fpm8.1
[147949.035863] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-resolved.service,mems_allowed=0,global_oom,task_memcg=/system.slice/php8.1-fpm.service,task=php-fpm8.1,pid=23071,uid=33
[147949.035882] Out of memory: Killed process 23071 (php-fpm8.1) total-vm:319152kB, anon-rss:112388kB, file-rss:1824kB, shmem-rss:6108kB, UID:33 pgtables:368kB oom_score_adj:0
us flag
What is the web application you run on the server?
cl flag
@Tero Kilkanen This application is a image server. It serves dynamically resized images from object storage or images from another server. It has very small php script which resizes images and serve them. Nginx is used for caching processed images.
us flag
What is the output of `dmesg`?
cl flag
[147949.035863] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-resolved.service,mems_allowed=0,global_oom,task_memcg=/system.slice/php8.1-fpm.service,task=php-fpm8.1,pid=23071,uid=33 [147949.035882] Out of memory: Killed process 23071 (php-fpm8.1) total-vm:319152kB, anon-rss:112388kB, file-rss:1824kB, shmem-rss:6108kB, UID:33 pgtables:368kB oom_score_adj:0
Score:1
us flag

Your server runs out of memory at this point.

Maybe there is some scheduled task that launches multiple long-lasting and memory-intensive operations. This then triggers the Linux kernel out of memory killer, that terminates the processes that consume most memory.

You need to increase either memory or swap space. Or look into the memory usage of your application and see if it can be reduced.

cl flag
Most strange part is, Digital Ocean monitoring system did not reported High Memory use when crash happend. https://imgur.com/a/sVeJORt By the way, today I have added 8GB swap space and monitoring closely. Lets see what happens between 12-1 AM.
cl flag
Adding swap worked! At about 12:07AM I noticed a sudden increase in RAM usage from 2.3GB to 7GB. But swap space saved the server from crashing. This sudden increase was for about few seconds after that everything went normal. Thank you for your suggestion.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.