Mariadb creates a subprocess after crash

We recently had a crash on our Mariadb server (10.6.5) on a Debian 9 VM. The process was restarted without error but now there's a second mariadb process like so :

mysql    26718  5.1 42.4 15062644 12788308 ?   Ssl  Apr13  61:53 /usr/sbin/mariadbd
mysql    26777  1.9 11.5 15152500 3488712 ?    S    Apr13  22:38 /usr/sbin/mariadbd

which seems to be a subprocess of the first one.

We can't stop those two processes with

systemctl stop mariadb

The command hangs indefinitely. The only way to get rid of them is to kill -9 both processes. If we restart the mariadb process, the second process will eventually start with no reason. If the subprocess is killed first, it will reappear a bit later. No error is visible in the logs.

The Database is also only accessible in read, any try to write will hang forever. I'm considering dumping and reinstalling the database.

EDIT 1

Here's the log when the server crashed :

Apr 12 22:51:21 my-server mariadbd[23495]: 2023-04-12 22:51:21 0 [ERROR] [FATAL] InnoDB: Page old data size 15834 new data size 13676, page old max ins size 40 new max ins size 2198
Apr 12 22:51:21 my-server mariadbd[23495]: 230412 22:51:21 [ERROR] mysqld got signal 6 ;
Apr 12 22:51:21 my-server mariadbd[23495]: This could be because you hit a bug. It is also possible that this binary
Apr 12 22:51:21 my-server mariadbd[23495]: or one of the libraries it was linked against is corrupt, improperly built,
Apr 12 22:51:21 my-server mariadbd[23495]: or misconfigured. This error can also be caused by malfunctioning hardware.
Apr 12 22:51:21 my-server mariadbd[23495]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Apr 12 22:51:21 my-server mariadbd[23495]: We will try our best to scrape up some info that will hopefully help
Apr 12 22:51:21 my-server mariadbd[23495]: diagnose the problem, but since we have already crashed,
Apr 12 22:51:21 my-server mariadbd[23495]: something is definitely wrong and this may fail.

First, its not a subprocess, its a previous version of MariaDB that was running before. Most likely its from a instance that failed to stop within the timeout.

Obviously MariaDB isn't meant to run like this. The result is two instances without awareness of each other modifying the same files which as you can see is fatal. You will need to restore from backup.

Newer versions of MariaDB (10.6.9+) lock aria_log_control and ibdata1 to prevent such corruption. Also newer versions of systemd (v242+) combined with a newer service file that included SendSIGKILL=no also prevent duplicate starting by the same service(ref).

I'm sorry you ended up in a situation without these protections.