Score:0

MariaDB 11.0.2 fails to start after upgrade from 10.11.4

cn flag

I am on Debian Bullseye and just upgraded (or at least attempted to upgrade) MariaDB from 10.11.4 to 11.0.2.

Unfortunately, the service fails to start now. All I'm getting through systemctl status mariadb.service and journalctl -xe is pretty unhelpful, like (respectively):

● mariadb.service - MariaDB 11.0.2 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/mariadb.service.d
             └─migrated-from-my.cnf-settings.conf
     Active: inactive (dead) (Result: signal) since Fri 2023-06-16 22:05:13 CEST; 4s ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 6291 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCE>
    Process: 6292 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCC>
    Process: 6294 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/>
    Process: 6323 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=killed, si>
   Main PID: 6323 (code=killed, signal=ABRT)
        CPU: 144ms
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max processes             128262               128262               processes
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max open files            32768                32768                files
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max locked memory         524288               524288               bytes
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max address space         unlimited            unlimited            bytes
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max file locks            unlimited            unlimited            locks
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max pending signals       128262               128262               signals
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max msgqueue size         819200               819200               bytes
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max nice priority         0                    0
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max realtime priority     0                    0
Jun 16 22:05:10 XXXXX mariadbd[6323]: Max realtime timeout      unlimited            unlimited            us
Jun 16 22:05:10 XXXXX mariadbd[6323]: Core pattern: core
Jun 16 22:05:10 XXXXX mariadbd[6323]: Kernel version: Linux version 5.10.0-23-amd64 ([email protected]) >
Jun 16 22:05:10 XXXXX systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit mariadb.service has exited.
░░
░░ The process' exit code is 'killed' and its exit status is 6.
Jun 16 22:05:10 XXXXX systemd[1]: mariadb.service: Failed with result 'signal'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'signal'.
Jun 16 22:05:10 XXXXX systemd[1]: Failed to start MariaDB 11.0.2 database server.
░░ Subject: A start job for unit mariadb.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit mariadb.service has finished with a failure.
░░
░░ The job identifier is 6703 and the job result is failed.

I would try to debug this further through error logs, unfortunately MariaDB somehow refuses to log errors even though I set log_error=/var/log/mysql/error.log and hunted down all instances of skip_log_error as well as syslog that seem to be preconfigured in the Debian config files.

Now, if I rename /var/lib/mysql and re-initialize it, the service works. However, this isn't a solution for me since I need to recover the data and of course I didn't back it up this one time.

Any ideas or potential solutions?

A.B avatar
cl flag
A.B
recover data? install back previous version which will hopefully start. dump all databases, dump privileges, upgrade, clean slate, restore dumped databases and grants.
Chaeska avatar
cn flag
Unfortunately the downgraded DB wasn't starting either by default. I needed to do a fresh installation and then copy over the raw DB files one by one without even dumping or converting them. Funnily enough even the system DBs worked so I literally didn't need to re-configure a single thing including privileges.
danblack avatar
dz flag
[with debug info packages](https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/#installing-debug-info-packages-from-mariadbs-debian-or-ubuntu-repository) you can run `sudo -u mysql gdb --args /usr/sbin/mariadbd` then `bt -frame-arguments all full` after the crash to [report a bug](https://jira.mariadb.org). Generally if log_error is blank it ends up at stdout/stderr which ends up easily in the systemd journal for easy access.
Score:1
cn flag

Not quite an answer for all future users but I managed to recover all data by first downgrading to a fresh installation (backing up the previous data directory) and then copying over databases and other files from the data directory one-by-one. Funnily enough this resulted in forwards compatibility as well when proceeding this way. I believe the issue was that the upgrade scripts failed if I had to do them on all databases at once - I got a couple large ones, around 50GB and 25GB respectively, and these might have been blocking the upgrade.

danblack avatar
dz flag
normally an upgrade doesn't need to look through all tables or database. 144ms of CPU time doesn't seem to correspond to any processing issue. A simple downgrade would have probably sufficed. It sounds definitely like it was a bug.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.