Mariadb instance won't start, what are some troubleshooting steps?

Greg

2/14/24, 11:04 PM

An existing mariadb instance that has existed for a number of years and been through multiple upgrades over the years will not start anymore. Running on Linux Mint I just noticed that it was down and not exactly sure when the problem started.

    systemctl status mariadb.service
    mariadb.service - MariaDB 10.1.48 database server
      Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Active: failed (Result: exit-code) since Tue 2023-02-14 16:22:28 EST; 1h 25min ago
        Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
    Process: 4815 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
    Process: 4741 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCES
    Process: 4739 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 4738 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 4815 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Feb 14 16:22:24 LM18-Virtual-Machine systemd[1]: Starting MariaDB 10.1.48 database server...
Feb 14 16:22:25 LM18-Virtual-Machine mysqld[4815]: 2023-02-14 16:22:25 3078166144 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0ubuntu0.18.04.1) starting as process 4815 ...
Feb 14 16:22:28 LM18-Virtual-Machine systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 16:22:28 LM18-Virtual-Machine systemd[1]: mariadb.service: Failed with result 'exit-code'.
Feb 14 16:22:28 LM18-Virtual-Machine systemd[1]: Failed to start MariaDB 10.1.48 database server.

There is plenty of diskspace on the partition. One of the tables is 15GB but that is far below max size.

Plenty of free memory on the machine.

Following is the result of a journalctl -xe after a failed attempt to start the service.

-- Unit mariadb.service has begun starting up.
Feb 14 18:05:44 LM18-Virtual-Machine mysqld[5884]: 2023-02-14 18:05:44 3078329984 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0ubuntu0.18.04.1) starting as process 5884 ...
Feb 14 18:05:47 LM18-Virtual-Machine systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 18:05:47 LM18-Virtual-Machine systemd[1]: mariadb.service: Failed with result 'exit-code'.
Feb 14 18:05:47 LM18-Virtual-Machine systemd[1]: Failed to start MariaDB 10.1.48 database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit mariadb.service has failed.
--
-- The result is RESULT.

The result is RESULT is a really helpful error code.

What should I look at next to ascertain the problem?

285

1 + 0

mysql

mariadb

Score:1

Server

Unai

2/15/24, 9:37 AM

The next thing I'd do is look at the error log (more info). This might be at /var/log/mysql/error.log or /var/log/mysql/mariadb.err It's possible that it's not set (and you need to set it). To know where it is (and assuming you cannot run queries because you cannot start the database) you could do something like this:

find /etc/mysql -type f -exec grep log_error {} \;

On my server it looks like this:

# find /etc/mysql -type f -exec grep log_error {} \;
log_error = /var/log/mysql/error.log
skip_log_error

So I know that it is on /var/log/mysql/error.log

If this error log is not explicit enough the next thing I'd do is start the database using mysqld_safe or mariadbd-safe (more info). On my server it looks like this:

# mysqld_safe
230215 09:23:28 mysqld_safe Logging to syslog.
230215 09:23:28 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

And if I check syslog:

# grep mysqld /var/log/syslog
Feb 15 09:23:28 birch mysqld_safe: Starting mariadbd daemon with databases from /var/lib/mysql
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-0+deb11u1-log) starting as process 2035918 ...
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Uses event mutexes
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Number of pools: 1
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Using Linux native AIO
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Completed initialization of buffer pool
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: 128 rollback segments are active.
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: 10.5.12 started; log sequence number 134979417; transaction id 120284
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] Plugin 'FEEDBACK' is disabled.
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] Server socket created on IP: '0.0.0.0'.
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] Reading of all Master_info entries succeeded
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] Added new Master_info '' to hash table
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] /usr/sbin/mariadbd: ready for connections.
Feb 15 09:23:28 birch mysqld: Version: '10.5.12-MariaDB-0+deb11u1-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Feb 15 09:23:28 birch mysqld: 2023-02-15  9:23:28 0 [Note] InnoDB: Buffer pool(s) load completed at 230215  9:23:28
Feb 15 09:24:20 birch mysqld: 2023-02-15  9:24:20 4 [Note] Start binlog_dump to slave_server(3), pos(mysql-bin.000006, 342), using_gtid(0), gtid('')

To stop mysqld_safe you'd need to run:

# mysqladmin -p shutdown
Enter password:

+ 1

Greg

2/16/24, 7:47 PM

Error.log showed a reference to the tc.log file. It was a 0 byte size file so I deleted it and restarted the service and it worked.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Mariadb instance won't start, what are some troubleshooting steps?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.