I'm running into an issue on an Ubunutu 20.04 server when upgrading from MySQL Server 8.0.32 to 8.0.33 that's causing the upgrade package to fail, leaving it only partially installed. After many hours of troubleshooting, I was able to find that this only occurs when running the 8.0.33 install on a server that has Ubuntu Advantage FIPS enabled. I'm not sure if this is a bug in FIPS or in MySQL.
Luckily, I have a snapshot that was taken right before the upgrade so I'm able to confirm that everything was working fine on 8.0.32, but as soon as I try to run
$ sudo apt-get --only-upgrade install mysql-server-8.0
It fails to load the MySQL server and gives me the following output:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
mysql-client-8.0 mysql-server-core-8.0
Suggested packages:
tinyca
The following packages will be upgraded:
mysql-client-8.0 mysql-server-8.0 mysql-server-core-8.0
3 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
Need to get 24.0 MB of archives.
After this operation, 323 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 mysql-client-8.0 amd64 8.0.33-0ubuntu0.20.04.2 [22.0 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 mysql-server-8.0 amd64 8.0.33-0ubuntu0.20.04.2 [1,321 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 mysql-server-core-8.0 amd64 8.0.33-0ubuntu0.20.04.2 [22.6 MB]
Fetched 24.0 MB in 2s (13.0 MB/s)
Preconfiguring packages ...
(Reading database ... 177434 files and directories currently installed.)
Preparing to unpack .../mysql-client-8.0_8.0.33-0ubuntu0.20.04.2_amd64.deb ...
Unpacking mysql-client-8.0 (8.0.33-0ubuntu0.20.04.2) over (8.0.32-0ubuntu0.20.04.2) ...
Preparing to unpack .../mysql-server-8.0_8.0.33-0ubuntu0.20.04.2_amd64.deb ...
Unpacking mysql-server-8.0 (8.0.33-0ubuntu0.20.04.2) over (8.0.32-0ubuntu0.20.04.2) ...
Preparing to unpack .../mysql-server-core-8.0_8.0.33-0ubuntu0.20.04.2_amd64.deb ...
Unpacking mysql-server-core-8.0 (8.0.33-0ubuntu0.20.04.2) over (8.0.32-0ubuntu0.20.04.2) ...
Setting up mysql-server-core-8.0 (8.0.33-0ubuntu0.20.04.2) ...
Setting up mysql-client-8.0 (8.0.33-0ubuntu0.20.04.2) ...
Setting up mysql-server-8.0 (8.0.33-0ubuntu0.20.04.2) ...
mysqld will log errors to /var/log/mysql/error.log
2023-05-15T18:39:17.482061Z 0 [ERROR] [MY-011065] [Server] Unable to determine if daemon is running: No such file or directory (rc=0).
2023-05-15T18:39:17.482074Z 0 [ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log.
Warning: Unable to start the server.
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
invoke-rc.d: initscript mysql, action "start" failed.
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2023-05-15 14:39:25 EDT; 69ms ago
Process: 3955 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 3964 ExecStart=/usr/sbin/mysqld (code=exited, status=2)
Main PID: 3964 (code=exited, status=2)
Status: "Server startup in progress"
May 15 14:39:25 iws.local systemd[1]: mysql.service: Scheduled restart job, restart counter is at 1.
May 15 14:39:25 iws.local systemd[1]: Stopped MySQL Community Server.
May 15 14:39:25 iws.local systemd[1]: Starting MySQL Community Server...
dpkg: error processing package mysql-server-8.0 (--configure):
installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
Processing triggers for systemd (245.4-4ubuntu3.21) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Errors were encountered while processing:
mysql-server-8.0
E: Sub-process /usr/bin/dpkg returned an error code (1)
Here's some of the output from the MySQL error.log:
2023-05-15T18:38:49.386464Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.32-0ubuntu0.20.04.2).
2023-05-15T18:38:50.302209Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.32-0ubuntu0.20.04.2) (Ubuntu).
2023-05-15T18:38:58.895566Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2023-05-15T18:38:58.919766Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.2) starting as process 3521
2023-05-15T18:38:59.071912Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-15T18:38:59.768432Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-15T18:39:04.501046Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.
2023-05-15T18:39:17.124578Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' completed.
2023-05-15T18:39:17Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=a2a809a7f24eea4fd1a805026e8bbcdde5d2cc2e
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55ad67d49a41]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39b) [0x55ad66bc6c5b]
/usr/sbin/mysqld(handle_fatal_signal+0xa5) [0x55ad66bc6d15]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6d19d83420]
/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_get_security_level+0x4) [0x7f6d1a185554]
/usr/sbin/mysqld(security_level()+0x23) [0x55ad66c07ce3]
/usr/sbin/mysqld(do_auto_cert_generation(ssl_artifacts_status, char const**, char const**, char const**)+0x57) [0x55ad66c0ed87]
/usr/sbin/mysqld(Ssl_init_callback_server_main::provision_certs()+0x72) [0x55ad66b4bfd2]
/usr/sbin/mysqld(TLS_channel::singleton_init(Ssl_acceptor_context_container**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Ssl_init_callback*, bool)+0x265) [0x55ad66b4a4a5]
/usr/sbin/mysqld(mysqld_main(int, char**)+0x3329) [0x55ad6694bb39]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f6d1984d083]
/usr/sbin/mysqld(_start+0x2e) [0x55ad6692ae8e]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2023-05-15T18:39:17.124579Z 0 [System] [MY-013951] [Server] 2023-05-15T18:39:17Z UTC - mysqld got signal 11 ;
2023-05-15T18:39:17.124580Z 0 [System] [MY-013951] [Server] Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
2023-05-15T18:39:17.124581Z 0 [System] [MY-013951] [Server] BuildID[sha1]=a2a809a7f24eea4fd1a805026e8bbcdde5d2cc2e
2023-05-15T18:39:17.124582Z 0 [System] [MY-013951] [Server] Thread pointer: 0x0
2023-05-15T18:39:17.124583Z 0 [System] [MY-013951] [Server] Attempting backtrace. You can use the following information to find out
2023-05-15T18:39:17.124584Z 0 [System] [MY-013951] [Server] where mysqld died. If you see no messages after this, something went
2023-05-15T18:39:17.124585Z 0 [System] [MY-013951] [Server] terribly wrong...
2023-05-15T18:39:17.124586Z 0 [System] [MY-013951] [Server] stack_bottom = 0 thread_stack 0x100000
2023-05-15T18:39:17.124587Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55ad67d49a41]
2023-05-15T18:39:17.124588Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(print_fatal_signal(int)+0x39b) [0x55ad66bc6c5b]
2023-05-15T18:39:17.124589Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(handle_fatal_signal+0xa5) [0x55ad66bc6d15]
2023-05-15T18:39:17.124590Z 0 [System] [MY-013951] [Server] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f6d19d83420]
2023-05-15T18:39:17.124591Z 0 [System] [MY-013951] [Server] /lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_get_security_level+0x4) [0x7f6d1a185554]
2023-05-15T18:39:17.124592Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(security_level()+0x23) [0x55ad66c07ce3]
2023-05-15T18:39:17.124593Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(do_auto_cert_generation(ssl_artifacts_status, char const**, char const**, char const**)+0x57) [0x55ad66c0ed87]
2023-05-15T18:39:17.124594Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(Ssl_init_callback_server_main::provision_certs()+0x72) [0x55ad66b4bfd2]
2023-05-15T18:39:17.124595Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(TLS_channel::singleton_init(Ssl_acceptor_context_container**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Ssl_init_callback*, bool)+0x265) [0x55ad66b4a4a5]
2023-05-15T18:39:17.124596Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(mysqld_main(int, char**)+0x3329) [0x55ad6694bb39]
2023-05-15T18:39:17.124597Z 0 [System] [MY-013951] [Server] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f6d1984d083]
2023-05-15T18:39:17.124598Z 0 [System] [MY-013951] [Server] /usr/sbin/mysqld(_start+0x2e) [0x55ad6692ae8e]
2023-05-15T18:39:17.124599Z 0 [System] [MY-013951] [Server] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
2023-05-15T18:39:17.124600Z 0 [System] [MY-013951] [Server] information that should help you find out what is causing the crash.
2023-05-15T18:39:20.616110Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
2023-05-15T18:39:20.618682Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.2) starting as process 3964
2023-05-15T18:39:20.623848Z 0 [System] [MY-013951] [Server] A backtrace was processed and added to the main error-log in the appropriate format.
2023-05-15T18:39:20.629675Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-15T18:39:24.817813Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-15T18:39:25.078058Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2023-05-15T18:39:25.088104Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2023-05-15T18:39:25Z UTC - mysqld got signal 11 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=a2a809a7f24eea4fd1a805026e8bbcdde5d2cc2e
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x561138fb9a41]
/usr/sbin/mysqld(print_fatal_signal(int)+0x39b) [0x561137e36c5b]
/usr/sbin/mysqld(handle_fatal_signal+0xa5) [0x561137e36d15]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f05f5574420]
/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_get_security_level+0x4) [0x7f05f5976554]
/usr/sbin/mysqld(security_level()+0x23) [0x561137e77ce3]
/usr/sbin/mysqld(do_auto_cert_generation(ssl_artifacts_status, char const**, char const**, char const**)+0x57) [0x561137e7ed87]
/usr/sbin/mysqld(Ssl_init_callback_server_main::provision_certs()+0x72) [0x561137dbbfd2]
/usr/sbin/mysqld(TLS_channel::singleton_init(Ssl_acceptor_context_container**, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, Ssl_init_callback*, bool)+0x265) [0x561137dba4a5]
/usr/sbin/mysqld(mysqld_main(int, char**)+0x3329) [0x561137bbbb39]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f05f503e083]
/usr/sbin/mysqld(_start+0x2e) [0x561137b9ae8e]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Running $ sudo systemctl status mysql returns the following:
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2023-05-15 14:39:37 EDT; 16min ago
Process: 4581 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Process: 4589 ExecStart=/usr/sbin/mysqld (code=exited, status=2)
Main PID: 4589 (code=exited, status=2)
Status: "Server startup in progress"
May 15 14:39:37 iws.local systemd[1]: mysql.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 15 14:39:37 iws.local systemd[1]: mysql.service: Failed with result 'exit-code'.
May 15 14:39:37 iws.local systemd[1]: Failed to start MySQL Community Server.
May 15 14:39:37 iws.local systemd[1]: mysql.service: Scheduled restart job, restart counter is at 7.
May 15 14:39:37 iws.local systemd[1]: Stopped MySQL Community Server.
May 15 14:39:37 iws.local systemd[1]: mysql.service: Start request repeated too quickly.
May 15 14:39:37 iws.local systemd[1]: mysql.service: Failed with result 'exit-code'.
May 15 14:39:37 iws.local systemd[1]: Failed to start MySQL Community Server.
Here's what I've tried so far. All have resulted in the same error:
- Rebooting the server several times.
- Stopping mysql and then running the upgrade.
- Stopping auditd service and then running the upgrade.
- Stopping apparmor and then running the upgrade.
- I returned to the snapshot with the working version 8.0.32, removed the mysql-server and mysql-server-8.0 packages, rebooted and then tried to install mysql-server. Still no luck.
- Starting a fresh install of Ubuntu Server 20.04 and only installed Ubuntu Advantage with FIPS enabled, and then tried to install mysql-server. Failed again.
- Installing everything except Ubuntu Advantage FIPS allows the MySQL 8.0.33 install without failure.
