Score:1

Mysql stopped working on Ubuntu 22.04. Failed to upgrade server. Error

jm flag

I have a basic DigitalOcean droplet (1gb ram and 1 cpu) that has a few websites on it. There are a couple nodejs sites and a couple of Wordpress sites. All are pretty low traffic. It normally works just fine but today none of the wordpress sites were working. So I logged in and noticed that the cpu usage was pegged at 100% and it normally is at like 2-3%. I restarted the server and the cpu usage went down. But the sites were still not working. So when I tried to access the sites it errors and says there is not database connection. So I checked the status of mysql and it was not started. I tried to restart the service but that just fails. Here is the result of sudo systemctl status mysql

mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2023-05-09 19:44:41 CDT; 48s ago
    Process: 3859 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 3867 (mysqld)
     Status: "Server upgrade in progress"
      Tasks: 28 (limit: 1115)
     Memory: 491.4M
        CPU: 5.561s
     CGroup: /system.slice/mysql.service
             └─3867 /usr/sbin/mysqld

May 09 19:44:41 sk-webservice.com systemd[1]: mysql.service: Scheduled restart job, restart counter is at 7.
May 09 19:44:41 sk-webservice.com systemd[1]: Stopped MySQL Community Server.
May 09 19:44:41 sk-webservice.com systemd[1]: mysql.service: Consumed 5.529s CPU time.
May 09 19:44:41 sk-webservice.com systemd[1]: Starting MySQL Community Server...

And then I get the following in /var/log/mysql/error.log

2023-05-10T00:41:48.072177Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.22.04.1) starting as process 3037
2023-05-10T00:41:48.081239Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-10T00:41:48.660106Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-10T00:41:49.429920Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.
2023-05-10T00:42:44.456107Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '-- Deprecate spatial reference systems -- Deprecation must happen before new SRSs are added/updated since there -- typically are new SRSs with the same names. REPLACE INTO mysql.st_spatial_reference_systems(id, catalog_id, name, organization, organization_coordsys_id, definition, description) VALUES (6956, 1, 'VN-2000 / TM-3 zone 481 (6956 deprecated)', 'EPSG', 6956, 'PROJCS["VN-2000 / TM-3 zone 481",GEOGCS["VN-2000",DATUM["Vietnam 2000",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],TOWGS84[-191.90441429,-39.30318279,-111.45032835,0.00928836,-0.01975479,0.00427372,0.252906278],AUTHORITY["EPSG","6756"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.017453292519943278,AUTHORITY["EPSG","9122"]],AXIS["Lat",NORTH],AXIS["Lon",EAST],AUTHORITY["EPSG","4756"]],PROJECTION["Transverse Mercator",AUTHORITY["EPSG","9807"]],PARAMETER["Latitude of natural origin",0,AUTHORITY["EPSG","8801"]],PARAMETER["Longitude of natural origin",102,AUTHORITY["EPSG","8802"]],PARAMETER["Scale factor at natural origin",0.9999,AUTHORITY["EPSG","8805"]],PARAMETER["False easting",0,AUTHORITY["EPSG","8806"]],PARAMETER["False northing",500000,AUTHORITY["EPSG","8807"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["E",EAST],AXIS["N",NORTH],AUTHORITY["EPSG","6956"]]', 'Deprecated since EPSG Dataset 9.3. Superseded by SRID 5896.'); ' failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.
2023-05-10T00:42:44.463355Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.
2023-05-10T00:42:44.463830Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-10T00:42:45.325713Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.04.1)  (Ubuntu).

So it seems like mysql was trying to upgrade and just failed? I don't remember running any upgrades recently. This is just a personal server and I am by no means a wiz on the server side of things. Does anyone know how I can fix this?

ng flag
The error at the end of your long string is `Lock wait timeout exceeded; try restarting transaction`. That's unusual on its own upgrade process. I can't readily find anything about this elsewhere. You may need to restore a backup...
shodanshok avatar
ca flag
Please add the output of `grep -i mysql var/log/apt/history.log`
Steve K avatar
jm flag
@shodanshok I ended up just uninstalling and reinstalling mysql. I have no Idea what happened. I am about 100mb away from my ram limit most of the time so I wonder if maybe I ran out of ram and that caused some kind of issue and then when I rebooted something got corrupted. I have no idea.
Wilson Hauck avatar
jp flag
@SteveK Digital Ocean may have attempted to upgrade your instance - check with DO support team.
Score:0
gb flag

I got this same issue today and this is How I solved this.

  1. First of all I stopped mysql: service mysql stop

  2. After stopping mysql I ran MYSQLD with upgrade: /usr/sbin/mysqld --upgrade=minimal Make sure upgrade argument is minimal.

  3. Now if you get error like unable to create sock file then you can create folder like /var/run/mysqld

  4. Do check error logs in a seperate terminal while doing this: tail -f /var/log/mysql/error.log

  5. Now you will see mysqld running. Finally you can DUMP all the database by this cmd:

for DB in $(mysql -u root -pPassword -e 'show databases' -s --skip-column-names); do mysqldump -u root -pPassword $DB > "$DB.sql"; done

  1. Now you can uninstall mysql and install a stable version and Import all the databases.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.