Score:0

Restore MySQL database is very slow

ar flag

I've just bought a new home desktop. I have three MySQL databases that I want to move across. I dumped them all using mysqldump before removing the drive from the old machine.

Restoring the largest of the databases is taking ages (getting on for 24 hours so far). I'm restoring via:

mysql -uxxxx -p
Enter password: yyyy
mysql> create database foo;
mysql> use foo;
mysql> start transaction;
mysql> \. <dump-file>

I pray I remember to commit when this finishes!

I abandoned my first attempt to restore, without the transaction as it was taking so long. I intend not to bother with the transaction when restoring the other two databases as it seems to make no difference.

Is there anything I can do to speed up the restore? Should I have done the backup differently - I just used a vanilla mysqldump command line, with --skip-column-statistics since the dump would otherwise fail, as proposed in this stackoverflow answer.

For reference, the new machine runs Windows 10 and MySQL 8.0.25 Community; the old machine ran Windows 7 and MySQL 5.6.22.

Looking at the dump files more closely (to find the server version), I see that the tables are created with their indexes, followed by a lot of INSERT INTO statements. This doesn't seem particularly clever. Is there a way to induce mysqldump to produce a script that:

  1. Creates the tables without indexes
  2. Inserts the rows; and then
  3. Creates the indexes.

which would seem a better approach for when the database is being restored. (It's a little surprising that this isn't the default!) I assume this is possible with MySQL - I've been using SQLLite for the past 3 years and it's certainly possible there.

Update

This question isn't completely answered by How can I speed up a MySQL restore from a dump file?, since that question particularly refers to MyISAM tables, and mine are InnoDB. (Apologies. I probably should have mentioned the storage engine in my original question.)

That answer did, however motivate and inform my attempts to resolve matters, which I document in my own answer below.

nurdglaw avatar
ar flag
Hmm. It might well do. I'll have to take a closer look. For what it's worth, the restore has now been running for ~24 hours. Judging by the `AUTO_INCREMENT=` values in the dump file, the three tables have, respectively 2.6M, 250k and 8.5M rows. It's currently restoring the third table. I haven't checked properly how many rows are insrted with each INSERT INTO command, but each now takes 3-4 minutes to complete. This certainly feels like an indexing problem. I have no foreign keys, which are the focus of the referenced article, but AUTOCOMMIT looks a likely culprit. Thanks!
nurdglaw avatar
ar flag
I'm tempted to wait out this restore and look into modifying the dumps of the other two (much smaller) databases to create the indexes only after completing all the INSERTs. I'll then try repeating the restore for the big db with those changes.
nurdglaw avatar
ar flag
@MichaelHampton PPS: If you make your comment into an answer, I'll certainly upvote it (if I have sufficient rep.) Thanks again.
Michael Hampton avatar
cz flag
If it solved your problem, then just click the button above.
nurdglaw avatar
ar flag
@Laura Elvira Hernández Lara - Thanks for proposing an edit. Please forgive me for rejecting your suggestion and replacing the "Thanks in advance" for what (I think) I originally intended to put in there. Heaven knows why I thought "Thanks in advance" was an appropriate placeholder.
ua flag
Unless you are running 8.0, your concerns about `START` and `COMMIT` are irrelevant. DDL statements implicitly `COMMIT`. So, a crash will leave some tables loaded, some not.
Score:0
ar flag

I eventually gave up waiting for the import to complete, after >24 hours.

I modified the scripts generated by mysqldump so as to create the tables without secondary indexes, and to create the indexes only when all the INSERT statements had been executed. With this change, the import had completed when I checked three hours after starting.

I then further modified the script to insert the rows within a transaction, by adding SET AUTOCOMMIT=0; START TRANSACTION; at the start, and COMMIT; at the end. With this change, the restore completed in 90 minutes.

While editing the generated dumps, I noticed that they include

/*!40000 ALTER TABLE `xxx` DISABLE KEYS */;

and

/*!40000 ALTER TABLE `xxx` ENABLE KEYS */;

surrounding the INSERT statements. I looked up the documentation on these commands and discovered that they only apply to MyISAM tables. The documentation on the mysqldump utility states that these lines (including the comment delimiters and the magic number 40000) are generated if you specify --disable-keys on the command line. The utility documentation also states that it will add the "insert everything in a single transaction" behaviour that I found desirable if you specify --no-autocommit on the command line.


In summary, I should probably have taken the backups with

 mysqldump --no-autocommit...

At least in my case (mysqldump version 8.0.25 and an InnoDb table) the --disable-keys option makes no difference; ALTER TABLE ... ENABLE|DISABLE KEYS statements (commented out, and with a magic number) are always generated. I do need to modify the generated dump to create the tables without secondary keys, and to add them once all the INSERT INTO statements have been executed.

nurdglaw avatar
ar flag
Why the downvote / delete vote, please?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.