mysql NDB backup fail

Julien

4/9/24, 9:27 AM

I'm trying to abort a backup on a NDB cluster like this:

ndb_mgm> abort backup 2304081512
Abort of backup 2304081512 ordered

But there is no change, the backup seems to be stalled :

ndb_mgm> all report backup
Node 2: Backup not started
Node 3: Local backup status: backup 2304081512 started from node 12
 #Records: 1950 #LogRecords: 8705
 Data: 635312 bytes Log: 2676612 bytes
Node 4: Local backup status: backup 2304081512 started from node 12
 #Records: 1768 #LogRecords: 7211
 Data: 522256 bytes Log: 2133088 bytes

An if I try to start a new one, I've got an error because a backup is already running (2304081512):

ndb_mgm> start backup 2304091121 snapshotstart
Connected to Management Server at: 192.168.169.40:1186
Waiting for completed, this may take several minutes
Node 3: Backup request from 12 failed to start. Error: 1302
Backup failed
*  3001: Could not start backup
*        A backup is already running: Permanent error: Application error

All the nodes seem to be okay excepted an anoying message I got on the Node 2 NDB engine:

2023-04-07 10:31:57 [ndbd] ALERT    -- Node 2: Forced node shutdown completed. Caused by error 2315: 'Node declared dead. See error log for details(Arbitration error). Temporary error, restart node'.

I really don't know how to get out of this and I need help

Best regards Julien

1 + 0

mysql

backup

cluster

Score:0

Server

Julien

7/20/24, 7:53 AM

I had to perform a rolling update on that cluster this weekend in order to upgrade the vms to debian 12 (bookworm). NDB backups were stopped for 3 months and just for curiosity, I ran one after the update and... surprise, it worked !

I really don't know what happened but performing the following tasks on each node one by one solved the issue:

stop vrrp to force requests on another node.
stop local ndb and mysqld engines
stop local ndb-master
updgrade operating system to latest bullseye
upgrade operating system to bookworm
reboot vm
start ndb, mysqld and ndb-master
control logs and wait for the replication to occur
if all is okay, switch to the next node

I don't think this answer is acceptable but it may help someone in the same situation.

(Note that the blocking backup 2304081512 disapeared from all the nodes but I don't know when)

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: mysql NDB backup fail

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.