Score:1

Best practice for stopping Apache during backups

gu flag
QF0

During backups, I need to 'idle' Apache, so that I can stop MySQL, and then back up MySQL and SQLite databases. The only access to the databases is through Apache.

Currently, I'm thinking of doing this with a script that modifies the Apache config to redirect all accesses to a 'site down' page, and then:

apache2ctl graceful
systemctl stop mysqld

However, this doesn't seem like a particularly good idea. graceful doesn't stop all the child workers, and presumably apache2ctl returns immediately. This is a very low-volume server, and the backup runs in the middle of the night, in one time zone, and I'm pretty sure that no request will last longer than a minute.

Is there a 'good' way to handle this, instead of just putting in a sleep 60 after the graceful? I'm thinking that it may be a better idea to do a graceful-stop and not bother with a site down page, set GracefulShutdownTimeout to 60s, and sleep for 60s as well.

John Mahowald avatar
cn flag
Which service manager are you using to mange web and DB processes? systemctl implies systemd Linux, however you also are using apache2ctl directly.
QF0 avatar
gu flag
QF0
@JohnMahowald: yes, systemd on Ubuntu 22.04. I would normally use `systemctl` but all the Apache and Ubuntu docs talk about `apache2ctl` - does it matter?
Score:1
cn flag

Downtime procedures are relatively easy when just shutting it down for a long window is acceptable. More or less what you are doing.

Prior to downtime, while its still running, consider a reminder on the site that it has maintenance. Organizations vary in what they expect for this, some systems have a banner that its going down tonight, some have a separate maintenance calendar.

As the web server is being shut down, there is not much of an opportunity to show a maintenance page. Probably not bother for now. Maybe in the future, this will be behind a load balancer, and that can show a static down page while connections are being drained.

Where available, use the same service manager to start and stop all applications. Using one tool allows consistent reporting of status, dependencies, handling cleanup of failed stops, and other features. Confusingly, not every distro likes using the apachectl program, but that's fine because how httpd stops via signals is well documented.

Stop web and then database server. Rely on the service manager, in this case systemd units, to provide relatively graceful shutdowns, similar to apachectl graceful-stop. Read the units for the details of how they are implemented.

# Stop web first to stop application from accessing data
systemctl stop apache2
# A long wait here is likely not necessary
# the service manager is shutting down processes, and the database is going away soon as well

# Stop one at a time to ensure done in this order
systemctl stop mysqld
# Wait a bit for database files to be closed, just in case
# FIXME this delay is arbitrary
sleep 10

If you take backups by copying the live database files, take care that no processes still access them. Copying files still being written to has a risk of the copies being corrupt and a bad backup. During normal situations, a graceful stop will quickly prevent new requests. And under light load, clean up and shut down work finishes fast. The tricky part is ensuring this is true even under extraordinary circumstances.

A careful backup could check that nothing has the files open, such as fuser in a script. Possibly also sending a signal to any remaining processes, like a SIGTERM.

Requiring downtime, plus a little paranoia about ensuring the files are closed, makes this file copy backup method not as easy as it might seem. Consider alternative online backup methods, like using the database's built in methods to hot copy their data or dump exports to files.

For completeness, another option is consistent, online, storage level snapshots and copy those files. Also avoids the downtime, but can be even trickier as the database was never shut down. Doing this safely means pausing writes, and understanding how databases do startup recovery.

Test restores occasionally. Actually restore the data to some staging database in a test environment, start it up, and spot check data is correct.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.