Score:1

Why does MySQL 5.7 inside a docker container always crash?

cn flag

I have a MySQL 5.7 docker container that was provisioned via a docker-compose file, the file looks like this:

version: '3'

# Run command: docker compose --env-file .env.local up
services:
  # Database
  db_hm:
    image: mysql:5.7
    volumes:
      - db_data_hm:/var/lib/mysql
    restart: always
    ports:
      - '127.0.0.1:3316:3306'
    environment:
      MYSQL_ROOT_PASSWORD: ${HD_DB_PASSWORD}
      MYSQL_DATABASE: ${HD_DB_NAME}
      MYSQL_USER: ${HD_DB_USER}
      MYSQL_PASSWORD: ${HD_DB_PASSWORD}
    networks:
      - wpsite_hm
  # Wordpress
  wordpress_hm:
    depends_on:
      - db_hm
    build:
      context: ./docker/wordpress
      dockerfile: ./Dockerfile
    read_only: false
    ports:
      - '8001:80'
    expose:
      - 80
    restart: always
    volumes: 
        - './:/var/www/html'
        - './php.ini:/usr/local/etc/php/php.ini'
    environment:
      HDB_DB_HOST: ${HD_DB_HOST}
      HDB_DB_USER: ${HD_DB_USER}
      HDB_DB_NAME: ${HD_DB_NAME}
      HDB_DB_PASSWORD: ${HD_DB_PASSWORD}
    networks:
      - wpsite_hm

volumes:
  db_data_hm:

I have a stress test script created via JMeter. Whenever I run the test using 30 concurrent users, the MySQL container always log something like this:

2023-01-28T19:23:59.927259Z 0 [Note] mysqld: ready for connections.
 Version: '5.7.40'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)
 2023-01-28 19:24:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.40-1.el7 started.
 2023-01-28 19:24:20+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
 2023-01-28 19:24:20+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.40-1.el7 started.
 '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
 2023-01-28T19:24:21.595285Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
 2023-01-28T19:24:21.600648Z 0 [Note] mysqld (mysqld 5.7.40) starting as process 1 ...
 2023-01-28T19:24:21.659734Z 0 [Note] InnoDB: PUNCH HOLE support available
 2023-01-28T19:24:21.660855Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
 2023-01-28T19:24:21.660865Z 0 [Note] InnoDB: Uses event mutexes
 2023-01-28T19:24:21.660870Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
 2023-01-28T19:24:21.660875Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.12
 2023-01-28T19:24:21.660888Z 0 [Note] InnoDB: Using Linux native AIO
 2023-01-28T19:24:21.664699Z 0 [Note] InnoDB: Number of pools: 1
 2023-01-28T19:24:21.668572Z 0 [Note] InnoDB: Using CPU crc32 instructions
 2023-01-28T19:24:21.673989Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
 2023-01-28T19:24:21.687345Z 0 [Note] InnoDB: Completed initialization of buffer pool
 2023-01-28T19:24:21.691938Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
 2023-01-28T19:24:21.715903Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
 2023-01-28T19:24:21.723552Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 4363179685
 2023-01-28T19:24:21.723590Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 4363179694
 2023-01-28T19:24:21.723597Z 0 [Note] InnoDB: Database was not shutdown normally!
 2023-01-28T19:24:21.723601Z 0 [Note] InnoDB: Starting crash recovery.
 2023-01-28T19:24:21.989305Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 2023-01-28T19:24:21.989429Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 2023-01-28T19:24:21.989781Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 2023-01-28T19:24:22.011166Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
 2023-01-28T19:24:22.012602Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
 2023-01-28T19:24:22.012633Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
 2023-01-28T19:24:22.013269Z 0 [Note] InnoDB: Waiting for purge to start
 2023-01-28T19:24:22.064048Z 0 [Note] InnoDB: 5.7.40 started; log sequence number 4363179694
 2023-01-28T19:24:22.065063Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
 2023-01-28T19:24:22.072885Z 0 [Note] Plugin 'FEDERATED' is disabled.
 2023-01-28T19:24:22.088705Z 0 [Note] InnoDB: Buffer pool(s) load completed at 230128 19:24:22
 2023-01-28T19:24:22.127157Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
 2023-01-28T19:24:22.127199Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
 2023-01-28T19:24:22.127213Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
 2023-01-28T19:24:22.127219Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
 2023-01-28T19:24:22.134621Z 0 [Warning] CA certificate ca.pem is self signed.
 2023-01-28T19:24:22.136068Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
 2023-01-28T19:24:22.153990Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
 2023-01-28T19:24:22.155336Z 0 [Note] IPv6 is available.
 2023-01-28T19:24:22.155380Z 0 [Note]   - '::' resolves to '::';
 2023-01-28T19:24:22.155429Z 0 [Note] Server socket created on IP: '::'.
 2023-01-28T19:24:22.175855Z 0 [Warning] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
 2023-01-28T19:24:22.331036Z 0 [Note] Event Scheduler: Loaded 0 events
 2023-01-28T19:24:22.334317Z 0 [Note] mysqld: ready for connections.

Note that, there are these 2 lines:

 2023-01-28T19:24:21.723597Z 0 [Note] InnoDB: Database was not shutdown normally!
 2023-01-28T19:24:21.723601Z 0 [Note] InnoDB: Starting crash recovery.

My server has 8 vCPU and 16 GB of RAM. But, during the test the CPU utilization became so high, it even reached more than 200%.

I'd searched for answers regarding this kind of issue, most were related to insufficient memory allocation. But during the test, the memory utilization was always below 40%. Other possibilities would be a corrupt database or something that killed the MySQL instance, but unfortunately, I can't see any message related to that.

Any kind of help will be much appreciated.

HBruijn avatar
in flag
I’m unclear what you’re asking. You have a startup log that ends with `[Note] mysqld: ready for connections.` which looks good to me.
Budianto IP avatar
cn flag
@HBruijin, the log says: ``` Database was not shutdown normally! Starting crash recovery. ```
djdomi avatar
za flag
a cpu utilization of load level 2 of 8 is still fine. and the sentence is a warning for you, from the last start, does the server works?
Budianto IP avatar
cn flag
The MySQL container did not crash during the test, but the MySQL service inside the container crashed.
Arrow Root avatar
sm flag
Is there any clue on `/var/log/messages` from inside the container?
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.