For the past few days, my three servers have been experiencing the same problem almost at the same time, about every 24 hours.
I haven't changed anything in the code using MySQL or ubuntu configuration for months.
What I can see in the error logs (/var/log/mysql/error.log) is that apparently MySQL stops by itself for no reason:
2022-02-10T06:48:17.246813Z 0 [Note] Giving 10 client threads a chance to die gracefully
2022-02-10T06:48:17.246853Z 0 [Note] Shutting down slave threads
2022-02-10T06:48:19.246990Z 0 [Note] Forcefully disconnecting 10 remaining clients
2022-02-10T06:48:19.247028Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 7 user: 'root'
2022-02-10T06:48:19.249583Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 10 user: 'root'
2022-02-10T06:48:19.249613Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 8 user: 'root'
2022-02-10T06:48:19.249628Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 51 user: 'root'
2022-02-10T06:48:19.249641Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 3 user: 'root'
2022-02-10T06:48:19.249654Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 2 user: 'root'
2022-02-10T06:48:19.249667Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 4 user: 'root'
2022-02-10T06:48:19.249680Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 5 user: 'root'
2022-02-10T06:48:19.249693Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 6 user: 'root'
2022-02-10T06:48:19.249705Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 9 user: 'root'
2022-02-10T06:48:19.249723Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-02-10T06:48:19.252212Z 0 [Note] Binlog end
2022-02-10T06:48:19.254111Z 0 [Note] Shutting down plugin 'ngram'
2022-02-10T06:48:19.254125Z 0 [Note] Shutting down plugin 'partition'
2022-02-10T06:48:19.254130Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2022-02-10T06:48:19.254135Z 0 [Note] Shutting down plugin 'ARCHIVE'
2022-02-10T06:48:19.254139Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2022-02-10T06:48:19.254168Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2022-02-10T06:48:19.254173Z 0 [Note] Shutting down plugin 'MyISAM'
2022-02-10T06:48:19.254180Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2022-02-10T06:48:19.254184Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2022-02-10T06:48:19.254188Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2022-02-10T06:48:19.254191Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2022-02-10T06:48:19.254195Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2022-02-10T06:48:19.254198Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2022-02-10T06:48:19.254202Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2022-02-10T06:48:19.254206Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2022-02-10T06:48:19.254209Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2022-02-10T06:48:19.254213Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2022-02-10T06:48:19.254216Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2022-02-10T06:48:19.254220Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2022-02-10T06:48:19.254223Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2022-02-10T06:48:19.254227Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2022-02-10T06:48:19.254230Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2022-02-10T06:48:19.254234Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2022-02-10T06:48:19.254237Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2022-02-10T06:48:19.254241Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2022-02-10T06:48:19.254244Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2022-02-10T06:48:19.254248Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2022-02-10T06:48:19.254251Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2022-02-10T06:48:19.254255Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2022-02-10T06:48:19.254258Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2022-02-10T06:48:19.254262Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2022-02-10T06:48:19.254265Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2022-02-10T06:48:19.254269Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2022-02-10T06:48:19.254272Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2022-02-10T06:48:19.254276Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2022-02-10T06:48:19.254279Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2022-02-10T06:48:19.254295Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2022-02-10T06:48:19.254299Z 0 [Note] Shutting down plugin 'InnoDB'
2022-02-10T06:48:19.254402Z 0 [Note] InnoDB: FTS optimize thread exiting.
2022-02-10T06:48:19.254552Z 0 [Note] InnoDB: Starting shutdown...
2022-02-10T06:48:19.354843Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2022-02-10T06:48:19.356501Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 220210 6:48:19
2022-02-10T06:48:20.819810Z 0 [Note] InnoDB: Shutdown completed; log sequence number 36134591308
2022-02-10T06:48:20.820191Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-02-10T06:48:20.820204Z 0 [Note] Shutting down plugin 'MEMORY'
2022-02-10T06:48:20.820215Z 0 [Note] Shutting down plugin 'CSV'
2022-02-10T06:48:20.820222Z 0 [Note] Shutting down plugin 'sha256_password'
2022-02-10T06:48:20.820225Z 0 [Note] Shutting down plugin 'mysql_native_password'
2022-02-10T06:48:20.820358Z 0 [Note] Shutting down plugin 'binlog'
2022-02-10T06:48:20.821311Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
2022-02-10T06:48:21.370898Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-02-10T06:48:21.372832Z 0 [Note] mysqld (mysqld 5.7.37-0ubuntu0.18.04.1) starting as process 8693 ...
2022-02-10T06:48:21.377512Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-02-10T06:48:21.377533Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-02-10T06:48:21.377538Z 0 [Note] InnoDB: Uses event mutexes
2022-02-10T06:48:21.377543Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-02-10T06:48:21.377547Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-02-10T06:48:21.377551Z 0 [Note] InnoDB: Using Linux native AIO
2022-02-10T06:48:21.377803Z 0 [Note] InnoDB: Number of pools: 1
2022-02-10T06:48:21.377915Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-02-10T06:48:21.379635Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2022-02-10T06:48:21.447914Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-02-10T06:48:21.463713Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-02-10T06:48:21.484266Z 0 [Note] InnoDB: Opened 3 undo tablespaces
2022-02-10T06:48:21.484292Z 0 [Note] InnoDB: 0 undo tablespaces made active
2022-02-10T06:48:21.484511Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-02-10T06:48:21.520917Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-02-10T06:48:21.521030Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-02-10T06:48:21.552476Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-02-10T06:48:21.552668Z 0 [Warning] InnoDB: The parameter innodb_undo_logs is deprecated and may be removed in future releases. Please use innodb_rollback_segments instead. See http://dev.mysql.com/doc/refman/5.7/en/innodb-undo-logs.html
2022-02-10T06:48:21.553292Z 0 [Note] InnoDB: 1 redo rollback segment(s) found. 1 redo rollback segment(s) are active.
2022-02-10T06:48:21.553304Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2022-02-10T06:48:21.553598Z 0 [Note] InnoDB: Waiting for purge to start
2022-02-10T06:48:21.603836Z 0 [Note] InnoDB: 5.7.37 started; log sequence number 36134591308
2022-02-10T06:48:21.604219Z 0 [Note] Plugin 'FEDERATED' is disabled.
2022-02-10T06:48:21.604678Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-02-10T06:48:21.606857Z 0 [Warning] InnoDB: Cannot open table mysql/gtid_executed from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
mysqld: Table 'mysql.gtid_executed' doesn't exist
2022-02-10T06:48:21.606890Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2022-02-10T06:48:21.608219Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2022-02-10T06:48:21.608233Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2022-02-10T06:48:21.608238Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2022-02-10T06:48:21.608241Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2022-02-10T06:48:21.611484Z 0 [Warning] CA certificate ca.pem is self signed.
2022-02-10T06:48:21.611520Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2022-02-10T06:48:21.614815Z 0 [Warning] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2022-02-10T06:48:21.615451Z 0 [Warning] InnoDB: Cannot open table mysql/server_cost from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T06:48:21.615470Z 0 [Warning] Failed to open optimizer cost constant tables
2022-02-10T06:48:21.616297Z 0 [Warning] InnoDB: Cannot open table mysql/time_zone_leap_second from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T06:48:21.616314Z 0 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
2022-02-10T06:48:21.616955Z 0 [Warning] InnoDB: Cannot open table mysql/servers from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T06:48:21.617005Z 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2022-02-10T06:48:21.682478Z 0 [Note] mysqld: ready for connections.
Version: '5.7.37-0ubuntu0.18.04.1' socket: '/tmp/tmp.q6ChLS0FIM/mysqld.sock' port: 0 (Ubuntu)
2022-02-10T06:48:21.747960Z 3 [Warning] InnoDB: Cannot open table mysql/plugin from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T06:48:21.751266Z 0 [Note] Giving 0 client threads a chance to die gracefully
2022-02-10T06:48:21.751284Z 0 [Note] Shutting down slave threads
2022-02-10T06:48:21.751291Z 0 [Note] Forcefully disconnecting 0 remaining clients
2022-02-10T06:48:21.751331Z 0 [Note] Binlog end
2022-02-10T06:48:21.751679Z 0 [Note] Shutting down plugin 'ngram'
2022-02-10T06:48:21.751688Z 0 [Note] Shutting down plugin 'partition'
2022-02-10T06:48:21.751692Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2022-02-10T06:48:21.751696Z 0 [Note] Shutting down plugin 'ARCHIVE'
2022-02-10T06:48:21.751699Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2022-02-10T06:48:21.751726Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2022-02-10T06:48:21.751730Z 0 [Note] Shutting down plugin 'MyISAM'
2022-02-10T06:48:21.751736Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2022-02-10T06:48:21.751740Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2022-02-10T06:48:21.751743Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2022-02-10T06:48:21.751747Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2022-02-10T06:48:21.751750Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2022-02-10T06:48:21.751753Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2022-02-10T06:48:21.751757Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2022-02-10T06:48:21.751760Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2022-02-10T06:48:21.751771Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2022-02-10T06:48:21.751775Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2022-02-10T06:48:21.751778Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2022-02-10T06:48:21.751782Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2022-02-10T06:48:21.751785Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2022-02-10T06:48:21.751788Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2022-02-10T06:48:21.751792Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2022-02-10T06:48:21.751795Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2022-02-10T06:48:21.751798Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2022-02-10T06:48:21.751802Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2022-02-10T06:48:21.751805Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2022-02-10T06:48:21.751808Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2022-02-10T06:48:21.751812Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2022-02-10T06:48:21.751815Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2022-02-10T06:48:21.751818Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2022-02-10T06:48:21.751822Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2022-02-10T06:48:21.751825Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2022-02-10T06:48:21.751828Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2022-02-10T06:48:21.751832Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2022-02-10T06:48:21.751835Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2022-02-10T06:48:21.751838Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2022-02-10T06:48:21.751842Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2022-02-10T06:48:21.751845Z 0 [Note] Shutting down plugin 'InnoDB'
2022-02-10T06:48:21.751876Z 0 [Note] InnoDB: FTS optimize thread exiting.
2022-02-10T06:48:21.751974Z 0 [Note] InnoDB: Starting shutdown...
2022-02-10T06:48:21.752015Z 0 [Note] InnoDB: Buffer pool(s) load completed at 220210 6:48:21
2022-02-10T06:48:21.752037Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2022-02-10T06:48:21.752203Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 220210 6:48:21
2022-02-10T06:48:23.180121Z 0 [Note] InnoDB: Shutdown completed; log sequence number 36134591327
2022-02-10T06:48:23.181629Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-02-10T06:48:23.181644Z 0 [Note] Shutting down plugin 'MEMORY'
2022-02-10T06:48:23.181651Z 0 [Note] Shutting down plugin 'CSV'
2022-02-10T06:48:23.181656Z 0 [Note] Shutting down plugin 'sha256_password'
2022-02-10T06:48:23.181660Z 0 [Note] Shutting down plugin 'mysql_native_password'
2022-02-10T06:48:23.181788Z 0 [Note] Shutting down plugin 'binlog'
2022-02-10T06:48:23.182138Z 0 [Note] mysqld: Shutdown complete
When I start MySQL, this is what the logs say:
2022-02-10T07:00:47.819747Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2022-02-10T07:00:47.821725Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.37-0ubuntu0.18.04.1) starting as process 8871 ...
2022-02-10T07:00:47.826117Z 0 [Note] InnoDB: PUNCH HOLE support available
2022-02-10T07:00:47.826140Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-02-10T07:00:47.826145Z 0 [Note] InnoDB: Uses event mutexes
2022-02-10T07:00:47.826150Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2022-02-10T07:00:47.826155Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-02-10T07:00:47.826160Z 0 [Note] InnoDB: Using Linux native AIO
2022-02-10T07:00:47.826411Z 0 [Note] InnoDB: Number of pools: 1
2022-02-10T07:00:47.826520Z 0 [Note] InnoDB: Using CPU crc32 instructions
2022-02-10T07:00:47.829890Z 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2022-02-10T07:00:47.900431Z 0 [Note] InnoDB: Completed initialization of buffer pool
2022-02-10T07:00:47.916391Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-02-10T07:00:47.928336Z 0 [Note] InnoDB: Opened 3 undo tablespaces
2022-02-10T07:00:47.928362Z 0 [Note] InnoDB: 0 undo tablespaces made active
2022-02-10T07:00:47.928568Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2022-02-10T07:00:47.958351Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-02-10T07:00:47.958418Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-02-10T07:00:47.988870Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-02-10T07:00:47.989095Z 0 [Warning] InnoDB: The parameter innodb_undo_logs is deprecated and may be removed in future releases. Please use innodb_rollback_segments instead. See http://dev.mysql.com/doc/refman/5.7/en/innodb-undo-logs.html
2022-02-10T07:00:47.989671Z 0 [Note] InnoDB: 1 redo rollback segment(s) found. 1 redo rollback segment(s) are active.
2022-02-10T07:00:47.989680Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2022-02-10T07:00:47.989974Z 0 [Note] InnoDB: Waiting for purge to start
2022-02-10T07:00:48.040194Z 0 [Note] InnoDB: 5.7.37 started; log sequence number 36134591327
2022-02-10T07:00:48.040548Z 0 [Note] Plugin 'FEDERATED' is disabled.
2022-02-10T07:00:48.041907Z 0 [Warning] InnoDB: Cannot open table mysql/plugin from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
mysqld: Table 'mysql.plugin' doesn't exist
2022-02-10T07:00:48.041936Z 0 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2022-02-10T07:00:48.042370Z 0 [Warning] InnoDB: Cannot open table mysql/gtid_executed from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
mysqld: Table 'mysql.gtid_executed' doesn't exist
2022-02-10T07:00:48.042391Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2022-02-10T07:00:48.043734Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2022-02-10T07:00:48.043748Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2022-02-10T07:00:48.043753Z 0 [Warning] A deprecated TLS version TLSv1 is enabled. Please use TLSv1.2 or higher.
2022-02-10T07:00:48.043756Z 0 [Warning] A deprecated TLS version TLSv1.1 is enabled. Please use TLSv1.2 or higher.
2022-02-10T07:00:48.044304Z 0 [Warning] CA certificate ca.pem is self signed.
2022-02-10T07:00:48.044336Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2022-02-10T07:00:48.044406Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2022-02-10T07:00:48.044423Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2022-02-10T07:00:48.046215Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-02-10T07:00:48.046442Z 0 [Warning] InnoDB: Cannot open table mysql/server_cost from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T07:00:48.046460Z 0 [Warning] Failed to open optimizer cost constant tables
2022-02-10T07:00:48.046921Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-02-10T07:00:48.050380Z 0 [Warning] InnoDB: Cannot open table mysql/time_zone_leap_second from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T07:00:48.050401Z 0 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
2022-02-10T07:00:48.053135Z 0 [Warning] InnoDB: Cannot open table mysql/servers from the internal data dictionary of InnoDB though the .frm file for the table exists. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html for how to resolve the issue.
2022-02-10T07:00:48.053164Z 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2022-02-10T07:00:48.057666Z 0 [Note] InnoDB: Buffer pool(s) load completed at 220210 7:00:48
2022-02-10T07:00:48.060428Z 0 [Note] Event Scheduler: Loaded 0 events
2022-02-10T07:00:48.060602Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.37-0ubuntu0.18.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
EDIT
Also, I use some custom parameters (/etc/mysql/mysql.conf.d/mysqld.cnf):
innodb_buffer_pool_size = 1G
innodb_undo_tablespaces = 3
innodb_undo_directory = /var/lib/mysql/undo
innodb_undo_log_truncate = ON
innodb_max_undo_log_size = 16M
innodb_undo_logs = 1
Edit 2
Last night I restarted the MySQL server in one of the three machines to see if this morning it would not crash like the last few days (as they seem to stop about 24 hours after being started).
Unfortunately, MySQL stopped again at about the same time.
I checked the available RAM in the machine a few minutes before the shutdown, I had about 1 GB available. It is usually the RAM that is available. PM2 consumes almost nothing.
I also reinstalled MySQL completely in another of the three machines (with the same configuration). And on this one, there is no crash so far.
EDIT 3
Thanks to Gerrit last comment, I found something interesting in /var/log/dpkg.log:
2022-02-11 06:08:17 startup packages remove
2022-02-11 06:08:17 status installed linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1
2022-02-11 06:08:17 remove linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1 <none>
2022-02-11 06:08:17 status half-configured linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1
2022-02-11 06:08:17 status half-installed linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1
2022-02-11 06:08:17 status config-files linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1
2022-02-11 06:08:17 status config-files linux-modules-5.4.0-1063-aws:amd64 5.4.0-1063.66~18.04.1
2022-02-11 06:08:17 startup packages configure
2022-02-11 06:08:17 configure mysql-server-5.7:amd64 5.7.37-0ubuntu0.18.04.1 <none>
2022-02-11 06:08:17 status half-configured mysql-server-5.7:amd64 5.7.37-0ubuntu0.18.04.1
This happened at the same time as the MySQL server stopped.
How can I disable this "configure mysql-server-5.7" script or finish it manually to fix the issue?
