Score:1

Bad configuration MariaDB / MySQL

fi flag

I currently have a LAMP on a dedicated server but with serious problems with MariaDB (MySQL).

I have installed a website with 100k daily pageviews and due to the need for the queries it is necessary for the database server to have a cache or the site would be very slow.

The problem is that I have a bad configuration in the parameters of the file:

/etc/my.cnf.d/server.cnf

The server goes up to 100% CPU and 100% memory, does not save cache and crash.

Hardware
Server type: Dedicated
CPU: Xeon E5-2650 v4 (x2)
RAM: 128GB
SSD: 2T

Software 
Os: CentOS 7
Server: MariaDB v10.11.4
Tables: 66
Rows: 346,215,143
Engine: innoDB
Size data base: 74,837 MB
Collation database: utf8mb4_general_ci
Callation tables: utf8mb4_general_ci

This is the file:

[server]

[mysqld]
datadir                         = /var/lib/mysql
socket                          = /var/lib/mysql/mysql.sock
log-error                       = /var/log/mariadb/mariadb.log
pid-file                        = /var/lib/mysql/hippo.pid
port                            = 5000
#skip-networking
symbolic-links                  = 0
bind-address                    = 0.0.0.0
default_authentication_plugin   = mysql_native_password
skip-name-resolve               = 1 
max_connections                 = 15000
max_user_connections            = 15000
query_cache_type                = 1
query_cache_size                = 64G
query_cache_limit               = 2G
innodb_file_per_table           = ON
innodb_buffer_pool_size         = 46G
innodb_flush_method             = O_DIRECT
innodb_read_io_threads          = 16
innodb_write_io_threads         = 16
innodb_io_capacity              = 5000
open_files_limit                = 32768
innodb_open_files               = 32768
innodb_log_buffer_size          = 32M
innodb_log_file_size            = 19G
innodb_data_file_path           = ibdata1:10M:autoextend
innodb_flush_log_at_timeout     = 3
key_buffer_size                 = 16G
sort_buffer_size                = 16G
read_buffer_size                = 16G
read_rnd_buffer_size            = 16G
join_buffer_size                = 128M 
thread_cache_size               = 64
tmp_table_size                  = 32G
max_heap_table_size             = 32G
wait_timeout                    = 300
net_read_timeout                = 300
net_write_timeout               = 600
table_open_cache                = 32768

[mysqld_safe]
open_files_limit                = 32768

[galera]

[embedded]

[mariadb]

[mariadb-10.3]

Thank you :)

Romeo Ninov avatar
in flag
100k pageviews `*` 10 database requests `/` 15k connection ~ 70 seconds. Do you really expect such load? Also do you really need 32k open files for one database?
HBruijn avatar
in flag
Very frequently database issues are neither caused by the database server, nor its tuning or the size of the dataset, but can be fully attributed to badly designed SQL queries, a badly designed schema and/or missing indexes. Sufficient hardware and server tuning often mitigate such underlying issues for a time but that won't scale and can't keep up with sharply increasing numbers of site visits. ***Tune your database schema and SQL queries***, that's where probably most optimisation and benefit is to be had. https://dba.stackexchange.com may be more suitable for database related questions
Wilson Hauck avatar
jp flag
Additional DB information request, please. Post TEXT data on justpaste.it and share the links. From your SSH login root, Text results of: A) SELECT COUNT(*), sum(data_length), sum(index_length), sum(data_free) FROM information_schema.tables; B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) STATUS; not SHOW STATUS, just STATUS; G) SHOW ENGINE INNODB STATUS; for server workload tuning analysis to provide suggestions.
Wilson Hauck avatar
jp flag
Post TEXT data on justpaste.it and share the links. Additional very helpful OS information includes - please, htop 1st page, if available, TERMINATE, top -b -n 1 for most active apps, top -b -n 1 -H for details on your mysql threads memory and cpu usage, ulimit -a for list of limits, iostat -xm 5 3 for IOPS by device & core/cpu count, df -h for Used - Free space by device, df -i for inode info by device, free -h for Used - Free Mem: and Swap:, cat /proc/meminfo includes VMallocUused, for server workload tuning analysis to provide suggestions.
Wilson Hauck avatar
jp flag
Have you made the changes suggested by Rick James to reduce RAM requests? You only have 128GB RAM and if you add up all your RAM requests for G volumes, your requests are way beyond availability. Please post Additional information requested on Jun 20, 2023 after 24 hours of uptime after applying Rick's suggestions and stop/start for workload tuning analysis.
Score:1
ua flag

A few pages per second should not be a problem. (1K/second could be a problem.)

Many settings are threatening to overflow your large memory and should be shrunken:

max_connections                 = 15000  --> 200
max_user_connections            = 15000  --> 200
query_cache_size                = 64G  --> 100M
query_cache_limit               = 2G   --> 50M
innodb_buffer_pool_size         = 46G  --> 70G (yes, increase)
key_buffer_size                 = 16G  --> 50M (if not using MyISAM)
sort_buffer_size                = 16G  --> 1G
read_buffer_size                = 16G  --> 1G
read_rnd_buffer_size            = 16G  --> 1G
tmp_table_size                  = 32G  --> 1G
max_heap_table_size             = 32G  --> 1G

The "Query cache" is not very efficient when it is big.

Each connection consumes lots of RAM, so 15000*"lots" is possibly causing your memory problem. Since you deliver only a few pages per second, you don't need more than a few connections.

The buffer_pool is the may caching mechanism; most of RAM should be given to it. It you hit 100% RAM, decrease it further.

Turn on the SlowLog to see which queries are the slowest; then let's look at the indexes involved.

If you are running a web server (Apache?) on the same server, maybe it is over-configured, too.

Score:0
ws flag

This really is a comment, but its a bit too long to fit in the wee box.

I have installed a website with 100k daily pageviews

This is a meaningless metric.

does not save cache and crash

What do you mean it crashes? Core dumps? OOM Killer? Something else?

You certainly have capacity issues - how you fix those takes a lot more time, information and effort than a quick post here.

If your memory usage is getting to 100% then your configuration is wrong - but not necessarily your database config. While the max connections for mysql is much too high, you need to address this on your webserver.

Whether the other settings in your database are appropriate or wrong is IMPOSSIBLE to tell without a detailled understanding of the workload.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.