RHEL8.4 with Percona8.0.29 - Cluster fails error code 7

CliveG

10/8/23, 7:29 PM

I have installed 3 x RHEL 8.4 Servers and have also installed the latest version of Percona XtraDB Cluster. I have left the my.cnf file pretty basic, wsrep also basic with just the required changes to node name, gcomm addressing and cluster name. The Primary Node starts perfectly. It has no addressing in gcomm as it will be learned as nodes are added. Just to confirm, I did try adding addressing and restarting the bootstrap. When adding the second node to the cluster with the systemctl start mysqld I am getting a failure to join. here is the output from the log file:

2022-10-08T18:59:10.228425Z 0 [Note] [MY-000000] [WSREP] Starting replication 2022-10-08T18:59:10.228469Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 0 2022-10-08T18:59:10.228486Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1 2022-10-08T18:59:10.228530Z 0 [Note] [MY-000000] [Galera] protonet asio version 0 2022-10-08T18:59:10.228864Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums. 2022-10-08T18:59:10.228882Z 0 [Note] [MY-000000] [Galera] backend: asio 2022-10-08T18:59:10.228942Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 2022-10-08T18:59:10.229004Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booting after a graceful shutdown 2022-10-08T18:59:10.229018Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown 2022-10-08T18:59:10.229115Z 0 [Note] [MY-000000] [Galera] GMCast version 0 2022-10-08T18:59:10.230004Z 0 [Warning] [MY-000000] [Galera] Failed to resolve ssl:// 192.168.56.151:4567 2022-10-08T18:59:10.230068Z 0 [Warning] [MY-000000] [Galera] Failed to resolve ssl:// 192.168.56.150:4567 2022-10-08T18:59:10.230110Z 0 [Note] [MY-000000] [Galera] (4ad2459a-8f81, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567 2022-10-08T18:59:10.230121Z 0 [Note] [MY-000000] [Galera] (4ad2459a-8f81, 'ssl://0.0.0.0:4567') multicast: , ttl: 1 2022-10-08T18:59:10.230305Z 0 [Note] [MY-000000] [Galera] EVS version 1 2022-10-08T18:59:10.230404Z 0 [Note] [MY-000000] [Galera] gcomm: connecting to group 'travelex-cluster', peer '192.168.56.152:, 192.168.56.151:, 192.168.56.150:' 2022-10-08T18:59:13.233756Z 0 [Note] [MY-000000] [Galera] announce period timed out (pc.announce_timeout) 2022-10-08T18:59:13.233864Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1 2022-10-08T18:59:13.233880Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1 2022-10-08T18:59:13.233907Z 0 [Warning] [MY-000000] [Galera] no nodes coming from prim view, prim not possible 2022-10-08T18:59:13.233921Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view (view_id(NON_PRIM,4ad2459a-8f81,1) memb { 4ad2459a-8f81,0 } joined { } left { } partitioned { } ) 2022-10-08T18:59:13.734091Z 0 [Warning] [MY-000000] [Galera] last inactive check more than PT1.5S (3*evs.inactive_check_period) ago (PT3.50379S), skipping check 2022-10-08T18:59:43.262806Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 2022-10-08T18:59:43.262849Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node view ((empty)) 2022-10-08T18:59:43.263049Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) at gcomm/src/pc.cpp:connect():161 2022-10-08T18:59:43.263068Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out) 2022-10-08T18:59:44.263935Z 0 [Note] [MY-000000] [Galera] gcomm: terminating thread 2022-10-08T18:59:44.263993Z 0 [Note] [MY-000000] [Galera] gcomm: joining thread 2022-10-08T18:59:44.264082Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1811: Failed to open channel 'travelex-cluster' at 'gcomm://192.168.56.152, 192.168.56.151, 192.168.56.150': -110 (Connection timed out) 2022-10-08T18:59:44.264097Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out 2022-10-08T18:59:44.264107Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://192.168.56.152, 192.168.56.151, 192.168.56.150) failed to establish connection with cluster (reason: 7) 2022-10-08T18:59:44.264124Z 0 [ERROR] [MY-010119] [Server] Aborting 2022-10-08T18:59:44.264289Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29-21.1) Percona XtraDB Cluster (GPL), Release rel21, Revision 250bc93, WSREP version 26.4.3. 2022-10-08T18:59:44.264960Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED 2022-10-08T18:59:44.264991Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-10-08T18:59:44.266239Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-10-08T18:59:44.267395Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-10-08T18:59:44.268526Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 2022-10-08T18:59:44.268541Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 2022-10-08T18:59:44.268546Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 2022-10-08T18:59:44.268549Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 2022-10-08T18:59:44.268555Z 0 [Note] [MY-000000] [Galera] avg deps dist 0 2022-10-08T18:59:44.268559Z 0 [Note] [MY-000000] [Galera] avg cert interval 0 2022-10-08T18:59:44.268562Z 0 [Note] [MY-000000] [Galera] cert index size 0 2022-10-08T18:59:44.268630Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed. 2022-10-08T18:59:44.268787Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 2022-10-08T18:59:44.268800Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 2022-10-08T18:59:44.269107Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> DESTROYED (TO: 0) 2022-10-08T18:59:44.276640Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk...

I have tried many different options to try and fix this issue but nothing is working. Any help would be great.

465

1 + 3

mysql-replication

percona-xtradb-cluster

rhel8

Wilson Hauck

10/9/23, 2:17 PM

This line from your error log looks abnormal. 2022-10-08T18:59:10.229004Z 0 [Note] [MY-000000] [Galera] Fail to access the file (/var/lib/mysql//gvwstate.dat) error (No such file or directory). We do not usually see two forward slashes together in a filename path. Near the beginning of your error log posting. Did you have a 'graceful shutdown' of the previous version before starting the upgrade?

CliveG

10/10/23, 12:43 PM

Many apologies. I now have the systems in a fully functioning replication cluster with percona. This is, however, without the SSL Certs being involved. I will be investigating this later today. Many thanks for the response.

Wilson Hauck

10/10/23, 2:12 PM

CliveG, Glad you have made progress.

Score:0

Server

CliveG

10/10/23, 8:42 PM

Well, it is all working now with my own openssl self signed certificates.

Freeradius working and radman with percona cluster.

Many thanks for the help Wilson Hauck

+ 1

Wilson Hauck

11/5/23, 2:51 PM

Glad to assist. Happy Holidays.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: RHEL8.4 with Percona8.0.29 - Cluster fails error code 7

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.