Score:1

On a fresh install of Redhat8, on a new EC2 instance, RabbitMQ fails, failed_to_start_child

in flag

UPDATE:

If I do:

yum upgrade

I see:

Skip Packages With Broken Dependencies When 

How do I fix this?

Older post:

I found the file erl_crash.dump here:

/var/lib/rabbitmq/erl_crash.dump 

If I do this:

head -n 50 /var/lib/rabbitmq/erl_crash.dump

I get:

=erl_crash_dump:0.5
Tue Mar  1 20:00:36 2022
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<185,
System version: Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
Compiled: Fri Sep  6 18:47:55 2019
Taints: crypto
Atoms: 13923
Calling Thread: scheduler:1
=scheduler:1
Scheduler Sleep Info Flags: 
Scheduler Sleep Info Aux Work: THR_PRGR_LATER_OP
Current Port: 
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | EXEC
Current Process: <0.0.0>
Current Process State: Running
Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
Current Process Program counter: 0x0000ffff5cd5d020 (init:printable_list/1 + 24)
Current Process CP: 0x0000ffff5cd5dcc8 (init:boot_loop/2 + 1944)
Current Process Limited Stack Trace:
0x0000ffff5a93b9f0:SReturn addr 0xACD16DC0 (<terminate process normally>)

I'm not sure how to read this. Is failed_to_start_child the error? If yes, what does it mean? How do I fix it?

I've got a new EC2 instance, and a fresh install of Redhat8, with everything just installed with yum or dnf but I keep getting errors on RabbitMQ, which tries to start but fails instantly. If I do:

tail -n 500 /var/log/messages

Then I see this message, repeating over and over again:

Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: {"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,84,120,32,178,183,28,70,220,34,20,13,145,69,180,207,145,18,30,76,255,205,110,198,166,173,68,94,39,2,201,5,116,139,211,47,32,125,29,33,161,98,58,78,127,162,252,146,192,110,153,88,149,58,190,136,74,66,191,238,168,79,190,2,120,197,45,101,48,216,168,33,102,142,44,160,25,38,229,27,132,117,91,23,171,84,244,84,230,129,67,131,57,152,36,238,207,222,137,164,79,90,128,114,75,111,48,205,252,239,253,68,193,32,61,190,149,117,145,224,144,191,59,109,247,145,102,240>>,[<<167,114,26,126,33,160,146,70,65,253,176,24,5,41,155,193>>,<<0,0,0,1>>],32],[]},{credentials_obfuscation_pbe,xor_sum,7,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,160}]},{credentials_obfuscation_pbe,blocks,10,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,152}]},{credentials_obfuscation_pbe,make_key,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,94}]},{credentials_obfuscation_pbe,encrypt,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,76}]},{credentials_obfuscation_svc,check,3,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,163}]},{credentials_obfuscation_svc,init_state,0,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,133}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}"}
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,8
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: #015
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Crash dump is being written to: erl_crash.dump...done
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: Failed to start RabbitMQ broker.

I'm struggling to think what the problem is. I believe I've opened up all the ports necessary.

Crash dump is being written to: erl_crash.dump...done

I have read this:

https://stackoverflow.com/questions/63053451/distribution-failed-failed-to-start-child-rabbitmq-docker-compose-issue

I have opened up port 4369 on the security group, but that did not help.

If I run:

epmd -names

I get:

epmd: up and running on port 4369 with data

So that is working, but RabbitMQ is not.

If I do this:

rabbitmqctl start_app 

I get:

Starting node rabbit@ip-172-31-56-22 ...
Error: unable to perform an operation on node 'rabbit@ip-172-31-56-22'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@ip-172-31-56-22
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-172-31-56-22']

rabbit@ip-172-31-56-22:
  * connected to epmd (port 4369) on ip-172-31-56-22
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on ip-172-31-56-22
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-871-rabbit@ip-12-131-56-22'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Gug6Jfak+f3/k+ww6vhA==

I have the impression that something failed to install. Whenever I use, I see something about RabbitMQ, even when my command has nothing to do with RabbitMQ. For instance, I do this:

sudo yum install screen

And part of the output is:

rabbitmq_rabbitmq-server                                                                    5.2 kB/s | 1.8 kB     00:00    

rabbitmq_rabbitmq-server-source                                                                   3.4 kB/s | 951  B     00:00 

I see those 2 lines whenever I use yum install and it makes me uneasy that RabbitMQ is mentioned with every use of yum install as if there is some weird failed cache issue.

I also see this:

 [/usr/lib/tmpfiles.d/rabbitmq-server.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly.
Score:0
in flag

Apparently some install went bad. I noticed this message when running yum install:

"Skipping packages with broken dependencies"

I was not able to fix this. I had to give up on that EC2 instance and start over with a fresh EC2 instance.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.