Kubernetes RabbitMQ Operator: Cluster is having startup issues (readinessProbes are failing), how to troubleshoot further?

My RabbitMQ Cluster is not switching into ready state (readiness probes are failing), as it is having troubles on startup.

The cluster was deployed using the official RabbitMQ operator. Unfortunately the logs do not contain any valuable information.

The RabbitmqCluster resource is more or less the default hello world example:

cat <<EOF | kubectl apply -f -
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: why-not-ready
spec:
  persistence:
    storageClassName: nfs-client
---
EOF

Log output:

rabbitmq 09:06:44.95
rabbitmq 09:06:44.96 Welcome to the Bitnami rabbitmq container
rabbitmq 09:06:44.96 Subscribe to project updates by watching https://github.com/bitnami/containers
rabbitmq 09:06:44.96 Submit issues and feature requests at https://github.com/bitnami/containers/issues
rabbitmq 09:06:44.96
rabbitmq 09:06:44.96 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 09:06:44.98 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 09:06:44.99 INFO  ==> Initializing RabbitMQ...
rabbitmq 09:06:45.03 INFO  ==> Starting RabbitMQ in background...
rabbitmq 09:09:03.93 ERROR ==> Couldn't start RabbitMQ in background.

Tried to enable additional logging, unfortunately there was no more output visible:

cat <<EOF | kubectl apply -n rabbitmq -f -
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: why-not-ready
spec:
  persistence:
    storageClassName: nfs-client
  rabbitmq:
     additionalConfig: |
      log.default.level = debug
      log.file = false
      log.console = true
      log.console.stdio = stdout   
      log.console.level = debug
---
EOF

The setup-container seems to run fine > all files are correctly copied to the desired locations with a valid content:

> cat /var/lib/rabbitmq/.erlang.cookie
uxdvZFvkpHFMdAgXM5azu0LOCE283kzY

> cat /operator/enabled_plugins
[rabbitmq_peer_discovery_k8s,rabbitmq_prometheus,rabbitmq_management].

> cat /var/lib/rabbitmq/.rabbitmqadmin.conf
[default]
username = default_user_l4yC9XizAcUHQEIe3Ep
password = zjvyy5XYCss-kFxu2dCnH2vDrd3AUsbu

Some additional insights from debugging:

• /var/lib/rabbitmq/mnesia/ is writeable from inside the container and also updated on the persistent volume.
• I then checked how rabbitmq is starting up the application. There is a setup script /opt/bitnami/scripts/rabbitmq/setup.sh, which is called on startup. Inside rabbitmq_initialize is called, which causes the following stdout line INFO ==> Starting RabbitMQ in background... as seen in the logs. Unfortunately it is not providing any additional output once I call the script manually.

Does someone have an idea, what is failing on startup or how to enable additional debug logs? Any help would be appreciated!

Thanks and BR,

Philipp

I think this case was very special > still I would have liked to have some kind of logging for rabbitmq_initialize, as this is currently missing.

In my case it was a non reachable core-dns (due to invalid K8s routing configuration), that seem to be necessary when starting rabbitmq.

I was also able to reproduce the problem by removing write privileges on the persistent volume. However this was just because I was running out of ideas :).