Join Log in
Score:0
PSKP avatar Ubuntu

Juju deployment for rabbitmq stuck in Pending state on LXD

in flag
PSKP

I am new to juju and lxd. Trying to install RabbitMQ on LXD using JUJU. Installed LXD and containers are working as expected.

$ juju --version
2.9.42-ubuntu-amd64

Followed below steps

Bootstrapped controller

juju bootstrap localhost rabbitmq

After this, there was one container in LXD, used lxc list

Created model

juju add-model messaging

Deployed RabbitMQ

$ juju deploy rabbitmq-server --debug -n3 --config min-cluster-size=3 rabbitmq
20:11:56 INFO  juju.cmd supercommand.go:56 running juju [2.9.42 7b871e782195bdac9c90f8a8f01723cc3e08ab92 gc go1.18.10]
20:11:56 DEBUG juju.cmd supercommand.go:57   args: []string{"/snap/juju/22345/bin/juju", "deploy", "rabbitmq-server", "--debug", "-n3", "--config", "min-cluster-size=3", "rabbitmq"}
20:11:56 INFO  juju.juju api.go:86 connecting to API addresses: [10.37.130.2:17070]
20:11:56 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://10.37.130.2:17070/api"
20:11:56 INFO  juju.api apiclient.go:687 connection established to "wss://10.37.130.2:17070/api"
20:11:56 INFO  juju.juju api.go:86 connecting to API addresses: [10.37.130.2:17070]
20:11:56 DEBUG juju.api apiclient.go:1152 successfully dialed "wss://10.37.130.2:17070/model/b3ce1455-e650-461d-8a69-6496b6ce573e/api"
20:11:56 INFO  juju.api apiclient.go:687 connection established to "wss://10.37.130.2:17070/model/b3ce1455-e650-461d-8a69-6496b6ce573e/api"
20:11:56 DEBUG juju.cmd.juju.application.deployer deployer.go:396 cannot interpret as local charm: file does not exist
20:11:56 DEBUG juju.cmd.juju.application.deployer deployer.go:208 cannot interpret as a redeployment of a local charm from the controller
20:12:00 DEBUG juju.cmd.juju.application.store charmadapter.go:142 cannot interpret as charmstore bundle: xenial (series) != "bundle"
20:12:00 INFO  cmd charm.go:452 Preparing to deploy "rabbitmq-server" from the charmhub
20:12:15 INFO  cmd charm.go:550 Located charm "rabbitmq-server" in charm-hub, revision 123
20:12:15 INFO  cmd charm.go:236 Deploying "rabbitmq" from charm-hub charm "rabbitmq-server", revision 123 in channel stable on xenial
20:12:16 DEBUG juju.api monitor.go:35 RPC connection died
20:12:16 DEBUG juju.api monitor.go:35 RPC connection died
20:12:16 INFO  cmd supercommand.go:544 command finished

Status of LXD Containers

Those 3 containers which dont have IPs, when they appeared in lxc list, they were in stopped state. after some time they gone into Running state.

$ lxc list
+---------------+---------+--------------------+------+-----------+-----------+
|     NAME      |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |
+---------------+---------+--------------------+------+-----------+-----------+
| juju-ce573e-0 | RUNNING |                    |      | CONTAINER | 0         |
+---------------+---------+--------------------+------+-----------+-----------+
| juju-ce573e-1 | RUNNING |                    |      | CONTAINER | 0         |
+---------------+---------+--------------------+------+-----------+-----------+
| juju-ce573e-2 | RUNNING |                    |      | CONTAINER | 0         |
+---------------+---------+--------------------+------+-----------+-----------+
| juju-dac435-0 | RUNNING | 10.37.130.2 (eth0) |      | CONTAINER | 0         |
+---------------+---------+--------------------+------+-----------+-----------+

Status of juju

$ juju status
Model      Controller  Cloud/Region         Version  SLA          Timestamp
messaging  rabbit1     localhost/localhost  2.9.42   unsupported  20:33:35+05:30

App       Version  Status   Scale  Charm            Channel  Rev  Exposed  Message
rabbitmq           waiting    0/3  rabbitmq-server  stable   123  no       waiting for machine

Unit        Workload  Agent       Machine  Public address  Ports  Message
rabbitmq/0  waiting   allocating  0                               waiting for machine
rabbitmq/1  waiting   allocating  1                               waiting for machine
rabbitmq/2  waiting   allocating  2                               waiting for machine

Machine  State    Address  Inst id        Series  AZ  Message
0        pending           juju-ce573e-0  xenial      Running
1        pending           juju-ce573e-1  xenial      Running
2        pending           juju-ce573e-2  xenial      Running

Debug Logs

$ juju debug-log --replay
controller-0: 20:09:51 INFO juju.worker.apicaller [b3ce14] "machine-0" successfully connected to "localhost:17070"
controller-0: 20:09:51 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 20:09:51 INFO juju.worker.logger logger worker started
controller-0: 20:09:51 INFO juju.worker.machineundertaker setting up machine undertaker
controller-0: 20:09:51 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for messaging (b3ce1455-e650-461d-8a69-6496b6ce573e)
controller-0: 20:09:51 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for messaging (b3ce1455-e650-461d-8a69-6496b6ce573e)
controller-0: 20:09:51 INFO juju.worker.provisioner entering provisioner task loop; using provisioner pool with 16 workers
controller-0: 20:09:51 INFO juju.worker.provisioner provisioning in zones: [pskp]
controller-0: 20:12:18 INFO juju.worker.provisioner provisioning in zones: [pskp]
controller-0: 20:12:18 INFO juju.worker.provisioner found machine pending provisioning id:0, details:0
controller-0: 20:12:23 INFO juju.worker.provisioner provisioning in zones: [pskp]
controller-0: 20:12:23 INFO juju.worker.provisioner found machine pending provisioning id:1, details:1
controller-0: 20:12:23 INFO juju.worker.provisioner trying machine 0 StartInstance in availability zone pskp
controller-0: 20:12:23 INFO juju.worker.provisioner provisioning in zones: [pskp]
controller-0: 20:12:23 INFO juju.worker.provisioner found machine pending provisioning id:2, details:2
controller-0: 20:12:23 INFO juju.worker.provisioner trying machine 1 StartInstance in availability zone pskp
controller-0: 20:12:23 INFO juju.worker.provisioner trying machine 2 StartInstance in availability zone pskp
controller-0: 20:14:42 INFO juju.worker.provisioner started machine 2 as instance juju-ce573e-2 with hardware "arch=amd64 cores=0 mem=0M", network config [], volumes [], volume attachments map[], subnets to zones [], lxd profiles []
controller-0: 20:14:42 INFO juju.worker.provisioner started machine 0 as instance juju-ce573e-0 with hardware "arch=amd64 cores=0 mem=0M", network config [], volumes [], volume attachments map[], subnets to zones [], lxd profiles []
controller-0: 20:14:42 INFO juju.worker.provisioner started machine 1 as instance juju-ce573e-1 with hardware "arch=amd64 cores=0 mem=0M", network config [], volumes [], volume attachments map[], subnets to zones [], lxd profiles []
controller-0: 20:14:46 INFO juju.worker.instancemutater.environ no changes necessary to machine-1 lxd profiles ([default juju-messaging])
controller-0: 20:14:46 INFO juju.worker.instancemutater.environ no changes necessary to machine-0 lxd profiles ([default juju-messaging])
controller-0: 20:14:46 INFO juju.worker.instancemutater.environ no changes necessary to machine-2 lxd profiles ([default juju-messaging])
controller-0: 20:15:05 INFO juju.worker.instancepoller machine "1" (instance ID "juju-ce573e-1") instance status changed from {"running" "Container started"} to {"running" "Running"}
controller-0: 20:15:05 INFO juju.worker.instancepoller machine "2" (instance ID "juju-ce573e-2") instance status changed from {"running" "Container started"} to {"running" "Running"}
controller-0: 20:15:05 INFO juju.worker.instancepoller machine "0" (instance ID "juju-ce573e-0") instance status changed from {"running" "Container started"} to {"running" "Running"}

Multiple Questions

  1. Is deployment stuck?
  2. How can I check controller logs?
  3. How can I check juju agent logs?
  4. There are not IPs for 3 replica rabbitmq containers, is this expected?

Edits

Another observation is, dns resolution is failing inside containers created for rabbitmq while controller container is working fine.

enter image description here

81
0 + 0
lxc
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.