Problem Statement:
Kolla Ansible installation fails on "deploy" play with the below errors ('Errors' Section).
About the Deployment:
- Lab Openstack deployment using Kolla Ansible
- 1 Controller, 2 Compute nodes, 1 Block Storage node using LVM / Cinder Volumes
- All nodes running Ubuntu 22.04
- "Bootstrap-servers" and "prechecks" plays succeed. The Deploy fails
Troubleshooting done:
- https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030641.html
- This link seems to match my exact issue.
- However, I'm not trying to do SSL on HA Proxy.
- Deployment fails with the same error irrespective of whether "enable_haproxy" is kept on or off.
- Deployment fails with the same error irrespective of whether the "external fqdn" and "external vip" are commented or not / commented.
- My globals.yml output below.
Errors:
TASK [nova-cell : Waiting for nova-compute services to register themselves] ******************************************************************************************************************************************************************
skipping: [compute2]
skipping: [controller]
fatal: [compute1 -> controller]: FAILED! => {"msg": "The conditional check '(nova_compute_services.stdout | from_json | map(attribute='Host') | list) is superset(expected_compute_service_hosts)' failed. The error was: Expecting value: line 1 column 1 (char 0)"}
TASK [nova-cell : Fail if nova-compute service failed to register] ***************************************************************************************************************************************************************************
fatal: [compute2]: FAILED! => {"msg": "The conditional check 'any_failed_services or (nova_compute_registration_fatal | bool and\n failed_compute_service_hosts | length > 0)' failed. The error was: error while evaluating conditional (any_failed_services or (nova_compute_registration_fatal | bool and\n failed_compute_service_hosts | length > 0)): {{ ansible_facts.nodename in failed_compute_service_hosts or\n (ansible_facts.hostname ~ \"-ironic\") in failed_compute_service_hosts }}: {{ expected_compute_service_hosts | difference(nova_compute_service_hosts) | list }}: {{ hostvars[all_computes_in_batch[0]].nova_compute_services.stdout |\n from_json |\n map(attribute='Host') |\n list }}: Unable to look up a name or access an attribute in template string ({{ hostvars[all_computes_in_batch[0]].nova_compute_services.stdout |\n from_json |\n map(attribute='Host') |\n list }}).\nMake sure your variable name does not contain invalid characters like '-': the JSON object must be str, bytes or bytearray, not AnsibleUndefined\n\nThe error appears to be in '/usr/local/share/kolla-ansible/ansible/roles/nova-cell/tasks/wait_discover_computes.yml': line 45, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n # that failed to register.\n - name: Fail if nova-compute service failed to register\n ^ here\n"}
fatal: [controller]: FAILED! => {"msg": "The conditional check 'any_failed_services or (nova_compute_registration_fatal | bool and\n failed_compute_service_hosts | length > 0)' failed. The error was: error while evaluating conditional (any_failed_services or (nova_compute_registration_fatal | bool and\n failed_compute_service_hosts | length > 0)): {{ ansible_facts.nodename in failed_compute_service_hosts or\n (ansible_facts.hostname ~ \"-ironic\") in failed_compute_service_hosts }}: {{ expected_compute_service_hosts | difference(nova_compute_service_hosts) | list }}: {{ hostvars[all_computes_in_batch[0]].nova_compute_services.stdout |\n from_json |\n map(attribute='Host') |\n list }}: Unable to look up a name or access an attribute in template string ({{ hostvars[all_computes_in_batch[0]].nova_compute_services.stdout |\n from_json |\n map(attribute='Host') |\n list }}).\nMake sure your variable name does not contain invalid characters like '-': the JSON object must be str, bytes or bytearray, not AnsibleUndefined\n\nThe error appears to be in '/usr/local/share/kolla-ansible/ansible/roles/nova-cell/tasks/wait_discover_computes.yml': line 45, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n # that failed to register.\n - name: Fail if nova-compute service failed to register\n ^ here\n"}
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
block1 : ok=31 changed=5 unreachable=0 failed=0 skipped=9 rescued=0 ignored=0
compute1 : ok=37 changed=5 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0
compute2 : ok=37 changed=5 unreachable=0 failed=1 skipped=16 rescued=0 ignored=0
controller : ok=163 changed=15 unreachable=0 failed=1 skipped=93 rescued=0 ignored=0
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
globals.yml output:
zindagi@deployment:~$ grep -v '^\s*$\|^\s*\#' /etc/kolla/globals.yml
---
workaround_ansible_issue_8743: yes
config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
openstack_release: "master"
kolla_internal_vip_address: "10.0.0.10"
kolla_internal_fqdn: "{{ kolla_internal_vip_address }}"
kolla_external_vip_address: "{{ kolla_internal_vip_address }}"
kolla_external_fqdn: "{{ kolla_external_vip_address }}"
network_interface: "eth1"
neutron_external_interface: "eth2"
neutron_plugin_agent: "openvswitch"
keepalived_virtual_router_id: "51"
enable_openstack_core: "yes"
enable_glance: "{{ enable_openstack_core | bool }}"
enable_haproxy: "no"
enable_keystone: "{{ enable_openstack_core | bool }}"
enable_neutron: "{{ enable_openstack_core | bool }}"
enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_iscsi: "{{ enable_cinder_backend_lvm | bool }}"
enable_cinder_backend_lvm: "yes"
enable_heat: "{{ enable_openstack_core | bool }}"
enable_horizon: "{{ enable_openstack_core | bool }}"
enable_openvswitch: "{{ enable_neutron | bool and neutron_plugin_agent != 'linuxbridge' }}"
fernet_token_expiry: 86400
glance_backend_file: "yes"
cinder_volume_group: "cinder-volumes"
nova_compute_virt_type: "qemu"
nova_console: "novnc"
UPDATE 1 - 8th May 4 PM IST:
- I re-built this deployment from scratch.
- The previous deployment was on VirtualBox - switched over to ESXi 8.x (should not make a difference - still pointing it out)
- Increased specs of the VMs being used
- This time i used "rocky" as the "base distro" in "globals.yml".
- Some more changes to my globals.yml (below)
- This time the deployment DID NOT fail on nova-compute services failing to register. However, it broken on a different error message:
ERROR:
RUNNING HANDLER [iscsi : Restart tgtd container] *********************************************************************************************************************************************************************************************
fatal: [kolla-block1]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n File \"/usr/local/lib/python3.10/dist-packages/docker/api/client.py\", line 268, in _raise_for_status\\n response.raise_for_status()\\n File \"/usr/lib/python3/dist-packages/requests/models.py\", line 943, in raise_for_status\\n raise HTTPError(http_error_msg, response=self)\\nrequests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.42/images/create?tag=master-rocky-9&fromImage=quay.io%2Fopenstack.kolla%2Ftgtd\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \"/tmp/ansible_kolla_docker_payload_y2hdgwrz/ansible_kolla_docker_payload.zip/ansible/modules/kolla_docker.py\", line 406, in main\\n File \"/tmp/ansible_kolla_docker_payload_y2hdgwrz/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 370, in recreate_or_restart_container\\n self.start_container()\\n File \"/tmp/ansible_kolla_docker_payload_y2hdgwrz/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 388, in start_container\\n self.pull_image()\\n File \"/tmp/ansible_kolla_docker_payload_y2hdgwrz/ansible_kolla_docker_payload.zip/ansible/module_utils/kolla_docker_worker.py\", line 211, in pull_image\\n json.loads(line.strip().decode(\\'utf-8\\')) for line in self.dc.pull(\\n File \"/usr/local/lib/python3.10/dist-packages/docker/api/image.py\", line 430, in pull\\n self._raise_for_status(response)\\n File \"/usr/local/lib/python3.10/dist-packages/docker/api/client.py\", line 270, in _raise_for_status\\n raise create_api_error_from_http_exception(e)\\n File \"/usr/local/lib/python3.10/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n raise cls(e, response=response, explanation=explanation)\\ndocker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.42/images/create?tag=master-rocky-9&fromImage=quay.io%2Fopenstack.kolla%2Ftgtd: Not Found (\"manifest for quay.io/openstack.kolla/tgtd:master-rocky-9 not found: manifest unknown: manifest unknown\")\\n'"}
Revised globals.yml
zindagi@kolla-deployment:~$ grep -v '^\s*$\|^\s*\#' /etc/kolla/globals.yml
---
workaround_ansible_issue_8743: yes
kolla_base_distro: "rocky"
kolla_internal_vip_address: "192.168.1.250"
network_interface: "eth1"
neutron_external_interface: "eth2"
neutron_plugin_agent: "openvswitch"
enable_openstack_core: "yes"
enable_haproxy: "yes"
enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_lvm: "yes"
enable_openvswitch: "{{ enable_neutron | bool and neutron_plugin_agent != 'linuxbridge' }}"
fernet_token_expiry: 86400
glance_backend_file: "yes"
cinder_volume_group: "cinder-volumes"
nova_compute_virt_type: "qemu"
nova_console: "novnc"
UPDATE 1 - 8th May 5:30 PM:
- I believe this is resolved now.
- Changed "base distro" back to Ubuntu
- Changed enable_cinder_backend_iscsi back to "yes"
- The biggest change i made was changing the release from "master" to "zed".
Output of the "deploy" play of Kolla
PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
kolla-block1 : ok=34 changed=23 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
kolla-compute1 : ok=69 changed=43 unreachable=0 failed=0 skipped=51 rescued=0 ignored=0
kolla-compute2 : ok=64 changed=43 unreachable=0 failed=0 skipped=50 rescued=0 ignored=0
kolla-controller : ok=323 changed=215 unreachable=0 failed=0 skipped=242 rescued=0 ignored=1
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Final globals.yml file:
root@kolla-deployment:/home/zindagi# grep -v '^\s*$\|^\s*\#' /etc/kolla/globals.yml
---
workaround_ansible_issue_8743: yes
kolla_base_distro: "ubuntu"
kolla_internal_vip_address: "192.168.1.250"
network_interface: "eth1"
neutron_external_interface: "eth2"
neutron_plugin_agent: "openvswitch"
enable_openstack_core: "yes"
enable_haproxy: "yes"
enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_iscsi: "yes"
enable_cinder_backend_lvm: "yes"
enable_openvswitch: "{{ enable_neutron | bool and neutron_plugin_agent != 'linuxbridge' }}"
fernet_token_expiry: 86400
glance_backend_file: "yes"
cinder_volume_group: "cinder-volumes"
nova_compute_virt_type: "qemu"
nova_console: "novnc"
Outputs of docker ps -a from compute, controller and block/storage nodes:
root@kolla-compute1:/home/zindagi# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f5130d4135b quay.io/openstack.kolla/neutron-openvswitch-agent:master-ubuntu-jammy "dumb-init --single-…" 3 minutes ago Up 3 minutes (healthy) neutron_openvswitch_agent
bdff1895f3af quay.io/openstack.kolla/openvswitch-vswitchd:master-ubuntu-jammy "dumb-init --single-…" 7 minutes ago Up 7 minutes (healthy) openvswitch_vswitchd
11ab834e7db4 quay.io/openstack.kolla/openvswitch-db-server:master-ubuntu-jammy "dumb-init --single-…" 8 minutes ago Up 8 minutes (healthy) openvswitch_db
8f2734cb33f4 quay.io/openstack.kolla/nova-compute:master-ubuntu-jammy "dumb-init --single-…" 9 minutes ago Up 9 minutes (healthy) nova_compute
2bbf5b3bbc35 quay.io/openstack.kolla/nova-libvirt:master-ubuntu-jammy "dumb-init --single-…" 10 minutes ago Up 10 minutes (healthy) nova_libvirt
5b5281614f7d quay.io/openstack.kolla/nova-ssh:master-ubuntu-jammy "dumb-init --single-…" 12 minutes ago Up 11 minutes (healthy) nova_ssh
32cf358b66de quay.io/openstack.kolla/iscsid:master-ubuntu-jammy "dumb-init --single-…" 29 minutes ago Up 29 minutes iscsid
2a5adf6983a4 quay.io/openstack.kolla/cron:master-ubuntu-jammy "dumb-init --single-…" 34 minutes ago Up 34 minutes cron
a1a2ed0aa775 quay.io/openstack.kolla/kolla-toolbox:master-ubuntu-jammy "dumb-init --single-…" 36 minutes ago Up 35 minutes kolla_toolbox
4ed9854f1826 quay.io/openstack.kolla/fluentd:master-ubuntu-jammy "dumb-init --single-…" 37 minutes ago Up 37 minutes fluentd
root@kolla-controller:/home/zindagi# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
01750827b82b quay.io/openstack.kolla/heat-engine:master-ubuntu-jammy "dumb-init --single-…" About a minute ago Up 57 seconds (healthy) heat_engine
dbba9ef19e19 quay.io/openstack.kolla/heat-api-cfn:master-ubuntu-jammy "dumb-init --single-…" About a minute ago Up About a minute (healthy) heat_api_cfn
c42514c10c14 quay.io/openstack.kolla/heat-api:master-ubuntu-jammy "dumb-init --single-…" About a minute ago Up About a minute (healthy) heat_api
60fc15caabe4 quay.io/openstack.kolla/neutron-metadata-agent:master-ubuntu-jammy "dumb-init --single-…" 3 minutes ago Up 3 minutes (healthy) neutron_metadata_agent
affd17658759 quay.io/openstack.kolla/neutron-l3-agent:master-ubuntu-jammy "dumb-init --single-…" 3 minutes ago Up 3 minutes (healthy) neutron_l3_agent
cbb256538649 quay.io/openstack.kolla/neutron-dhcp-agent:master-ubuntu-jammy "dumb-init --single-…" 4 minutes ago Up 3 minutes (healthy) neutron_dhcp_agent
67d4c8f1b6b6 quay.io/openstack.kolla/neutron-openvswitch-agent:master-ubuntu-jammy "dumb-init --single-…" 4 minutes ago Up 4 minutes (healthy) neutron_openvswitch_agent
f34d7a7db2ee quay.io/openstack.kolla/neutron-server:master-ubuntu-jammy "dumb-init --single-…" 4 minutes ago Up 4 minutes (healthy) neutron_server
3beefd2cd97a quay.io/openstack.kolla/openvswitch-vswitchd:master-ubuntu-jammy "dumb-init --single-…" 8 minutes ago Up 8 minutes (healthy) openvswitch_vswitchd
83fc1eee9ae1 quay.io/openstack.kolla/openvswitch-db-server:master-ubuntu-jammy "dumb-init --single-…" 8 minutes ago Up 8 minutes (healthy) openvswitch_db
65992167e59a quay.io/openstack.kolla/nova-novncproxy:master-ubuntu-jammy "dumb-init --single-…" 14 minutes ago Up 13 minutes (healthy) nova_novncproxy
cb58408a8728 quay.io/openstack.kolla/nova-conductor:master-ubuntu-jammy "dumb-init --single-…" 14 minutes ago Up 14 minutes (healthy) nova_conductor
6caa3a38cb4e quay.io/openstack.kolla/nova-api:master-ubuntu-jammy "dumb-init --single-…" 15 minutes ago Up 15 minutes (healthy) nova_api
7341fbb3ec59 quay.io/openstack.kolla/nova-scheduler:master-ubuntu-jammy "dumb-init --single-…" 15 minutes ago Up 15 minutes (healthy) nova_scheduler
ad3580d98c0b quay.io/openstack.kolla/placement-api:master-ubuntu-jammy "dumb-init --single-…" 18 minutes ago Up 18 minutes (healthy) placement_api
0701d637d49d quay.io/openstack.kolla/cinder-scheduler:master-ubuntu-jammy "dumb-init --single-…" 21 minutes ago Up 21 minutes (healthy) cinder_scheduler
6320ffe512fc quay.io/openstack.kolla/cinder-api:master-ubuntu-jammy "dumb-init --single-…" 21 minutes ago Up 21 minutes (healthy) cinder_api
ad0f2adb96a7 quay.io/openstack.kolla/glance-api:master-ubuntu-jammy "dumb-init --single-…" 23 minutes ago Up 23 minutes (healthy) glance_api
f01034594a50 quay.io/openstack.kolla/keystone:master-ubuntu-jammy "dumb-init --single-…" 26 minutes ago Up 26 minutes (healthy) keystone
2c4ed3eb1553 quay.io/openstack.kolla/keystone-fernet:master-ubuntu-jammy "dumb-init --single-…" 26 minutes ago Up 26 minutes (healthy) keystone_fernet
8b63d3416385 quay.io/openstack.kolla/keystone-ssh:master-ubuntu-jammy "dumb-init --single-…" 26 minutes ago Up 26 minutes (healthy) keystone_ssh
dc2ae69f6196 quay.io/openstack.kolla/rabbitmq:master-ubuntu-jammy "dumb-init --single-…" 29 minutes ago Up 29 minutes (healthy) rabbitmq
8329d6a73929 quay.io/openstack.kolla/memcached:master-ubuntu-jammy "dumb-init --single-…" 30 minutes ago Up 30 minutes (healthy) memcached
35acc9734a53 quay.io/openstack.kolla/mariadb-clustercheck:master-ubuntu-jammy "dumb-init --single-…" 31 minutes ago Up 31 minutes mariadb_clustercheck
8d94088d4c74 quay.io/openstack.kolla/mariadb-server:master-ubuntu-jammy "dumb-init -- kolla_…" 31 minutes ago Up 31 minutes (healthy) mariadb
4aebfdba90a9 quay.io/openstack.kolla/keepalived:master-ubuntu-jammy "dumb-init --single-…" 33 minutes ago Up 33 minutes keepalived
123a4fc71aae quay.io/openstack.kolla/haproxy:master-ubuntu-jammy "dumb-init --single-…" 33 minutes ago Up 33 minutes (healthy) haproxy
7f6c2adc5e41 quay.io/openstack.kolla/cron:master-ubuntu-jammy "dumb-init --single-…" 35 minutes ago Up 35 minutes cron
f1abce91f91a quay.io/openstack.kolla/kolla-toolbox:master-ubuntu-jammy "dumb-init --single-…" 35 minutes ago Up 35 minutes kolla_toolbox
8056c8a6a470 quay.io/openstack.kolla/fluentd:master-ubuntu-jammy "dumb-init --single-…" 38 minutes ago Up 37 minutes fluentd
root@kolla-block1:/home/zindagi# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34eb37634e70 quay.io/openstack.kolla/cinder-volume:master-ubuntu-jammy "dumb-init --single-…" 26 minutes ago Up 26 minutes (healthy) cinder_volume
140d5029b8c7 quay.io/openstack.kolla/tgtd:master-ubuntu-jammy "dumb-init --single-…" 36 minutes ago Up 36 minutes tgtd
3e7eb6195b1f quay.io/openstack.kolla/iscsid:master-ubuntu-jammy "dumb-init --single-…" 36 minutes ago Up 36 minutes iscsid
a7e1464b43df quay.io/openstack.kolla/cron:master-ubuntu-jammy "dumb-init --single-…" 41 minutes ago Up 41 minutes cron
837217ad632d quay.io/openstack.kolla/kolla-toolbox:master-ubuntu-jammy "dumb-init --single-…" 42 minutes ago Up 41 minutes kolla_toolbox
f1cb0457cf12 quay.io/openstack.kolla/fluentd:master-ubuntu-jammy "dumb-init --single-…" 44 minutes ago Up 44 minutes fluentd
root@kolla-block1:/home/zindagi#
