Join Log in
Score:0
Febryandana avatar Server

Openstack - Failed to Create Instance using Availability Zone parameter

US flag
Febryandana

Summary

I have fairly small Openstack Cluster with 2 AZ : AZ-GOLD-6248R have 3 compute node and AZ-E5-2630 have 1 compute node (both AZ had different specs)

  1. I can create instance to AZ-E5-2630 using Horizon or openstack CLI
  2. I cannot create instance to AZ-GOLD-6248R using Horizon or openstack CLI with --availability-zone parameter, Error No Valid Host was Found
  3. But I can create instance to AZ-GOLD-6248R using --host parameter on openstack CLI
  4. It's happen on all projects and user accounts
  5. All hypervisors are up and have a lot of available resources

My cluster specification

  • 3 controller nodes, 1 compute node on AZ-GOLD-6248R, 3 compute nodes on AZ-E5-2630
  • Backend storage for VM, volume, and image (cinder, glance) using NFS storage on different cluster
  • Openstack deployment using Kolla Ansible, openstack ver. Yoga
  • OS Ubuntu 22.04 LTS

What I did

  1. Restart Neutron & OVN services on controller (neutron_server neutron_ovn_metadata_agent openvswitch_db ovn_northd ovn_nb_db ovn_sb_db ovn_controller)
  2. Restart Nova services on controller (nova_conductor nova_scheduler)
  3. Restart Neutron & OVN services on compute nodes (openvswitch_db ovn_controller neutron_ovn_metadata_agent)
  4. Restart Nova services on compute nodes (nova_compute nova_api)
  5. Restart Placement API service on controller (placement_api)

Logs

Screenshot when error

Compute Service and Network service status

Log when fail to create instance:

| fault                               | {'code': 500, 'created': '2023-04-16T23:57:35Z', 'message': 'No valid host was found. ', 'details': 'Traceback (most recent call last):\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances\n    host_lists = self._schedule_instances(context, request_specs[0],\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 908, in _schedule_instances\n    host_lists = self.query_client.select_destinations(\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/client/query.py", line 41, in select_destinations\n    return self.scheduler_rpcapi.select_destinations(context, spec_obj,\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n    return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call\n    result = self.transport._send(\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/tran
sport.py", line 123, in _send\n    return self._driver.send(target, ctxt, message,\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n    return self._send(target, 
ctxt, message, wait_for_reply, timeout,\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send\n    raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. \nTraceback (most recent call last):\n\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n    return func(*args, **kwargs)\n\n  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/manager.py", line 209, in select_destinations\n    raise exception.NoValidHost(reason="")\n\nnova.exception.NoValidHost: No valid host was found. \n\n'} |

Hypervisor show compute node on AZ-GOLD-6248R

+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP      | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |
+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+
|  6 | cmpt01              | QEMU            | 172.16.231.4 | up    |         24 |    48 |          44544 |    128472 |
+----+---------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+

Log nova-scheduler

2023-04-17 06:57:35.038 667 INFO nova.scheduler.manager [req-f50022f7-d4bb-4f43-994b-2331643b64f7 cd7057b2314642c1a9174692a07a3e67 cc6ab975695b44418e88082cc693be36 - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.
2023-04-17 07:07:03.173 668 INFO nova.scheduler.manager [req-cabaf533-78ac-41d6-b052-0751b0cbad94 cd7057b2314642c1a9174692a07a3e67 cc6ab975695b44418e88082cc693be36 - default default] Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.

Log nova-conductor

nova.exception.NoValidHost: No valid host was found.
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager Traceback (most recent call last):
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     host_lists = self._schedule_instances(context, request_specs[0],
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/conductor/manager.py", line 908, in _schedule_instances
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     host_lists = self.query_client.select_destinations(
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/client/query.py", line 41, in select_destinations
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     return self.scheduler_rpcapi.select_destinations(context, spec_obj,
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     return cctxt.call(ctxt, 'select_destinations', **msg_args)
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/client.py", line 189, in call
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     result = self.transport._send(
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/transport.py", line 123, in _send
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     return self._driver.send(target, ctxt, message,
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     return self._send(target, ctxt, message, wait_for_reply, timeout,
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     raise result
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager nova.exception_Remote.NoValidHost_Remote: No valid host was found. 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager Traceback (most recent call last):
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     return func(*args, **kwargs)
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/manager.py", line 209, in select_destinations
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager     raise exception.NoValidHost(reason="")
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager nova.exception.NoValidHost: No valid host was found. 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager 
2023-04-17 06:57:35.043 669 ERROR nova.conductor.manager 
2023-04-17 06:57:35.232 669 WARNING nova.scheduler.utils [req-f50022f7-d4bb-4f43-994b-2331643b64f7 cd7057b2314642c1a9174692a07a3e67 cc6ab975695b44418e88082cc693be36 - default default] Failed to compute_task_build_instances: No valid host was found. 
Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
    return func(*args, **kwargs)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/scheduler/manager.py", line 209, in select_destinations
    raise exception.NoValidHost(reason="")

nova.exception.NoValidHost: No valid host was found. 
: nova.exception_Remote.NoValidHost_Remote: No valid host was found. 
2023-04-17 06:57:35.233 669 WARNING nova.scheduler.utils [req-f50022f7-d4bb-4f43-994b-2331643b64f7 cd7057b2314642c1a9174692a07a3e67 cc6ab975695b44418e88082cc693be36 - default default] [instance: 60c10cff-183d-46ed-8dd4-a5f713510661] Setting instance to ERROR state.: nova.exception_Remote.NoValidHost_Remote: No valid host was found.

When trying allocate resource, all compute node are shown. Node number 3 is the one that have error No Valid Host was Found

openstack allocation candidate list --resource VCPU=20

openstack allocation candidate list

Thank you in advance for all your help

169
0 + 2
us flag
eblock
Could you add the output of `openstack availability zone list --long` to the question? The option `--host` doesn't address an AZ but a host, so I would assume that AZ-GOLD-6248R might not be visible, but it's just a first guess.
0
Reply
Febryandana avatar
md
Febryandana
Sorry for the late replies, I had a long joint holiday here. We finally decided to just recreate the AZ and move all VMs into new AZ because we need fast solution. It solved our problem but we don't know what will happens on the long run.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.