Join Log in
Score:0
gbayi_omo avatar Server

empty openstack network agent list

cn flag
gbayi_omo

I am configuring openstack HA and I have been able to configure up to Nova but neutron configuration using the openstack yoga documentation (https://docs.openstack.org/install-guide/). The only issue I am having is that when I run "openstack network agent list" it comes up empty.

I'm not sure what I am doing wrong.

Thank you for your assistance in advance

Below is the neutron.conf file configuration:

[DEFAULT]
core_plugin = ml2
#state_path 
bind_host = 192.168.120.10
auth_strategy = keystone
service_plugins = router
allow_overlapping_ips = true
notify_nova_on_port_status_changes = true
notify_nova_on_port_data_changes = true
transport_url = rabbit://openstack:rabbit_pass@controller:5671
rpc_response_timeout = 500
rpc_response_timeout = 200
l3_ha = true
service_token_roles_required = true

[database]
connection = mysql+pymysql://neutron:neutron_pass@192.168.120.16/neutron

[keystone_authtoken]
auth_type = password
project_domain_name = Default
project_name = service
user_domain_name = Default
username = neutron
password = neutron_pass
auth_url = https://controller:5000/v3
www_authenticate_uri = https://controller:5000/v3
memcached_servers = 192.168.120.11:11211,192.168.120.12:11211,192.168.120.13:11211
certfile = /etc/ssl/certs/neutron/openstack.pem
keyfile = /etc/ssl/private/neutron/openstackkey.pem
cafile = /etc/ssl/certs/ca.cer

[nova]
auth_type = password
project_domain_name = Default
project_name = service
user_domain_name = Default
username = nova
password = nova_pass
auth_url = https://controller:5000/v3
www_authenticate_uri = https://controller:5000/v3
memcached_servers = 192.168.120.11:11211,192.168.120.12:11211,192.168.120.13:11211
certfile = /etc/ssl/certs/neutron/openstack.pem
keyfile = /etc/ssl/private/neutron/openstackkey.pem
cafile = /etc/ssl/certs/ca.cer

[oslo_concurrency]
lock_path = /var/lib/neutron/tmp


**rabbitmq log**

`
dev-team@cont-00:~$ sudo tail -f /var/log/rabbitmq/rabbit@cont-00.log
2022-12-15 12:04:08.319846-06:00 [info] <0.9671.0> Mirrored queue 'conductor_fanout_135dfab533b146d7a5955ba17cd79639' in vhost '/': Adding mirror on node 'rabbit@cont-01': <18272.7289.0>
2022-12-15 12:05:50.873928-06:00 [info] <0.9927.0> accepting AMQP connection <0.9927.0> (192.168.10.11:54102 -> 192.168.10.11:5671)
2022-12-15 12:05:50.880573-06:00 [info] <0.9927.0> Connection <0.9927.0> (192.168.10.11:54102 -> 192.168.10.11:5671) has a client-provided name: neutron-dhcp-agent:15818:93eba9f1-a48a-4cc0-b217-78b6a47c4e7d
2022-12-15 12:05:50.881628-06:00 [info] <0.9927.0> connection <0.9927.0> (192.168.10.11:54102 -> 192.168.10.11:5671 - neutron-dhcp-agent:15818:93eba9f1-a48a-4cc0-b217-78b6a47c4e7d): user 'openstack' authenticated and granted access to vhost '/'
2022-12-15 12:05:50.896757-06:00 [info] <0.9940.0> Mirrored queue 'dhcp_agent' in vhost '/': Adding mirror on node 'rabbit@cont-02': <18271.7555.0>
2022-12-15 12:05:50.902008-06:00 [info] <0.9940.0> Mirrored queue 'dhcp_agent' in vhost '/': Adding mirror on node 'rabbit@cont-01': <18272.7490.0>
2022-12-15 12:05:50.934746-06:00 [info] <0.9951.0> Mirrored queue 'dhcp_agent.cont-02.altracom.net' in vhost '/': Adding mirror on node 'rabbit@cont-02': <18271.7567.0>
2022-12-15 12:05:50.939653-06:00 [info] <0.9951.0> Mirrored queue 'dhcp_agent.cont-02.altracom.net' in vhost '/': Adding mirror on node 'rabbit@cont-01': <18272.7501.0>
2022-12-15 12:05:50.973962-06:00 [info] <0.9964.0> Mirrored queue 'dhcp_agent_fanout_ac659e839d164932b5663f471874a104' in vhost '/': Adding mirror on node 'rabbit@cont-02': <18271.7578.0>
2022-12-15 12:05:50.985345-06:00 [info] <0.9964.0> Mirrored queue 'dhcp_agent_fanout_ac659e839d164932b5663f471874a104' in vhost '/': Adding mirror on node 'rabbit@cont-01': <18272.7512.0>
`

**Configure the ml2 component sudo vi /etc/neutron/plugins/ml2/ml2_conf.ini**

[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan/ml2/ml2_conf.ini
mechanism_drivers = linuxbridge,l2population
extension_drivers = port_security

[ml2_type_flat]
flat_networks = tunl

[ml2_type_vxlan]
vni_ranges = 1:1000

[securitygroup]
enable_ipset = true

**Configure the linux bridge sudo vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini**

[linux_bridge]
physical_interface_mappings = tunl:ens192

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver
enable_security_group = true

[vxlan]
enable_vxlan = true
local_ip = 192.168.120.11
l2_population = true


**Configure layer-3 Layer 3 switch agent sudo vi /etc/neutron/l3_agent.ini**

[DEFAULT]
interface_driver = linuxbridge

**Configure DHCP proxy, use DNSMASQ sudo vi /etc/neutron/dhcp_agent.ini**

[DEFAULT]
interface_driver = linuxbridge
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
enable_isolated_metadata = true

**Do neutron basic configuration sudo vi /etc/neutron/metadata_agent.ini**

[DEFAULT]
nova_metadata_host = controller
metadata_proxy_shared_secret = shared_secret


sudo tail -f /var/log/neutron/neutron-linuxbridge-agent.log
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return self._send(target, ctxt, message, wait_for_reply, timeout,
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = self._waiter.wait(msg_id, timeout,
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent     message = self.waiters.get(msg_id, timeout=timeout)
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise oslo_messaging.MessagingTimeout(
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 62c0b27563df4470818635c6515bdfc1
2022-12-16 14:56:01.943 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent
2022-12-16 14:56:31.946 1968 ERROR neutron.plugins.ml2.drivers.agent._common_agent [-] Failed reporting state!: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID fe0871b76c214fa6989c9e8313b6e0f6

**Here is the other Agent configuration:**

Configure the ml2 component sudo vi /etc/neutron/plugins/ml2/ml2_conf.ini on all controllers [ml2] type_drivers = flat,vlan,vxlan

tenant_network_types = vxlan

mechanism_drivers = linuxbridge,l2population

extension_drivers = port_security

[ml2_type_flat] flat_networks = provider

[ml2_type_vxlan] vni_ranges = 1:1000

[securitygroup] enable_ipset = true

Configure the linux bridge sudo vi /etc/neutron/plugins/ml2/linuxbridge_agent.ini on all controllers [linux_bridge] physical_interface_mappings = provider:ens192

[securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

enable_security_group = true

[vxlan] enable_vxlan = true

local_ip = 192.168.120.11

local_ip = 192.168.120.12

local_ip = 192.168.120.13

l2_population = true

Configure layer-3 Layer 3 switch agent sudo vi /etc/neutron/l3_agent.ini on all controllers [DEFAULT] interface_driver = linuxbridge

Configure DHCP proxy, use DNSMASQ sudo vi /etc/neutron/dhcp_agent.ini on all controllers [DEFAULT] interface_driver = linuxbridge

dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq

enable_isolated_metadata = true

Do neutron basic configuration sudo vi /etc/neutron/metadata_agent.ini on all controllers [DEFAULT] nova_metadata_host = controller

metadata_proxy_shared_secret = metadata_secret

memcache_servers = 192.168.10.11:11211,192.168.10.12:11211,192.168.10.13:11211

Configure nova again and add the above password to nova sudo vi /etc/nova/nova.conf on all controllers [neutron] auth_type = password project_domain_name = Default project_name = service user_domain_name = Default username = neutron password = FGG6A92 region_name = RegionOne auth_url = https://controller:5000/v3 www_authenticate_uri = https://controller:5000/v3 memcached_servers = 192.168.120.11:11211,192.168.120.12:11211,192.168.120.13:11211 certfile = /etc/ssl/certs/certitficate.pem keyfile = /etc/ssl/private/certitficatekey.pem cafile = /etc/ssl/certs/ca.cer service_metadata_proxy = true metadata_proxy_shared_secret = metadata_secret

sudo ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini

373
0 + 17
us flag
eblock
Are any of the agents running? If the list is empty it would mean that neutron-server is running, but no dhcp, l3, metadata or openvswitch (and possibly others) agents are running. Did you check those services and logs?
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
Thank you eblock. I found the following errors in the linuxbridge-agent log: ERROR neutron.plugins.ml2.drivers.agent._common_agent raise oslo_messaging.MessagingTimeout ERROR neutron.plugins.ml2.drivers.agent._common_agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 469faa1eecde49c7ad90fcb9b746894f 2022-12-14 12:31:10.692 207163 ERROR neutron.plugins.ml2.drivers.agent._common_agent Not sure what else to check rabbitmq seems to be running fine
0
Reply
us flag
eblock
Please update the question with the logs instead of commenting. Is the transport_url correctly configured for the linuxbridge agent? It would be helpful if you also added your neutron configs to the question. Does rabbitmq log anything about the agents trying to connect?
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
thank you eblock and apologies for adding logs to the comment. I have added the rabbitmq logs and the neutron.conf file in the question
0
Reply
us flag
eblock
All the `sed` commands in the output are irritating and more difficult to read, can you just paste the actual config, please? Anyway, it looks like the dhcp agent is successfully contacting rabbitmq (`neutron-dhcp-agent:15818:93eba9f1-a48a-4cc0-b217-78b6a47c4e7d): user 'openstack' authenticated and granted access to vhost '/'`), but I see no message about linuxbridge-agent or the `/etc/neutron/plugins/ml2/ml2_conf.ini` output in your question. Can you please update?
0
Reply
us flag
eblock
I don't see the `core_plugin = ml2` section in your configs. To me it looks like you might have skipped some of the configs, I'd recommend to check again and don't forget the [self-service networks](https://docs.openstack.org/neutron/yoga/install/controller-install-option2-ubuntu.html) and/or [provider networks](https://docs.openstack.org/neutron/yoga/install/controller-install-option1-ubuntu.html) sections.
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
Thank you @eblock. I have reviewed the documentation and haven't seen any thing amiss. The only change I made was to configure the transport_url to use the local IP address and was able to list all network agents except for the Alive that is displaying XXX instead of :-). The neutron logs keeps showing "ERROR neutron.plugins.ml2.drivers.agent._common_agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 2862a652faa4449bb2770849e7b4427e"
0
Reply
us flag
eblock
To me it looks like you only configured neutron-server but none of the networking options, did you read the docs from my last comment? For example, the `core_plugin` is missing. If you needed to change the transport_url to an IP address it means that your DNS is not set up properly.
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
I have added the configuration of other networking options as requested and reviewed the configuration (also ensure that the core_plugin was added) still getting the XXX in the Alive section of the "openstack network agent list"
0
Reply
us flag
eblock
I would enable debug logs, restart one agent and go through the logs to find further information.
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
Hello eblock how are you doing? Apologies for the long silence. I was able to fix this issue by adding the following to sysctl.conf net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 net.ipv4.tcp_retries2 = 3 everything was working until I tried configuring this on Zed version of openstack. Unfortunately linuxbridge is no longer supported. I am unable to get openvswitch or OVN to work. I am unable to find much on how to configure OVN. Any help will be appreciated. Thank you
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
I followed the steps in the link below to configure OVN for neutron: https://www.server-world.info/en/note?os=Ubuntu_22.04&p=openstack_zed&f=11 I get empty result after running "openstack network agent list" All agent logs shows no errors.
0
Reply
us flag
eblock
It's getting difficult to follow, so on the one hand you managed to fix what exactly with the sysctl settings? Those settings didn't work in Zed? Which version did you have before that? And did the agents actually show up in the agent list before Zed? It doesn't make sense to mix versions and then discuss both in this thread. I recommend to create a new thread. By the way, one thing I noticed: what is the 120.16 address? Is it the virtual IP of your cluster? The bind_host address should not be the virtual IP but the local IP as well. I haven't deployed Zed myself yet nor have I OVN experience.
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
apologies for not making myself clear. I was able to fix the the XXX in the Alive section of the "openstack network agent list" when using linuxbridge sgent on Yoga version. After installing the Zed version is where I discovered that linuxbridge can only be used as an experiment. I also tried to configure OVN on Zed which I encountered an empty list when I run openstack network agent list. I don't have120.16 in my config. I also do not use VIP for bind_host address. Please let me know if you need more clarifications and thank you for your help
0
Reply
us flag
eblock
But your database connection contains a 120.16 address: `connection = mysql+pymysql://neutron:neutron_pass@192.168.120.16/neutron` That's why I'm wondering if the agents fail to register to the agents database table.
0
Reply
gbayi_omo avatar
cn flag
gbayi_omo
oh yes thats the mariadb IP address. apologies for that
0
Reply
us flag
eblock
Did you adapt your neutron configuration according to https://docs.openstack.org/neutron/yoga/install/ovn/manual_install.html? So for example, the `service_plugins` should be `service_plugins = ovn-router` if you intend to use ovn. I would recommend to either close this question, create a new one and add only your current configs for ovn instead of linuxbridge. Or you edit this question and replace all required information to reflect your ovn attempts.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.