Join Log in
Score:1
avatar Server

Why does bgp OPEN message get Connect Socket: Connection reset by peer when node is on a different subnet/gateway

cn flag
tFlolo

My network setup:

Kubernetes network setup

With this setup, only nodes on same subnet can establish bgp connection. Other nodes (that do a full 3 way tcp handshake), responds to hte OPEN message with [FIN, ACK] then a [RST] hence the Connection reset by peer message in my calicoctl node status <- is on controller 3 (10.0.3.100)

    IPv4 BGP status
+--------------+-------------------+-------+----------+--------------------------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |              INFO              |
+--------------+-------------------+-------+----------+--------------------------------+
| 10.0.1.100   | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection     |
|              |                   |       |          | closed                         |
| 10.0.2.100   | node-to-node mesh | start | 07:12:01 | Connect                        |
| 10.0.1.101   | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection     |
|              |                   |       |          | reset by peer                  |
| 10.0.1.102   | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection     |
|              |                   |       |          | reset by peer                  |
| 10.0.2.102   | node-to-node mesh | start | 07:12:01 | Connect Socket: Connection     |
|              |                   |       |          | reset by peer                  |
| 10.0.3.101   | node-to-node mesh | up    | 07:14:13 | Established                    |
| 10.0.3.102   | node-to-node mesh | up    | 07:12:02 | Established                    |
+--------------+-------------------+-------+----------+--------------------------------+

My wireshark dump of the handshake + OPEN message from controller 3 (10.0.3.100) to node4 (10.0.2.102)

Wireshark bgp trace between 10.0.3.100 and 10.0.2.102
Wireshark bgp trace between 10.0.0.4(10.0.3.100) and 10.0.2.102
Maybe the issue is that node 4 sees the data coming from 10.0.0.4 and not 10.0.3.100?

What works

  1. Ping from all nodes to all nodes OK
  2. nc port 179 to all nodes succeeds
  3. Wireshark shows the full TCP handshake from controller 3 to node 4

Setup

  1. Kubernetes 1.21.1 (installed via kubespray)
  2. Calico 3.9 (default in kubespray)
  3. All gateways are pfSense 2.5.x, the "master" gateway has static routes for 10.0.1.0/24 via 10.0.0.2, 10.0.2.0/24 via 10.0.0.3 and 10.0.3.0/24 via 10.0.0.4.
  4. Firewalls are disabled on the datacenter routers both on wan and lan No NAT is enabled on any of the pfSense boxes. (NAT for ipsec vpn is on wan port for master gateway)
  5. As far as I can tell i have full IP connectivity between all nodes in all subnets
104
0 + 0
bgp
moonkotte avatar
in flag
moonkotte
Welcome to the community! There's a [similar issue](https://github.com/kubernetes-sigs/kubespray/issues/6675), but you're saying NAT is not used inside and full connectivity is presented. I suggest raising this issue to [Calico](https://github.com/projectcalico/calico) directly.
0
Reply
cn flag
tFlolo
Yeah, I'm sorry, I assumed there was no NAT when I cleared out the rules in my routers, seems I had to disable the auto NAT (i assumed that it was auto only for IPsec passtrough, but it was NAT'ing my traffic. Disabled it for sure and it started working., So the other issue is the solution to my problem
0
Reply
Score:0
avatar Server
cn flag
tFlolo

I wrongly assumed pfSense Auto NAT was only for IPsec passtrough, when I disabled all outbound NAT rule generation it started working as intended. My fault for not understanding the setting in my pfSense routers.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.