Why is iptables blocking return traffic?

OS: Ubuntu 18.04.6 Firewall type: IPtables. UFW is Disabled.

I have Kubernetes cluster with 3 nodes that provide the controlplane and etcd services, named cp01, cp02 and cp03.

I see that etcd traffic from each of the Controlplane/etcd nodes works fine. The etcd application seems to be working, although I noticed that I cannot do some things like etcdctl elect a new leader.

I noticed that sometimes the return traffic is getting blocked. Here's an example log message showing the block:

May 23 09:34:55 cp02 kernel: [1245818.175864] DROP-INPUT: IN=eth2 OUT= MAC=00:50:AA:BB:CC:DD:00:50:AA:BB:CC:11:08:00 SRC=192.168.101.188 DST=192.168.101.189 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=2380 DPT=36532 WINDOW=0 RES=0x00 RST URGP=0  

I don't understand why these packets are getting dropped. IPtables has a rule to accept all RELATED,ESTABLISHED traffic. It seems like on occasion, that's not happening and the packets are actually getting dropped.

Here is the relevant section of IPtables. Note the following:

• Lines 1-3 are inserted by Kubernetes.
• Lines 4-7 are standard rules (I believe they are.)
• Lines 22 & 23 will log then drop all packets that didn't match an existing rule
• Line 7 accepts all RELATED,ESTABLISHED traffic-- that is, if the traffic went out through this host it can return to this host since it marked as related or established traffic. However, the log message above suggests this is not happening.

cp03:~ # iptables -t filter -L INPUT --line-numbers -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     2763  407K cali-INPUT  all  --  any    any     anywhere             anywhere             /* cali:Cz_abcdefghijklm */
2     2763  407K KUBE-FIREWALL  all  --  any    any     anywhere             anywhere
3       32  6559 KUBE-EXTERNAL-SERVICES  all  --  any    any     anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
4        1    84 ACCEPT     icmp --  any    any     anywhere             anywhere             /* 000 accept all icmp */
5      773  112K ACCEPT     all  --  lo     any     anywhere             anywhere             /* 001 accept all to lo interface */
6        0     0 REJECT     all  --  !lo    any     anywhere             localhost/8          /* 002 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
7     1958  288K ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED /* 003 accept related established rules */
8        0     0 ACCEPT     tcp  --  any    any     admin.example.org      anywhere             multiport dports ssh /* 101 Allow SSH from management servers from 192.168.100.16 */
9        0     0 ACCEPT     udp  --  any    any     cp01.example.org  anywhere             multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.188 */
10       0     0 ACCEPT     udp  --  any    any     cp02.example.org  anywhere             multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.189 */
11       0     0 ACCEPT     udp  --  any    any     cp03.example.org  anywhere             multiport dports 8472 /* 101 Canal/Flannel VXLAN overlay networking from 192.168.101.190 */
12       0     0 ACCEPT     tcp  --  any    any     cp01.example.org  anywhere             multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.188 */
13       0     0 ACCEPT     tcp  --  any    any     cp02.example.org  anywhere             multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.189 */
14       0     0 ACCEPT     tcp  --  any    any     cp03.example.org  anywhere             multiport dports 6443 /* 101 Kubernetes apiserver from 192.168.101.190 */
15       4   220 ACCEPT     tcp  --  any    any     cp01.example.org  anywhere             multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.188 */
16       4   220 ACCEPT     tcp  --  any    any     cp02.example.org  anywhere             multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.189 */
17       0     0 ACCEPT     tcp  --  any    any     cp03.example.org  anywhere             multiport dports 2379:2380 /* 101 etcd client requests from 192.168.101.190 */
18       0     0 ACCEPT     tcp  --  any    any     cp01.example.org  anywhere             multiport dports 10250 /* 101 kubelet API from 192.168.101.188 */
19       0     0 ACCEPT     tcp  --  any    any     cp02.example.org  anywhere             multiport dports 10250 /* 101 kubelet API from 192.168.101.189 */
20       0     0 ACCEPT     tcp  --  any    any     cp03.example.org  anywhere             multiport dports 10250 /* 101 kubelet API from 192.168.101.190 */
21       1    40 LOG        all  --  any    any     anywhere             anywhere             limit: avg 3/min burst 5 /* 998 Log all drops */ LOG level warning prefix "DROP-INPUT: "
22       1    40 DROP       all  --  any    any     anywhere             anywhere             /* 999 drop all other requests */