::qWireguard unable to initiate handshake from one direction Site-to-Site VPN

I am trying to setup a Site-to-Site VPN between SiteA (OpenWrt Router) and SiteB (Oracle instance with public IP)

Since SiteA is OpenWRT, I use the GUI

OpenWRT_Server_Conf_Screenshot

OpenWRT_Peer_Conf_Screenshot

Here is wg showconf output of SiteA:

[Interface]
ListenPort = 51821
PrivateKey = REDACTED

[Peer]
PublicKey = BY...Cwo=
AllowedIPs = 10.2.0.0/16, 192.168.100.0/30
Endpoint = SITE_B_PUBLIC_IP:51821

Here is the config at SiteB:

[Interface]
Address = 192.168.100.2/30
ListenPort = 51821
PrivateKey = REDACTED

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o  enp0s3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp0s3 -j MASQUERADE

[Peer]
PublicKey = ZX...z4=
AllowedIPs = 192.168.100.0/30, 172.16.1.0/24, 172.16.255.0/24
Endpoint = SITE_A_PUBLIC_IP:51821

I am running into a very weird problem. From SiteB, if I ping any addresses on Site A, the tunnel is established and there will be traffic between them without any problems.

However, if the tunnel is not established in advance from Site B. I cannot ping anything from A to B. I see there is traffic coming from A to B with port 51821. I assume these are handshake package, but seem like Wireguard on B does not respond to it

root@ubuntu:~# tcpdump -v port 51821
tcpdump: listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:23:59.004307 IP (tos 0x20, ttl 49, id 2251, offset 0, flags [none], proto UDP (17), length 176)
    SITE_A_PUBLIC_IP.51821 > SITE_B_PUBLIC_IP.51821: UDP, length 148
09:24:04.052134 IP (tos 0x20, ttl 49, id 2467, offset 0, flags [none], proto UDP (17), length 176)
    SITE_A_PUBLIC_IP.51821 > SITE_B_PUBLIC_IP.51821: UDP, length 148
09:24:09.102989 IP (tos 0x20, ttl 49, id 2658, offset 0, flags [none], proto UDP (17), length 176)
    SITE_A_PUBLIC_IP.51821 > SITE_B_PUBLIC_IP.51821: UDP, length 148
09:24:14.152403 IP (tos 0x20, ttl 49, id 2769, offset 0, flags [none], proto UDP (17), length 176)
    SITE_A_PUBLIC_IP.51821 > SITE_B_PUBLIC_IP.51821: UDP, length 148
09:24:19.202805 IP (tos 0x20, ttl 49, id 3187, offset 0, flags [none], proto UDP (17), length 176)

I have enabled Wireguard debugging, but looks like there is no relevant logs on that

Jan 09 09:31:14 ubuntu wg-quick[868]: [#] ip link add wgA type wireguard
Jan 09 09:31:14 ubuntu kernel: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
Jan 09 09:31:14 ubuntu kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
Jan 09 09:31:15 ubuntu wg-quick[868]: [#] wg set wgA private-key /etc/wireguard/wgA.key

Replace the PostUp and Postdown with the following

PostUp = iptables -t nat -I POSTROUTING 1 -o $(route | grep '^default' | grep -o '[^ ]*$') -j MASQUERADE; iptables -I INPUT 1 -i %i -j ACCEPT; iptables -I FORWARD 1 -i $(route | grep '^default' | grep -o '[^ ]*$') -o %i -j ACCEPT; iptables -I FORWARD 1 -i %i -o $(route | grep '^default' | grep -o '[^ ]*$') -j ACCEPT; iptables -I INPUT 1 -i $(route | grep '^default' | grep -o '[^ ]*$') -p udp --dport 51821 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o $(route | grep '^default' | grep -o '[^ ]*$') -j MASQUERADE; iptables -D INPUT -i %i -j ACCEPT; iptables -D FORWARD -i $(route | grep '^default' | grep -o '[^ ]*$') -o %i -j ACCEPT; iptables -D FORWARD -i %i -o $(route | grep '^default' | grep -o '[^ ]*$') -j ACCEPT; iptables -D INPUT -i $(route | grep '^default' | grep -o '[^ ]*$') -p udp --dport 51821 -j ACCEPT

Credit goes to here https://www.reddit.com/r/WireGuard/comments/oxmcvx/comment/h7nl24o/?utm_source=share&utm_medium=web2x&context=3