How is IPsec (strongswan) working without opening ports in UFW?

I needed to setup a site-to-site VPN between servers A and B, where server A is being managed by me and server B is being managed by a client. Server A is running Ubuntu 20.04 and I am using strongswan to setup the VPN on my end. I am using UFW to manage server A's firewall.

Public IP address of A: 16.XX.XXX.17 Public IP address of B: 14.XXX.XXX.94

Now after making the necessary configuration changes for setting up the VPN and turning it on, I am able to see the following logs in /var/log/syslog

Mar 21 13:58:47 worker0 charon: 01[ENC] parsed IKE_AUTH response 1 [ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr ]
Mar 21 13:58:47 worker0 charon: 01[IKE] authentication of '14.XXX.XXX.94' with pre-shared key successful
Mar 21 13:58:47 worker0 charon: 01[IKE] IKE_SA s-to-s[1] established between 16.XX.XXX.17[16.XX.XXX.17]...14.XXX.XXX.94[14.XXX.XXX.94]
Mar 21 13:58:47 worker0 charon: 01[IKE] scheduling reauthentication in 2644s
Mar 21 13:58:47 worker0 charon: 01[IKE] maximum IKE_SA lifetime 3184s
Mar 21 13:58:47 worker0 charon: 01[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Mar 21 13:58:47 worker0 charon: 01[CFG] selected proposal: ESP:AES_CBC_256/HMAC_SHA2_256_128/NO_EXT_SEQ
Mar 21 13:58:47 worker0 charon: 01[IKE] CHILD_SA s-to-s{1} established with SPIs c97fea32_i f60175ca_o and TS 10.132.86.142/32 === 10.128.28.96/27
Mar 21 13:58:53 worker0 charon: 14[NET] received packet: from 14.XXX.XXX.94[4500] to 16.XX.XXX.17[4500] (80 bytes)
Mar 21 13:58:53 worker0 charon: 14[ENC] parsed INFORMATIONAL request 0 [ ]
Mar 21 13:58:53 worker0 charon: 14[ENC] generating INFORMATIONAL response 0 [ ]
Mar 21 13:58:53 worker0 charon: 14[NET] sending packet: from 16.XX.XXX.17[4500] to 14.XXX.XXX.94[4500] (80 bytes)
Mar 21 13:58:58 worker0 charon: 07[NET] received packet: from 14.XXX.XXX.94[4500] to 16.XX.XXX.17[4500] (80 bytes)
Mar 21 13:58:58 worker0 charon: 07[ENC] parsed INFORMATIONAL request 1 [ ]
Mar 21 13:58:58 worker0 charon: 07[ENC] generating INFORMATIONAL response 1 [ ]
Mar 21 13:58:58 worker0 charon: 07[NET] sending packet: from 16.XX.XXX.17[4500] to 14.XXX.XXX.94[4500] (80 bytes)
Mar 21 13:59:03 worker0 charon: 05[NET] received packet: from 14.XXX.XXX.94[4500] to 16.XX.XXX.17[4500] (80 bytes)

Output of ipsec status:

Security Associations (1 up, 0 connecting):
   s-to-s[1]: ESTABLISHED 5 minutes ago, 16.XX.XXX.17[16.XX.XXX.17]...14.XXX.XXX.94[14.XXX.XXX.94]
   s-to-s{1}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c97fea32_i f60175ca_o
   s-to-s{1}:   10.132.86.142/32 === 10.128.28.96/27

The ufw status

Status: active

To                         Action      From
--                         ------      ----
666                        LIMIT       Anywhere                   # For ssh
62626                      ALLOW       Anywhere                   # For wireguard
Anywhere on pv0            ALLOW       Anywhere                  
666 (v6)                   LIMIT       Anywhere (v6)              # For ssh
62626 (v6)                 ALLOW       Anywhere (v6)              # For wireguard
Anywhere (v6) on pv0       ALLOW       Anywhere (v6)             

How is the server able to receive packets on 4500 port (according to the ipsec logs in /var/log/syslog) without me opening that port? What am I missing here?

I have not touched IP tables directly at all. I have verified by running a simple http server on 8000 port (using python3 -m http.server) that I am not able to access other unopened ports from outside the server)

ufw is not the sole firewall module present in the system, it merely controls its own chain(s) in the global INPUT iptables/nftables chain. You can verify that the required rules are installed by running iptables -S on server A, you should see that there are some ACCEPT rules for UDP:4500 somewhere reachable from INPUT.

Alternatively, there is more than one routing/filtering table, and while ufw operates on "general" table (IIRC index 254) VPN systems usually allocate one of the tables with a smaller number (200 seems to be a common choice) and set up traffic that's subject to VPN control to be routed/filtered by that table. Verify by running nft list tables, then you can inspect a table by running nft list ruleset table (table). Check with your man pages for exact syntax please.