The main problem is with the server1. It has a direct connection to the Internet and it will send all the packets towards Internet through this connection. However, you want it to send all the packets that came from its VPN clients towards Internet through the tunnel. This is impossible to do using simple basic routing.
To resolve this, you have to use Linux's policy routing. I advise you to read the whole linked LARTC chapter in full (not just that page) before continuing, because it is essential for you to understand what's going on!
Now, when you are read that, let's use for our purpose.
Currently, your server1 has roughly the following routes (as shown by ip route command):
198.51.100.0/24 dev eno1 src 198.51.100.1
default via 198.51.100.2 dev eno1
172.20.9.0/30 dev wg1 src 172.20.9.1
10.8.0.0/30 dev tun0 src 10.8.0.1
10.8.0.0/24 via 10.8.0.2 dev tun0
The first two routes are the Internet connection of the server1, the third one is the tunnel and last two are OpenVPN-related. (I assume that the physical interface is called eno1, the tunnel is WireGuard virtual device called wg1, and OpenVPN is in server net30 mode using tun0.)
The rule table looks like this (ip rule) — as by default:
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
You need to add additional routing tables and rules:
ip route add default via 172.20.9.2 table 200
ip rule add from 10.8.0.0/24 lookup 200
You can use a symbolic name of a table in place of a number, if you register it into the /etc/iproute2/rt_tables file, as described in LARTC.
This doesn't change anything for the server itself. However, clients with addresses like 10.8.0.6 will get routed using the table 200 according to the policy, so their traffic to the Internet will get routed through the tunnel wg1 as defined in the routing table 200.
The rest is easy. On the server2, have a route towards VPN clients and the NAT for them:
ip route add 10.8.0.0/24 via 172.20.9.1
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eno1 -j MASQUERADE
(Again, I assumed the physical interface on the server2 is called eno1 too.)
If you want to use modern NFTables instead of legacy IPTables, change the masquerade command to something like the following:
nft add rule ip nat POSTROUTING oifname "eno1" ip saddr 10.8.0.0/24 masquerade
In the OpenVPN server config you set it up to push a default route through VPN to clients (as opposed to the comment by @dominix up there, you actually want to push a default route to all clients if you want them to browse Internet through the VPN):
push "route 0.0.0.0 0.0.0.0"
Don't forget to enable IP forwarding on both servers:
sysctl net.ipv4.ip_forward=1
That's all. You have to figure out how to make this permanent using your distro's network configuration facilities. Since you didn't specify what distro you use on your servers, I'll leave that as a homework.
