I have a LEMP server at home running Ubuntu 22.02 and an Oracle cloud instance running Ubuntu 20.04. The Oracle cloud instance is acting as a Wireguard server. The LEMP server at home is acting as the Wireguard Client, and is being tunneled through the Oracle server in order to obtain an IP address that is different from my home IP address. I have set up this Wireguard Client/Server configuration per this Linuxbabe.com tutorial. The wireguard client is up and running and can successfully ping the Oracle Server. The LEMP Server (Wireguard client) also successfully resolves to the Oracle Servers Public IP address. I have also installed openresolv on the VPN Client, and bind9 on the VPN server to use the Oracle Instance's DNS from the LEMP server (the Wireguard Client). My Domain Registrar is pointing it's DNS to the same IP as my Oracle Server. Now, I am attempting to install prosody on the LEMP server, and cannot obtain certificates through the certbot nginx plugin. It seems as though something is blocking port 80/443 and that port 80/443 is not open. When I run (on my LEMP server) the command:
sudo certbot -v --nginx --agree-tos --redirect --hsts --staple-ocsp --email [email protected] -d chat.example.com
I get the following error output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Requesting a certificate for chat.example.com
Performing the following challenges:
http-01 challenge for chat.example.com
Waiting for verification...
Challenge failed for domain chat.example.com
http-01 challenge for chat.example.com
Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: chat.example.com
Type: connection
Detail: 150.136.56.232: Fetching http://chat.example.com/.well-known/acme-challenge/GlzBhvxB_hDYefMW48qaHq3I-qc_NArj7VWml54bofM: Connection refused
Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
I am using UFW as my firewall, and my UFW status on the LEMP Server (the VPN Client) is:
Status: active
To Action From
-- ------ ----
[ 1] 22/tcp ALLOW IN my.local.lan.ip/24
[ 2] 43211/tcp ALLOW IN my.local.lan.ip/24
[ 3] 5222,5269/tcp ALLOW IN Anywhere
[ 4] 80,443/tcp ALLOW IN Anywhere
[ 5] 5222,5269/tcp (v6) ALLOW IN Anywhere (v6)
[ 6] 80,443/tcp (v6) ALLOW IN Anywhere (v6)
My UFW Status on the Oracle Cloud Instance is:
Status: active
To Action From
-- ------ ----
[ 1] Anywhere ALLOW IN 10.10.10.0/24
[ 2] 22/tcp ALLOW IN my.home.public.ip
[ 3] 22/tcp ALLOW IN my.work.public.ip
[ 4] 5222,5269/tcp ALLOW IN Anywhere
[ 5] 51820/udp ALLOW IN my.home.public.ip
[ 6] 80,443/tcp ALLOW IN Anywhere
[ 7] 5222,5269/tcp (v6) ALLOW IN Anywhere (v6)
[ 8] 80,443/tcp (v6) ALLOW IN Anywhere (v6)
Port 51820/udp is my wireguard port for both the Oracle Instance, as well as the Lemp Server. I have also forwarded my public IP address on the Oracle Instance to the VPN client, so that the client can send and receive on the same public port that the Oracle Instance uses. Below is my /etc/ufw/before.rules file. The port forwarding modifications I made are under the comments entitled "Linuxbabe...".
#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
# ufw-before-input
# ufw-before-output
# ufw-before-forward
#
# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines
# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT
# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT
## Linuxbabe tutorial
# allow forwarding for trusted network
-A ufw-before-forward -s 10.10.10.0/24 -j ACCEPT
-A ufw-before-forward -d 10.10.10.0/24 -j ACCEPT
# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT
#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local
# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT
# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT
### Linuxbabe - UFW Tutorial
## NAT table rules
*nat
# Nat Pre-Routing
:PREROUTING ACCEPT [0:0]
# !Http! Forward oracle.server.public.ip (Server Public IP) TCP port 80 to 10.10.10.2:80 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 80 -j DNAT --to-destination 10.10.10.2:80
# !HTTPS! Forward oracle.server.public.ip (Server Public IP) TCP port 443 to 10.10.10.2:443 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 443 -j DNAT --to-destination 10.10.10.2:443
# !PROSODY-5222! Forward oracle.server.public.ip (Server Public IP) TCP port 5222 to 10.10.10.2:5222 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 5222 -j DNAT --to-destination 10.10.10.2:5222
# !PROSODY-5269! Forward oracle.server.public.ip (Server Public IP) TCP port 5269 to 10.10.10.2:5269 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 5269 -j DNAT --to-destination 10.10.10.2:5269
# !PROSODY-BOSH-5280! Forward oracle.server.public.ip (Server Public IP) TCP port 5280 to 10.10.10.2:5280 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 5280 -j DNAT --to-destination 10.10.10.2:5280
# !PROSODY-BOSH-5281! Forward oracle.server.public.ip (Server Public IP) TCP port 5281 to 10.10.10.2:5281 (VPN Client IP)
-A PREROUTING -i ens3 -d oracle.server.public.ip -p tcp --dport 5281 -j DNAT --to-destination 10.10.10.2:5281
COMMIT
### Linuxbabe - Wireguard Tutorial
# Nat Postrouting
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.10.10.0/24 -o ens3 -j MASQUERADE
# End each table with the 'COMMIT' line or these rules won't be processed
COMMIT
My /etc/nginx/nginx.conf file on the LEMP Server (the wireguard client) looks like this:
#user nginx;
user www-data;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
My /etc/nginx/conf.d/prosody.conf file on the LEMP Server (VPN Client) looks like this:
server {
listen 80;
listen [::]:80;
server_name chat.example.com;
root /var/www/prosody/;
location ~ /.well-known/acme-challenge {
allow all;
}
}
Lastly, I have enabled IP Forwarding on the Oracle Server by uncommenting the line net.ipv4.ip_forward = 1 in /etc/sysctl.conf.
After all of this, my LEMP server seems to be successfully using the Oracle Instance's VPN Server Tunnel, HOWEVER, it still cannot obtain certificates from cerbot using the prosody.conf nginx config file. As far as I can tell, with all of the research I've done, this setup (most importantly the VPN forwarding rules in before.rules), should successfully allow my LEMP server to obtain certificates using the Oracle Instance's IP address. But it is NOT DOING SO!
So my question is, what must I do to debug this, what is blocking my LEMP servers port 80/443, and what must I do to successfully obtain certbot certificates for prosody using my Oracle Instance's Public IP address?
