I rebooted my Ubuntu server this morning because I was having what appeared to be a low-memory error (happens occasionally, hasn't been enough of a problem to try and fix it). But now, my site (which was previously working fine) is no longer accessible from the browser.
The setup: I'm running a NuxtJS site using pm2 to daemonize it, and nginx as a reverse proxy. I have a post-receive git hook so that I can push to my remote git repo, which then rebuilds the app and restarts the pm2 instance.
I can only access my site from inside the server, inside a terminal window. Lynx, wget, and cURL all work, and even follow the 301 redirect to HTTPS. And they're working when I request the domain itself, not just the localhost:3000 that's getting reverse proxied. As in, curl https://my-domain.org works. If I try to curl/lynx/etc from any other terminal window, it just waits until it times out. Same with the browser – waits until it times out.
Here are the things I've tried/looked at:
- I'm using UFW, so I checked to see if the firewall was the problem. But 80, 443, and 8080 are all set to ALLOW.
- I tried seeing if maybe nginx wasn't listening somehow, so I tried
sudo lsof -i -P -n | grep LISTEN. Here's the output of that:
nginx 2896 root 6u IPv4 668673557 0t0 TCP *:443 (LISTEN)
nginx 2896 root 7u IPv4 668673558 0t0 TCP *:80 (LISTEN)
nginx 2897 www-data 6u IPv4 668673557 0t0 TCP *:443 (LISTEN)
nginx 2897 www-data 7u IPv4 668673558 0t0 TCP *:80 (LISTEN)
nginx 2898 www-data 6u IPv4 668673557 0t0 TCP *:443 (LISTEN)
nginx 2898 www-data 7u IPv4 668673558 0t0 TCP *:80 (LISTEN)
- I tried checking nginx's access.log. All my curl/wget/Lynx requests are showing up as normal, but none of the browser requests are appearing. I also took a look at the error.log, and got this:
2021/07/31 11:51:52 [emerg] 885#885: bind() to 0.0.0.0:443 failed (98: Address already in use)
2021/07/31 11:51:52 [emerg] 885#885: bind() to 0.0.0.0:80 failed (98: Address already in use)
2021/07/31 11:51:52 [emerg] 885#885: bind() to 0.0.0.0:443 failed (98: Address already in use)
2021/07/31 11:51:52 [emerg] 885#885: bind() to 0.0.0.0:80 failed (98: Address already in use)
2021/07/31 11:51:52 [emerg] 885#885: still could not bind()
Thus far, I haven't found any solutions. I'm just baffled, because whatever changed, it changed because of a reboot. Any ideas are much appreciated.
EDIT to add some output:
sudo systemctl status nginx:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-07-31 15:05:53 EDT; 27min ago
Process: 6834 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status
Process: 6840 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 6837 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 6841 (nginx)
CGroup: /system.slice/nginx.service
├─6841 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
├─6842 nginx: worker process
└─6843 nginx: worker process
Jul 31 15:05:53 parrot systemd[1]: Starting A high performance web server and a reverse proxy server...
Jul 31 15:05:53 parrot systemd[1]: Started A high performance web server and a reverse proxy server.
Output of sudo nginx -T is long, so I made it a gist.
