Score:0

Server

RHEL server unexpectedly became inaccessible via any means (ssh, https)

a coder

12/10/23, 5:03 AM

The machine runs a LAMP stack (httpd, mariadb and php), and both web server and ssh are running but not accessible from remote systems. The issue began randomly an hour ago and the system immediately became unresponsive (both over my existing ssh connection and via httpd).

System is currently up and accessible via idrac, and I am able to log in and issue commands over idrac in terminal.

firewalld is active and shows ports for http, https and ssh are enabled (among others). Stopping firewalld service doesn't resolve the issue.

nmap shows varying results (commands issued from the server).

nmap localhost -p 22
Host is up
22/tcp open ssh

if I specify the server's IP or FQDN with nmap:

nmap xxx.xxx.xxx.xxx -p 22
Note: Host seems down

The same nmap check from a remote system shows host is up:

acoder@lappy ~ $ nmap my.domain.name -p 22
Starting Nmap 7.80 ( https://nmap.org ) at 2022-12-10 08:52 EST
Nmap scan report for my.domain.name (xxx.xxx.xxx.xxx)
Host is up (0.014s latency).

PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds

sshd is up and running

If I had local keys set up, I could probably log on to ssh via localhost with

ssh localhost

It gives permission denied (due to my not having keys set up)

If I issue the same but specify the server's actual IP:

ssh xxx.xxx.xxx.xxx

ssh returns Connection timed out

When issuing ssh my.domain.name from a remote system, I get Connection refused:

acoder@lappy ~ $ ssh -vvv my.domain.name
OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "my.domain.name" port 22
debug2: ssh_connect_direct
debug1: Connecting to my.domain.name [xxx.xxx.xxx.xxx] port 22.
debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
ssh: connect to host my.domain.name port 22: Connection refused

From the server, telnet refuses connection on port 22:

[[email protected]]# telnet localhost 22
Trying 127.0.0.1...
Connect3ed to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH7.4 
Connection refused by foreign host.
[[email protected]]#

I'm able to ssh FROM this server into other servers. Just not into it.

I show active link light on the network device:

ethtool em1

Conversely, if nmap shows port 443 as available when checking from a remote system:

acoder@lappy ~ $ nmap my.domain.name -p 443
Starting Nmap 7.80 ( https://nmap.org ) at 2022-12-09 23:59 EST
Nmap scan report for my.domain.name (xxx.xxx.xxx.xxx)
Host is up (0.013s latency).

PORT    STATE    SERVICE
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 0.41 seconds

Accessing the same domain in a browser yields "Unable to connect"

tcpdump -D:

Here is output of tcpdump -i em1 port 22:

I'm able to ping our DNS server from this server. The server is able to mount shared drives on other servers.

What else am I missing here?

0 + 9

ssh

redhat

apache-2.4

Salar

12/10/23, 6:49 AM

can you check your ip address by issue " ip addr " command. by the information you gave, i think your ip address has been changed. if your server has a private ip address and you publish your services via NAT or PAT please check things with your service provider ...

AlexD

12/10/23, 7:16 AM

'Connection refused' on localhost after showing SSH banner looks weird. Do you have fail2ban? Do you have any connection restrictions in sshd config (like MaxStartups etc)? What is in ssh logs (/var/log/secure or /var/log/auth.log)?

a coder

12/10/23, 1:28 PM

@Salar ip is unchanged. I've tried using the direct IP with all of my commands above, same results.

a coder

12/10/23, 1:43 PM

@AlexD, fail2ban is not installed. No special sshd restrictions (MaxStartups specifically is commented out). I'm not seeing anything in tail secure or 'journalctl -xe' immediately after a 'connection refused' attempt. sshd_config LogLevel is presently set to VERBOSE.

larsks

12/10/23, 1:54 PM

You said that stopping `firewalld` doesn't resolve the problem; does that actually clear the firewall? After stopping `firewalld`, are there still rules defined if you run `iptables-save`? If you run `tcpdump` on the server while trying to connect from a remote host, do you see the traffic show up on the server?

a coder

12/10/23, 2:06 PM

@larsks, stopping `firewalld` service clears the firewall. When `firewalld` is running, I see rules printed with `iptables-save`. After stopping `firewalld`, nothing is returned from `iptables-save`. Working on tcpdump now one sec.

a coder

12/10/23, 2:26 PM

@larsks `tcpdump -i em1 tcp port 22` shows no output while simultaneously attempting to connect via ssh/port 22 from a remote system.

larsks

12/10/23, 2:41 PM

That really suggests the problem is somewhere other than the local system.

a coder

12/10/23, 2:57 PM

Confirmed, there was an unexpected DNS change. thanks for your help.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: RHEL server unexpectedly became inaccessible via any means (ssh, https)

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.