Score:0

Need to investigate why our server was DDOSing our host service provider

bz flag

Hello everyone and hopefully somebody can give me a first step where I can begin investigating the reason to know why our Linux server appears to have attacked our service provider where the server is hosted on.

Today I received e-mail that one of our servers has been compromising one of our service provider customers as part of a coordinated DDoS botnet so they had to null-route it in order to mitigate so it went down and more or less all our business was down. They reviewed the captures from this attack and do not believe that our IP address was spoofed based on the limited number of distinct hosts attacking to they.

This is our Linux server hosting many different services connected to other our internal servers in infrastructure.

I need to investigate and be able to observe the attack that likely saturated the network adapter of the source. Since the source device is a member of a botnet that is being used for many attacks and I should see many other mysterious bursts of outbound traffic BUT THE PROBLEM IS:

We don't have any monitoring on this server installed so I cannot monitor the traffic that was going out from the server so the question is:

Is it possible to somehow track outbound traffic that was attacking our service provider's customers in Linux? Any commands that could help me? Maybe there are logs recorded?

I do have an information about last timestamps (at the very left). Source and destination IP addresses, protocols, and ports. Unfortunately I don't know where to start as this server doesn't have any monitoring and I don't have great Linux knowledge as I am pretty desperate now and of course, everything need to happen right before Christmas.

Any information will be kindly appreciated.

EDIT: I used journalctl by a given timestamp and now i'm able to see that there are numerous of attempts trying to connect to ssh but unsuccessfully tho one time session opened for root user with given response:

CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)

After that, session for this user is closed.

Anyone knows what this should mean about?

Score:0
jp flag

In general, network bandwidth usage isn't logged or monitored on Linux servers by default. If you didn't install a monitoring system before than you don't have that data.

sintezators avatar
bz flag
what you think about `journalctl`? I used it by a given timestamp and now i'm able to see that there are numerous of attempts trying to connect to ssh but unsuccessfully tho one time session opened for root user with given response: `CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)` After that, session for this user is closed.
jp flag
The messages are just normal log noise. If you are not a forensic expert, you won't find anything. You need to go and read https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server
Score:0
cn flag

If you were not logging at the time, you will not have details. Certainly not at the level of evidence that your host sent certain flows. In the future, consider enabling monitoring and logging, such as enabling logging of new connections via a firewall.

Backup the misbehaving host in case it you will have forensics done on it. Do not give (copies of) this host network access, and never give it internet again. It is likely compromised.

Destroy and rebuild the host from known good sources. Such as install a fresh copy of the OS. Restore data from backup.

As a follow up, get help with doing a detailed root cause investigation on how you might have been sending malicious traffic. Successful ssh logins, evidence of malware, installed software missing security patches, review the flows described by your hosting provider. We cannot help with the details in this Q&A format, however.

sintezators avatar
bz flag
Thank you. This was useful information for me as I haven't faced something like this before
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.