Some of application (it's unknown) makes sporadic nonregular rare short outbound HTTP(S) requests to a known host/port/url (this is a WAF honeypot, host/url/port is known) using HTTPS protocol.
Requests may occur once per 3-5 days. It's literally one short request per 3-5 days.
The goal is to define what application (path to binary, PID etc) makes these requests.
Server has plenty of software installed including nginx, php, mariadb, redis, docker etc.
For simplicity, honeypot IP will have 7.7.7.7 here.
What I've tried sofar:
- tcpdump
$ sudo tcpdump -i any dst host 7.7.7.7 and "tcp[tcpflags] & (tcp-syn) != 0" and "tcp[tcpflags] & (tcp-ack) == 0" &> /tmp/out_7.7.7.7_$(date "+%Y.%m.%d-%H.%M.%S").log &
It seems tcpdump doesn't allow to detect process id or executable which makes outbound http requests.
- auditctl/auditd
sudo auditctl -a exit,always -F arch=b64 -S connect -k connectLog
auditctl/auditd seems can output the path and pid, but it lacks filtering feature. If I start audit rule for a 3-5 days, my disk will be full of auditlog. Or, if audit logs are rotating, I could miss required log data in log file alreayd rotated and wiped out.
If auditctl would have filtering feature on write (not on parse logs) by target IP, it probably would be the best candidate.
Maybe I'm missing something and it has filtering feature?
Another option that I figured out is to make some bash script, which:
- started the auditing of
connect
- started the monitoring process like this:
( tail -f -n0 /var/log/audit/audit.log & ) | grep -q "7.7.7.7"
- once monitor detects this, stop auditing
auditctl -d...
The problem is, this event may occur after 3-5 days, when all disks will be full.
- netstat
$ sudo netstat -tupnc | grep 7.7.7.7
It seems that netstat with -c option (continuos) repeats reading each 1 second. Since requests are very short, it could miss this request.
- ss
ss seems doesn't display the process originated the outbound connection.
- lsof
lsof -i TCP:80,443 -r 1
The outbound connection is very short and quick, it may be not logged by running lsof each 1 second
wireshark
Wireshark has good filtering features (by IP), but it seems that it doesn't display the connection originator process name or pid.
Currently I'm stuck with the solution: combine syslog-ng (wich has filtering capabilities when receiving logs via TCP/network) as a receiver and auditd as a log events sender.
I succeeded starting syslog-ng on port 2222, receiveing data from network and filtering it by some string (tested with curl).
But I cannot manage auditd send logs to network.
What I've done:
7.1) Installed audisp-remote-plugin:
$ sudo apt install audispd-plugins
7.2) Enabled audisp-remote plugin:
in file /etc/audit/plugins.d/au-remote.conf:
set active = yes
7.3) Configured audisp-remote plugin:
in file /etc/audit/auditsp-remote.conf:
remote_server = 127.0.0.1
port = 2222
7.4) Disabled writing to a local log_file
in file /etc/audit/auditd.conf:
write_logs = no
and restarted auditd:
$ sudo systemctl restart auditd
7.5) Added audit rule to catch connections (connect syscalls):
$ sudo auditctl -a exit,always -F arch=b64 -F saddr_fam=2 -S connect -k sckt
7.6) Rotated logs to clean previous data in auditd logs
$ service auditd rotate
7.7) Made test HTTP call:
$ curl -v https://7.7.7.7/api/v1/t
7.8) Checked syslog-ng log, but it doesn't have expected records.
If any knows the appropriatest, easiest and simple the best approach to do this, any help will be appreciated!
