Join Log in
Score:1
Mr.P avatar Server

rsyslog won't start - pid already exist

ru flag
Mr.P

I was trying to configure rsyslog to forward logs from a specific file to a syslogserver and I ended up purging the whole configuration (the rsyslog service was stuck - wouldn't start) ..

so I purged everything, and installed everything from scratch again ... and now it fails when I try to start it.

here is the OS version:

~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

here is the rsyslog version

~# rsyslogd -v
rsyslogd  8.2212.0 (aka 2022.12) compiled with:
    PLATFORM:               x86_64-pc-linux-gnu
    PLATFORM (lsb_release -d):      
    FEATURE_REGEXP:             Yes
    GSSAPI Kerberos 5 support:      No
    FEATURE_DEBUG (debug build, slow code): No
    32bit Atomic operations supported:  Yes
    64bit Atomic operations supported:  Yes
    memory allocator:           system default
    Runtime Instrumentation (slow code):    No
    uuid support:               Yes
    systemd support:            No
    Config file:                /etc/rsyslog.conf
    PID file:               /var/run/rsyslogd.pid
    Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

Here is my rsyslog.conf

~# cat /etc/rsyslog.conf 
#  /etc/rsyslog.conf    Configuration file for rsyslog.
#
#           For more information see
#           /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")


###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf


###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*         /var/log/auth.log
*.*;auth,authpriv.none      -/var/log/syslog
#cron.*             /var/log/cron.log
daemon.*            -/var/log/daemon.log
kern.*              -/var/log/kern.log
lpr.*               -/var/log/lpr.log
mail.*              -/var/log/mail.log
user.*              -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info           -/var/log/mail.info
mail.warn           -/var/log/mail.warn
mail.err            /var/log/mail.err

#
# Some "catch-all" log files.
#
*.=debug;\
    auth,authpriv.none;\
    news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
    auth,authpriv.none;\
    cron,daemon.none;\
    mail,news.none      -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg             :omusrmsg:*

and lastly the systemd service (which is a soft link to /lib/systemd/system/rsyslog.service)

~# cat /etc/systemd/system/syslog.service 
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=man:rsyslog.conf(5)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n 
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
#LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

when I run /usr/sbin/rsyslogd -n I am getting

~# /usr/sbin/rsyslogd -n
rsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 6260 already exist.
If you want to run multiple instances of rsyslog, you need to specify
different pid files for them (-i option).
rsyslogd: run failed with error -3000 (see rsyslog.h or try https://www.rsyslog.com/e/3000 to learn what that number means)

service status says:

~# systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2022-12-22 08:05:56 CET; 8s ago
     Docs: man:rsyslogd(8)
           man:rsyslog.conf(5)
           https://www.rsyslog.com/doc/
  Process: 6464 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=1/FAILURE)
 Main PID: 6464 (code=exited, status=1/FAILURE)

Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: Failed to start System Logging Service.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Service RestartSec=100ms expired, scheduling restart.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 5.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: Stopped System Logging Service.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Start request repeated too quickly.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Dec 22 08:05:56 TW-3CXNFA-B systemd[1]: Failed to start System Logging Service.

I am a bit stuck here ... I googled the "pid already exist" message and many others but it led me nowhere :(

I would appreciate a bit of nudge here please :( Any ideas what I need to do?

----------- EDIT ------------

I tried this according to the tips from you guys ...

edited (r)syslog.service and add -iNONE as a param

~# cat /etc/systemd/system/syslog.service 
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=man:rsyslog.conf(5)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure

#Increase the default a bit in order to allow many simultaneous
#files to be monitored, we might need a lot of fds.
#LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

reloaded the daemon -> systemctl daemon-reload (no error messages)

stopped both systemctl stop syslog.socket rsyslog.service

checked the status of both

~# systemctl status syslog.socket rsyslog.service
● syslog.socket - Syslog Socket
   Loaded: loaded (/lib/systemd/system/syslog.socket; static; vendor preset: disabled)
   Active: inactive (dead) since Thu 2022-12-22 14:57:32 CET; 18s ago
     Docs: man:systemd.special(7)
           https://www.freedesktop.org/wiki/Software/systemd/syslog
   Listen: /run/systemd/journal/syslog (Datagram)

Dec 22 10:17:58 TW-3CXNFA-B systemd[1]: Listening on Syslog Socket.
Dec 22 14:57:32 TW-3CXNFA-B systemd[1]: syslog.socket: Succeeded.
Dec 22 14:57:32 TW-3CXNFA-B systemd[1]: Closed Syslog Socket.

● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2022-12-22 14:57:32 CET; 18s ago
     Docs: man:rsyslogd(8)
           man:rsyslog.conf(5)
           https://www.rsyslog.com/doc/
  Process: 22681 ExecStart=/usr/sbin/rsyslogd -n -iNONE (code=exited, status=0/SUCCESS)
 Main PID: 22681 (code=exited, status=0/SUCCESS)

Dec 22 14:56:59 TW-3CXNFA-B systemd[1]: Starting System Logging Service...
Dec 22 14:57:32 TW-3CXNFA-B systemd[1]: rsyslog.service: Succeeded.
Dec 22 14:57:32 TW-3CXNFA-B systemd[1]: Stopped System Logging Service.

both are inactive .. I then checked if there is any active PID (all good)

~# ps axu | grep rsyslog
root     22747  0.0  0.0   6072   888 pts/0    S+   14:59   0:00 grep rsyslog

verified if there is the pid file ls /var/run/ | grep syslog (none)

I then started the service which resulted in the timeout message

~# systemctl start rsyslog
Job for rsyslog.service failed because a timeout was exceeded.
See "systemctl status rsyslog.service" and "journalctl -xe" for details.

checking the status

~# systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2022-12-22 15:03:16 CET; 1min 26s ago
     Docs: man:rsyslogd(8)
           man:rsyslog.conf(5)
           https://www.rsyslog.com/doc/
 Main PID: 22896 (rsyslogd)
    Tasks: 4 (limit: 1136)
   Memory: 824.0K
   CGroup: /system.slice/rsyslog.service
           └─22896 /usr/sbin/rsyslogd -n -iNONE

Dec 22 15:03:16 TW-3CXNFA-B systemd[1]: Starting System Logging Service...

plus journal:

~# journalctl -xe
-- A stop job for unit rsyslog.service has finished.
-- 
-- The job identifier is 25478 and the job result is done.
Dec 22 15:03:16 TW-3CXNFA-B systemd[1]: Starting System Logging Service...
-- Subject: A start job for unit rsyslog.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rsyslog.service has begun execution.
-- 
-- The job identifier is 25478.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: rsyslog.service: Start operation timed out. Terminating.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: rsyslog.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit rsyslog.service has entered the 'failed' state with result 'timeout'.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: Failed to start System Logging Service.
-- Subject: A start job for unit rsyslog.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rsyslog.service has finished with a failure.
-- 
-- The job identifier is 25478 and the job result is failed.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: rsyslog.service: Service RestartSec=100ms expired, scheduling restart.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 2.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Automatic restarting of the unit rsyslog.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: Stopped System Logging Service.
-- Subject: A stop job for unit rsyslog.service has finished
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A stop job for unit rsyslog.service has finished.
-- 
-- The job identifier is 25554 and the job result is done.
Dec 22 15:04:46 TW-3CXNFA-B systemd[1]: Starting System Logging Service...
-- Subject: A start job for unit rsyslog.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rsyslog.service has begun execution.
-- 
-- The job identifier is 25554.
Dec 22 15:05:01 TW-3CXNFA-B CRON[22955]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 22 15:05:01 TW-3CXNFA-B CRON[22956]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Dec 22 15:05:01 TW-3CXNFA-B CRON[22955]: pam_unix(cron:session): session closed for user root

wth am I missing?

1406
2 + 2
Zareh Kasparian avatar
us flag
Zareh Kasparian
have you tried to kill the process through "kill -9 PID". kill all the pids related to rsyslog and start it again.
0
Reply
Mr.P avatar
ru flag
Mr.P
didn't help.. see the comment below pls for more details
0
Reply
Score:0
Zareh Kasparian avatar Server
us flag
Zareh Kasparian

I think this is a bug reported via link below:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=815862

This appears counter-intuitive...but the actual steps to perform is you have to stop TWO systemd units, and not just the rsyslog.service or killing the PID of rsyslog:

systemctl stop syslog.socket rsyslog.service
+ 6
Mr.P avatar
ru flag
Mr.P
well I figured that too during debugging ... the problem is .. even if I stop them both, they won't start ... specifically -> rsyslog won't start (syslog.socket will) .. after stopping both, they seem to be dead (when checking the status) .. but when I try to start the rsyslog .. the socket is active and fine .. but the rsyslog is "activating (start)" but will never become active
0
Reply
Zareh Kasparian avatar
us flag
Zareh Kasparian
@Mr.P You are correct. Let me tell you what's happening. as I described when you stop the rsyslog service, it starts itself. So already a PID is used, but in parallel you have started another instance of it manually, which is the second PID of the same service (without -i option). that's why its throwing an error. but you may resolve it by using -i switch "/usr/sbin/rsyslogd -n -iNONE" which causes not to write the PID to any file.
0
Reply
Mr.P avatar
ru flag
Mr.P
so I modified the `/lib/systemd/system/rsyslog.service` which is soft-linked to the /etc/systemd/system/syslog.service ... added the -iNONE as you mentioned (and as is in the D11) ... I reloaded the daemon `systemctl daemon-reload` .. stopped both `systemctl stop syslog.socket rsyslog.service` and started the rsyslog `systemctl start rsyslog` but am still getting the "timeout was exceeded" message :( checking service status for both, socket is running fine but the rsyslog.service ain't (activating)
0
Reply
Mr.P avatar
ru flag
Mr.P
in the logs - all seems to be fine (start)..
0
Reply
Zareh Kasparian avatar
us flag
Zareh Kasparian
@Mr.P in your logs it says: /usr/sbin/rsyslogd -n rsyslogd: pidfile '/var/run/rsyslogd.pid' and pid 6260 already exist.
0
Reply
Mr.P avatar
ru flag
Mr.P
exactly :) that's why this is so weird ... see the updated question pls (it won't format nice here)
0
Reply
Score:0
Nikita Kipriyanov avatar Server
za flag
Nikita Kipriyanov

The default PID file location for rsyslog is /var/run/rsyslogd.pid (as described in the man rsyslogd). cat that file to know what's inside.

Check if that PID is really rsyslogd (something like ps axu | grep $(cat /var/run/rsyslogd.pid) and ps axu | grep rsyslogd). If it is running, kill it as suggested and remove a PID file. This is the problem with the service manager thinking the service is not running, while it is (for example, it was failed to stop).

If PID exists but it is not rsyslogd, don't kill it. It is probably unrelated and that action will likely cause harm rather than good. Just remove the rsyslog PID file. This could have happened because rsyslog created the file, put its PID there and then stopped, the system later reused that PID for something else.

Now, when the service is clearly not running and there is no misleading PID file, try to start it again using service manager, observing the startup in /var/log/daemon.log and other log files (messages, syslog) and checking for any errors during the startup.

Interesting is that since Debian 11 it is configured to not to write any PID file (the startup like is /usr/sbin/rsyslogd -n -iNONE).

+ 4
Mr.P avatar
ru flag
Mr.P
I did find the pid and it was rsyslog `/usr/sbin/rsyslogd -n` ... however after killing it and removing the pid file... when I tried `systemctl start rsyslog` it stuck for a while and then said: 'Job for rsyslog.service failed because a timeout was exceeded. See "systemctl status rsyslog.service" and "journalctl -xe" for details.' .. the `/var/log/daemon.log` is empty ... `messages` says: `Dec 22 08:48:30 TW-3CXNFA-B rsyslogd: [origin software="rsyslogd" swVersion="8.2212.0" x-pid="8052" x-info="https://www.rsyslog.com"] start` .. and the syslog says the same
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
So well, log file seems to show it is successfully started, is it really running now, or not?
0
Reply
Mr.P avatar
ru flag
Mr.P
no .. in the status is says: `Active: activating (start) since Thu 2022-12-22 08:59:04 CET; 40s ago` but will never start ...
0
Reply
Mr.P avatar
ru flag
Mr.P
btw I noticed the `-iNONE` in D11 .. bec on the other machine (D11) the rsyslog is running just fine ..
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.