Score:0

Systemctl reports service failed after user manually restarted Apache. Can systemd and reality be synched without restarting processes?

ir flag

I have an Apache instance that starts with a systemd unit. A user manually stopped and restarted the instance. Now systemctl reports the service as "failed". Other than stopping and restarting Apache, is there a way to get systemd to recognize the service is running?

This is the status. (I've tried to hide the company information, so if httpd-apache and INT-1 don't make sense, it's because I've removed a part of the character string.)

$ systemctl status httpd-apache-int-1.service
* httpd-apache.service - The Apache HTTP Server for the INT-1 Instance
   Loaded: loaded (/usr/lib/systemd/system/httpd-apache-int-1.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2022-05-04 15:56:39 CDT; 2 days ago
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 26914 (code=exited, status=0/SUCCESS)

This is my unit file.

$ systemctl cat httpd-apache-int-1.service
# /usr/lib/systemd/system/httpd-apache-int-1.service
[Unit]
Description=The Apache HTTP Server for the INT-1 Instance
After=network.target remote-fs.target nss-lookup.target
Documentation=man:httpd(8)
Documentation=man:apachectl(8)

[Service]
Type=forking
# ExecStart isn't the "systemd way" of doing things.
#   There may be problems with things like running more
#   than one "systemctl ACTION httpd-apache-int-1"
#   at a time.  But using Environment(File) and
#   systemd override just aren't doing what I want.
ExecStart=/bin/sh -c '\
   source /etc/sysconfig/httpd.int-1 ; \
   source /opt/ca/webagent/ca_wa_env.sh ; \
   source /opt/apache/etc/int-1/WebAgent.conf ; \
   /usr/sbin/httpd $OPTIONS ; \
   exit 0'
ExecReload=/bin/kill -USR1 ${MAINPID}
ExecStop=/bin/kill -WINCH ${MAINPID}
# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target

These are the running processes.

$ ps -ef | grep [i]nt-1
apache    1030 27237  0 14:30 ?        00:00:01 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf
apache    3291 27237  0 May05 ?        00:00:12 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf
apache    9974 27237  0 May05 ?        00:00:15 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf
root     27237     1  0 May04 ?        00:00:05 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf
apache   27239     1  0 May04 ?        00:07:00 LLAWP /opt/apache/etc/int-1/WebAgent.conf -APACHE24
apache   27261 27237  0 May04 ?        00:00:41 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf
apache   27262 27237  0 May04 ?        00:00:37 /usr/sbin/httpd -k start -f /opt/apache/etc/int-1/httpd.conf

Other information that might be useful.

Apache
Server version: Apache/2.4.6 (Red Hat Enterprise Linux)

OS
Red Hat Enterprise Linux Server release 7.9 (Maipo)

$ systemctl --version
systemd 219
user10489 avatar
nc flag
short answer: No, but you could also ignore it. But why not restart it?
Marco avatar
in flag
Maybe use pidfile which systemd monitors?
Marco avatar
in flag
Could it be that due to the complex start of the apache server, somehow systemd can not find the correct pid of the main apache process (e.g. uses the pid of "/bin/sh") ?
iAmJeff avatar
ir flag
@user10489 Production system. Restarts need to be approved by stakeholders. Regarding, "Short answer: No" Dang. I was hoping there'd be some systemd trick.
iAmJeff avatar
ir flag
@Marco The pidfile is the one defined in httpd.conf, so it "should" be the same one. I recall seeing that the PID actually matched the man process when doing "ps -ef | grep [a]pache"
Marco avatar
in flag
@iAmJeff systemd has its own pid handling. It monitors the starting processes and tries to find the pid. This is completely independent of what is written in any config file of httpd. Compare the pids from `systemctl status httpd-apache-int-1` with the pid of the httpd.
iAmJeff avatar
ir flag
@Marco You're right! Looking at the data in my question, 26914 from the status output does not match 27237 listed by ps. Is there no easy way of synchronizing that? (I have no desire to edit the memory space of running processes.)
Marco avatar
in flag
You are using systemd in a way which does not work. Looks like you are still stuck in `init.d` thinking. Best, write a `init.d` script for this and let systemd generate the rest.
iAmJeff avatar
ir flag
@Marco "... let systemd generate the rest." Do you have a link or two with instructions on that?
iAmJeff avatar
ir flag
@Marco Regarding one of your earlier posts, can you share the names or locations of pidfile(s) that systemd monitors?
Marco avatar
in flag
I am not a developer of systemd, I don't know the internal details. Systemd does not monitor pid-files except you use the `PIDFile=` option. For each `init.d` script systemd internally generates a pseudo service which you can handle with `systemctl` the same way as native systemd services (see `man systemd-sysv-generator`). If you create complex systemd services you have to dig into the systemd manuals (https://www.freedesktop.org/software/systemd/man/). I can create an answer which exactly solves your problem, but better you learn it for future problems.
Marco avatar
in flag
In best case you have to change one line: put `exec` in front of `/usr/bin/httpd`
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.