Error starting httpd via systemctl

Hi,

Most of our systems have the apache customized, and started manually via the command line. One eg :

[root@dev1-myeg system]# ps -aux | grep http
root      6278  0.0  0.0 112708   984 pts/2    S+   16:32   0:00 grep --color=auto http
root      6398  0.0  0.3 570636 14376 ?        Ss   Sep17   8:34 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17671  0.0  5.5 2598496 209928 ?      Sl   Dec10   0:21 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17672  0.0  5.5 2601972 209792 ?      Sl   Dec10   0:20 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17673  0.0  5.3 2601972 201648 ?      Sl   Dec10   0:23 /usr/local/apache-2.4.33/bin/httpd -k start
apache   17799  0.0  5.5 2602232 209604 ?      Sl   Dec10   0:24 /usr/local/apache-2.4.33/bin/httpd -k start

To start the httpd service ,we use :

/usr/local/apache-2.4.33/bin/httpd -k start

However, we want to automate the service via systemd. Thus I modified the original script which is this :

[root@dev1-myeg system]# cat httpd24-httpd.service.bak
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/opt/rh/httpd24/root/etc/sysconfig/httpd
ExecStart=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -DFOREGROUND
ExecReload=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -k graceful
ExecStop=/opt/rh/httpd24/root/usr/sbin/httpd-scl-wrapper $OPTIONS -k graceful-stop
# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#




To this :

[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

I then killed the currently running httpd process, to test if the script works. I get this error when I start the script :

HTTP process killed :

[root@dev1-myeg system]# ps -aux | grep http
root      7272  0.0  0.0 112708   984 pts/2    S+   16:44   0:00 grep --color=auto http

Starting the httpd service after it had been killed :

[root@dev1-myeg system]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because a configured resource limit was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.

I then edit the script to this (edited the command to start httpd) :

[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
#ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

Trying to stop the service after the error :

[root@dev1-myeg system]# systemctl stop httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Still get error when trying to start the service :

[root@dev1-myeg system]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job  for httpd.service failed because a configured resource limit was  exceeded. See "systemctl status httpd.service" and "journalctl -xe" for  details.

Trying to reload httpd :

[root@dev1-myeg system]# systemctl daemon-reload httpd.service
Invalid number of arguments.

Checking the output from httpd status :

[root@dev1-myeg system]# systemctl status httpd.service
�- httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: resources)

Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 16 16:44:35 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[root@dev1-myeg system]#

This is the output from journalctl -xe :

-- Logs begin at Tue 2019-12-10 05:01:01 +08, end at Mon 2019-12-16 17:40:02 +08. --
Dec 16 16:42:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 133274 of user root.
-- Subject: Unit session-133274.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-133274.scope has finished starting up.
--
-- The start-up result is done.
Dec 16 16:42:01 dev1-myeg.int.myeg.com.my CROND[7051]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.lo
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/pickup[6538]: 34C8D479D7F: uid=0 from=<root>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 34C8D479D7F: message-id=<20191216084202.34C8D479D7F@dev1-myeg.int.myeg.com.my>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 34C8D479D7F: from=<root@dev1-myeg.int.myeg.com.my>, size=2539, nrcpt=1 (queue active)
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/local[4841]: 34C8D479D7F: to=<root@dev1-myeg.int.myeg.com.my>, orig_to=<root>, relay=local, delay=0.41, delays=0.39/0/0/0.01, dsn=5.2.2, st
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 3B9CC47C211: message-id=<20191216084202.3B9CC47C211@dev1-myeg.int.myeg.com.my>
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 3B9CC47C211: from=<>, size=4538, nrcpt=1 (queue active)
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/bounce[6960]: 34C8D479D7F: sender non-delivery notification: 3B9CC47C211
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 34C8D479D7F: removed
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/local[4841]: 3B9CC47C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (ca
Dec 16 16:42:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 3B9CC47C211: removed
Dec 16 16:42:15 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:15+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:7087:779850084 (system bus name :1.269220 [/usr/bin/pkttyagent --notify-fd 5 --fall
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to load environment files: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed to run 'start' task: Argument list too long
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has failed.
--
-- The result is failed.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
Dec 16 16:42:21 dev1-myeg.int.myeg.com.my polkitd[3254]: Unregistered Authentication Agent for unix-process:7087:779850084 (system bus name :1.269220, object path /org/freedesktop/PolicyKit
Dec 16 16:42:23 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:23+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 16 16:42:55 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-16T16:42:55+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 16 16:43:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 133275 of user root.
-- Subject: Unit session-133275.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-133275.scope has finished starting up.
--
-- The start-up result is done.
Dec 16 16:43:01 dev1-myeg.int.myeg.com.my CROND[7144]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.lo
Dec 16 16:43:02 dev1-myeg.int.myeg.com.my postfix/pickup[6538]: 36908479D7F: uid=0 from=<root>
Dec 16 16:43:02 dev1-myeg.int.myeg.com.my postfix/cleanup[4839]: 36908479D7F: message-id=<20191216084302.36908479D7F@dev1-myeg.int.myeg.com.my>
lines 1-48/1435 3%






Please help me debug this, as I am not sure where to start.

Is this CentOS giving you this problem?

If so, I think this is a known bug.

Ref:
Misleading error message: Job for httpd.service failed because a configured resource limit was exceeded . Issue #2716 . systemd/systemd . GitHub

EDIT: Or rather the error message doesn't mean what is says!!! It's telling lies!!!

How about we start with something simple like:

apachectl configtest

What is the results of this config test?

You have an error here i believe :

....
EnvironmentFile=/usr/local/apache-2.4.33/bin/httpd
.....

What i guess is going on, is that you are passing $OPTIONS as content of binary httpd file specified.
This is why you are getting limits error in log and similar.

So either omit the mentioned variable (apache will use default) or use per specification above.

Hope that helps
Regards
Peasant.

I have changed the httpd.service script like this (changed path to Environment File to default) :

[root@dev1-myeg system]# cat httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
EnvironmentFile=/opt/rh/httpd24/root/etc/sysconfig/httpd
#ExecStart=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -DFOREGROUND
ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start
ExecReload=/usr/local/apache-2.4.33/bin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}

# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[root@dev1-myeg system]#

But I still get this error :

[root@dev1-myeg ~]# systemctl start httpd.service
Warning: httpd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for httpd.service failed because a configured resource limit was exceeded. See "systemctl status httpd.service" and "journalctl -xe" for details.

Running this command :

apachectl configtest

provides "Syntax OK".

Need some guidance here, as I am not sure where/how to start to troubleshoot.

Enviroment file is not a apache configuration file, try removing that line entirely.
Every time you change a systemd unit, you need to reload the daemon

systemctl daemon-reload

Then try to start the service again.

systemctl start httpd

If the start fails again, check the status via systemctl status httpd -l
Also, look at apache error logs for hints, together with journalctl -xe

Regards
Peasant.

This time I can reload httpd. However when I attempt to start it I get this error :

[root@dev1-myeg logs]# systemctl status httpd
�- httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Thu 2019-12-19 15:16:38 +08; 1h 30min ago
  Process: 29026 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 29022 ExecStart=/usr/local/apache-2.4.33/bin/httpd -k start (code=exited, status=0/SUCCESS)
 Main PID: 29022 (code=exited, status=0/SUCCESS)

Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: Starting The Apache HTTP Server...
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my kill[29026]: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service stop-sigterm timed out. Killing.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: Failed to start The Apache HTTP Server.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: Unit httpd.service entered failed state.
Dec 19 15:16:38 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service failed.
[root@dev1-myeg logs]#

This is the error logged in journalctl -xe :

Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/bounce[28209]: 79383479D7F: sender non-delivery notification: 89A5C47C211
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 79383479D7F: removed
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 89A5C47C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (c
Dec 19 15:14:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 89A5C47C211: removed
Dec 19 15:14:09 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:14:38 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:14:49 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:14:49+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:14:50 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:28967:805245139 (system bus name :1.277996 [/usr/bin/pkttyagent --notify-fd 5 --fal
Dec 19 15:14:50 dev1-myeg.int.myeg.com.my systemd[1]: Reloading.
Dec 19 15:14:51 dev1-myeg.int.myeg.com.my polkitd[3254]: Unregistered Authentication Agent for unix-process:28967:805245139 (system bus name :1.277996, object path /org/freedesktop/PolicyKi
Dec 19 15:15:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 137606 of user root.
-- Subject: Unit session-137606.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-137606.scope has finished starting up.
--
-- The start-up result is done.
Dec 19 15:15:01 dev1-myeg.int.myeg.com.my CROND[28999]: (root) CMD (cd /usr/local/apache31-MYEG/htdocs/myeg_new_ui && php artisan get:survey > /usr/local/apache31-MYEG/logs/getsurvey-cron.l
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/pickup[25832]: 801D8479D7F: uid=0 from=<root>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/cleanup[28206]: 801D8479D7F: message-id=<20191219071502.801D8479D7F@dev1-myeg.int.myeg.com.my>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 801D8479D7F: from=<root@dev1-myeg.int.myeg.com.my>, size=2539, nrcpt=1 (queue active)
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 801D8479D7F: to=<root@dev1-myeg.int.myeg.com.my>, orig_to=<root>, relay=local, delay=0.43, delays=0.41/0/0/0.02, dsn=5.2.2, s
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/cleanup[28206]: 889D247C211: message-id=<20191219071502.889D247C211@dev1-myeg.int.myeg.com.my>
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 889D247C211: from=<>, size=4538, nrcpt=1 (queue active)
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/bounce[28209]: 801D8479D7F: sender non-delivery notification: 889D247C211
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 801D8479D7F: removed
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/local[28208]: 889D247C211: to=<root@dev1-myeg.int.myeg.com.my>, relay=local, delay=0.02, delays=0.01/0/0/0.01, dsn=5.2.2, status=bounced (c
Dec 19 15:15:02 dev1-myeg.int.myeg.com.my postfix/qmgr[5740]: 889D247C211: removed
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my polkitd[3254]: Registered Authentication Agent for unix-process:29016:805246855 (system bus name :1.277999 [/usr/bin/pkttyagent --notify-fd 5 --fal
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit httpd.service has begun starting up.
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my kill[29026]: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg.int.myeg.com.my systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 19 15:15:18 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my userhelper[29136]: pam_succeed_if(ovirt-container-list:auth): requirement "user = ovirtagent" was met by user "ovirtagent"
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my userhelper[29136]: running '/usr/share/ovirt-guest-agent/container-list' with root privileges on behalf of 'ovirtagent'
Dec 19 15:15:29 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server
Dec 19 15:15:58 dev1-myeg.int.myeg.com.my graylog-collector-sidecar[3251]: time="2019-12-19T15:15:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http:/
Dec 19 15:16:01 dev1-myeg.int.myeg.com.my systemd[1]: Started Session 137607 of user root.
-- Subject: Unit session-137607.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-137607.scope has finished starting up.
lines 1410-1458/1515 96%



This in httpd error logs :

[Thu Dec 19 15:15:08.522704 2019] [ssl:warn] [pid 29022:tid 140066466289472] AH01906: dev1.myeg.com.my:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Dec 19 15:15:08.532567 2019] [jk:warn] [pid 29022:tid 140066466289472] No JkShmFile defined in httpd.conf. Using default /usr/local/apache-2.4.33/logs/jk-runtime-status
[Thu Dec 19 15:15:08.556561 2019] [jk:error] [pid 29022] [Thu Dec 19 15:15:08 2019][29022:140066466289472] [debug] jk_shm_close::jk_shm.c (745): Closed shared memory /usr/local/apache-2.4.33/logs/jk-runtime-status.29022 childs=1
[Thu Dec 19 15:15:08.861044 2019] [ssl:warn] [pid 29025:tid 140066466289472] AH01906: dev1.myeg.com.my:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Dec 19 15:15:09.104449 2019] [jk:warn] [pid 29025:tid 140066466289472] No JkShmFile defined in httpd.conf. Using default /usr/local/apache-2.4.33/logs/jk-runtime-status
[Thu Dec 19 15:15:09.142371 2019] [mpm_event:notice] [pid 29025:tid 140066466289472] AH00489: Apache/2.4.33 (Unix) OpenSSL/1.0.2o PHP/5.6.36 mod_jk/1.2.43 configured -- resuming normal operations
[Thu Dec 19 15:15:09.142464 2019] [core:notice] [pid 29025:tid 140066466289472] AH00094: Command line: '/usr/local/apache-2.4.33/bin/httpd'
(END)

This in httpd access logs :

46.229.168.161 - - [19/Dec/2019:14:04:03 +0800] "GET /robots.txt HTTP/1.1" 200 24 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256
46.229.168.163 - - [19/Dec/2019:14:04:05 +0800] "GET /career/mymotor HTTP/1.1" 200 5341 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256
45.56.78.64 - - [19/Dec/2019:14:24:15 +0800] "GET /?0628182016134805143312 HTTP/1.1" 400 226
64.233.172.106 - - [19/Dec/2019:15:03:04 +0800] "GET /eservices/ HTTP/1.1" 200 1607
64.233.172.108 - - [19/Dec/2019:15:03:04 +0800] "GET /common/img/favicon.ico HTTP/1.1" 200 32942
(END)

This in /var/log/messages :

Dec 19 15:14:50 dev1-myeg systemd: Reloading.
Dec 19 15:15:01 dev1-myeg systemd: Started Session 137606 of user root.
Dec 19 15:15:08 dev1-myeg systemd: Starting The Apache HTTP Server...
Dec 19 15:15:08 dev1-myeg kill: kill: cannot find process ""
Dec 19 15:15:08 dev1-myeg systemd: httpd.service: control process exited, code=exited status=1
Dec 19 15:15:18 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:15:29 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.17
8.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:15:58 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:15:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:01 dev1-myeg systemd: Started Session 137607 of user root.
Dec 19 15:16:09 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.17
8.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:38 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap
i/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:16:38 dev1-myeg systemd: httpd.service stop-sigterm timed out. Killing.
Dec 19 15:16:38 dev1-myeg systemd: Failed to start The Apache HTTP Server.
Dec 19 15:16:38 dev1-myeg systemd: Unit httpd.service entered failed state.
Dec 19 15:16:38 dev1-myeg systemd: httpd.service failed.
Dec 19 15:16:49 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:16:49+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:01 dev1-myeg systemd: Started Session 137608 of user root.
Dec 19 15:17:18 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:18+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:29 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:29+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:17:58 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:17:58+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:18:01 dev1-myeg systemd: Started Session 137609 of user root.
Dec 19 15:18:09 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:18:09+08:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout"
Dec 19 15:18:38 dev1-myeg graylog-collector-sidecar: time="2019-12-19T15:18:38+08:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/ap:

I notice that somehow the graylog-collector-sidecar errors also show after the httpd failures, and also a lot of messages on "ovirt" appear together with them too. How are all of these related?

I ran the command :

# /usr/bin/graylog-collector-sidecar -debug

and these were the outputs I get :

[root@dev1-myeg collector-sidecar]# /usr/bin/graylog-collector-sidecar -debug
INFO[0000] Using collector-id: 3fb08426-6b4b-4928-969f-913cb690dc3f
INFO[0000] No node-id was configured, falling back to hostname
INFO[0000] Fetching configurations tagged by: [laravel apache]
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/collector_sidecar.log.%Y%m%d%H%M
DEBU[0000] Add collector backend: filebeat
INFO[0000] Starting signal distributor
DEBU[0000] [signal-processor] (seq=1) handling cmd: restart
INFO[0000] [filebeat] Starting (exec driver)
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0000] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0000] [signal-processor] (seq=1) cmd done: restart
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
DEBU[0001] [signal-processor] (seq=2) handling cmd: restart
INFO[0001] [filebeat] Stopping
DEBU[0003] [filebeat] SIGHUP ignored, killing process
DEBU[0003] [filebeat] Failed to kill process os: process already finished
INFO[0003] [filebeat] Starting (exec driver)
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0003] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0003] [signal-processor] (seq=2) cmd done: restart
ERRO[0004] [filebeat] Backend finished unexpectedly, trying to restart 2/3.
DEBU[0004] [signal-processor] (seq=3) handling cmd: restart
INFO[0004] [filebeat] Stopping
DEBU[0006] [filebeat] SIGHUP ignored, killing process
DEBU[0006] [filebeat] Failed to kill process os: process already finished
INFO[0006] [filebeat] Starting (exec driver)
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0006] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0006] [signal-processor] (seq=3) cmd done: restart
ERRO[0007] [filebeat] Backend finished unexpectedly, trying to restart 3/3.
DEBU[0007] [signal-processor] (seq=4) handling cmd: restart
INFO[0007] [filebeat] Stopping
DEBU[0009] [filebeat] SIGHUP ignored, killing process
DEBU[0009] [filebeat] Failed to kill process os: process already finished
INFO[0009] [filebeat] Starting (exec driver)
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0009] Creating rotated log writer for: /var/log/graylog/collector-sidecar/filebeat_stderr.log.%Y%m%d%H%M
DEBU[0009] [signal-processor] (seq=4) cmd done: restart
ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!







ERRO[1800] [RequestConfiguration] Fetching configuration failed: Get http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/3fb08426-6b4b-4928-969f-913cb690dc3f?tags=%5B%22laravel%22%2C%22apache%22%5D: dial tcp 13.228.178.92:9000: i/o timeout
ERRO[1800] [UpdateRegistration] Failed to report collector status to server: Put http://13.228.178.92:9000/api/plugins/org.graylog.plugins.collector/collectors/3fb08426-6b4b-4928-969f-913cb690dc3f: dial tcp 13.228.178.92:9000: i/o timeout