2017-05-01 21:13:22 ][ Tags: apache devops
If you spend any time mucking around config files in Linux you are likely to run into some syntax errors sooner or later. Recently I was setting up cgit on Debian 8 and was banging my head against the wall for a few minutes trying to figure out why apache was so unhappy.
The key issue was when I restarted apache2 like I normally would after adding a new configuration it spat out an angry message at me.
root@nuc:/etc/apache2# sudo service apache2 restart Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
The first place that I would look is the error logs. However, in this particular case they were not very helpful.
root@nuc:/etc/apache2# tail -f /var/log/apache2/error.log [Mon May 01 21:00:11.922943 2017] [mpm_prefork:notice] [pid 20454] AH00169: caught SIGTERM, shutting down
Next, I read the error message per the suggestion from the restart command. This was also not very helpful.
root@nuc:/etc/apache2# systemctl status apache2.service ● apache2.service - LSB: Apache2 web server Loaded: loaded (/etc/init.d/apache2) Drop-In: /lib/systemd/system/apache2.service.d └─forking.conf Active: failed (Result: exit-code) since Mon 2017-05-01 21:05:58 PDT; 1min 45s ago Process: 20746 ExecStop=/etc/init.d/apache2 stop (code=exited, status=0/SUCCESS) Process: 20697 ExecReload=/etc/init.d/apache2 reload (code=exited, status=1/FAILURE) Process: 20920 ExecStart=/etc/init.d/apache2 start (code=exited, status=1/FAILURE) May 01 21:05:58 nuc apache2: Starting web server: apache2 failed! May 01 21:05:58 nuc apache2: The apache2 configtest failed. ... (warning). May 01 21:05:58 nuc apache2: Output of config test was: May 01 21:05:58 nuc apache2: apache2: Syntax error on line 219 of /etc/apache2/apache2.conf: Syntax error on line 22 of /etc/a... section May 01 21:05:58 nuc apache2: Action 'configtest' failed. May 01 21:05:58 nuc apache2: The Apache error log may have more information. May 01 21:05:58 nuc systemd: apache2.service: control process exited, code=exited status=1 May 01 21:05:58 nuc systemd: Failed to start LSB: Apache2 web server. May 01 21:05:58 nuc systemd: Unit apache2.service entered failed state. Hint: Some lines were ellipsized, use -l to show in full.
Inspecting the error message, we see that it is unhappy with line 219 of
/etc/apache2/apache2.conf file. Looking at that line we can
see that it is simply loading all of the other config files in
sites-enabled which means that before it even gets to load my new cgit
config file it fails.
So now that we have done some basic troubleshooting. It's time to dig into the manual for further information. I know that the config file is failing to load, and knowing my fat fingers it is very likely a config error on my part. Before reading 200 pages of documentation on the apache website we should take a look at the built in help to see if we can find something of value.
root@nuc:/etc/apache2# apache2 -help Usage: apache2 [-D name] [-d directory] [-f file] [-C "directive"] [-c "directive"] [-k start|restart|graceful|graceful-stop|stop] [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X] Options: -D name : define a name for use in <IfDefine name> directives -d directory : specify an alternate initial ServerRoot -f file : specify an alternate ServerConfigFile -C "directive" : process directive before reading config files -c "directive" : process directive after reading config files -e level : show startup errors of level (see LogLevel) -E file : log startup errors to file -v : show version number -V : show compile settings -h : list available command line options (this page) -l : list compiled in modules -L : list available configuration directives -t -D DUMP_VHOSTS : show parsed vhost settings -t -D DUMP_RUN_CFG : show parsed run settings -S : a synonym for -t -D DUMP_VHOSTS -D DUMP_RUN_CFG -t -D DUMP_MODULES : show all loaded modules -M : a synonym for -t -D DUMP_MODULES -t : run syntax check for config files -T : start without DocumentRoot(s) check -X : debug mode (only one worker, do not detach)
Success! It turns out we can run a linter on a specific config file
root@nuc:/etc/apache2# apache2 -t -f sites-available/git.levlaz.org.conf apache2: Syntax error on line 22 of /etc/apache2/sites-available/git.levlaz.org.conf: </VirtualHost> without matching <VirtualHost> section
Doh! Such a silly mistake with a missing
bracket. Fixing this syntax error resolved the issue. The main takeaway
for me is that the best part about most Linux tools is that they usually
give you everything you need in order to succeed. We were able to
troubleshoot and resolve this issue without resorting to google and
running random commands that stranger posted on the internet 5 years