I just upgraded my development workstation from Kubuntu 18.04 to Kubuntu 22.04.2 LTS.
For context, I keep /home on a separate disk from /. The previous / is still present at /mnt/sdc5.
Kubuntu 18.04 was running:
$ /mnt/sdc5/usr/sbin/apache2 -v
Server version: Apache/2.4.56 (Ubuntu)
Server built: 2023-03-09T07:33:5
Whereas 22.04 is running:
$ apache2 -v
Server version: Apache/2.4.52 (Ubuntu)
Server built: 2023-03-08T17:32:01
(Yes, as of this past weekend, Bionic apparently had a newer apache2 than Jammy).
I have copied all the apache/ssl configs from /mnt/sdc5, as well as all the web roots (I have a several vhosts set up) from /mnt/sdc5/var/www/ to their functional location in /var/www/. As in the previous environment, symlinks under /var/www/*/ cross physical disks from sda5 to my ~ in sdb5 (/home/dracos/).
I keep pretty much everything under /var/www/* owned by dracos:www-data, and I have put myself in the www-data group:
$ groups dracos
dracos : dracos adm cdrom sudo dip www-data plugdev lpadmin lxd sambashare
Apache runs fine, except browsing any symlink returns a 403 error (ie, http://ref.harad/knotwork):
[Wed Mar 29 17:54:02.090919 2023] [core:error] [pid 439254] [client 192.168.0.21:51496] AH00037: Symbolic link not allowed or link target not accessible: /var/www/ref/htdocs/knotwork
In this case, htdocs is a directory, while knotwork is the symlink:
$ ll /var/www/ref/htdocs/|grep knotwork
lrwxrwxrwx 1 dracos www-data 30 Jun 18 2021 knotwork -> /home/dracos/graphics/knotwork
And the full path to it:
$ stat /var/
File: /var/
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 805h/2053d Inode: 2883585 Links: 15
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-03-29 11:03:25.026834422 -0600
Modify: 2023-03-27 22:03:04.511844496 -0600
Change: 2023-03-27 22:03:04.511844496 -0600
Birth: 2023-03-25 22:09:41.157329268 -0600
$ stat /var/www/
File: /var/www/
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 805h/2053d Inode: 3277054 Links: 24
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-03-29 15:47:21.907537823 -0600
Modify: 2023-03-29 15:46:51.232613614 -0600
Change: 2023-03-29 15:46:51.232613614 -0600
Birth: 2023-03-27 22:03:04.511844496 -0600
$ stat /var/www/ref
File: /var/www/ref
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 805h/2053d Inode: 3285520 Links: 6
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-03-29 15:47:22.487517480 -0600
Modify: 2021-05-01 11:12:49.857126723 -0600
Change: 2023-03-29 15:47:09.575970401 -0600
Birth: 2023-03-29 15:46:51.232613614 -0600
$ stat /var/www/ref/htdocs
File: /var/www/ref/htdocs
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 805h/2053d Inode: 3287655 Links: 41
Access: (0775/drwxrwxr-x) Uid: ( 1000/ dracos) Gid: ( 33/www-data)
Access: 2023-03-29 17:54:02.380602649 -0600
Modify: 2023-01-13 15:20:42.794065309 -0700
Change: 2023-03-29 17:53:56.380789344 -0600
Birth: 2023-03-29 15:46:51.724596365 -0600
$ stat /var/www/ref/htdocs/knotwork
File: /var/www/ref/htdocs/knotwork -> /home/dracos/graphics/knotwork
Size: 30 Blocks: 0 IO Block: 4096 symbolic link
Device: 805h/2053d Inode: 3287656 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ dracos) Gid: ( 33/www-data)
Access: 2023-03-29 17:54:02.084611865 -0600
Modify: 2021-06-18 16:02:41.686238932 -0600
Change: 2023-03-29 17:53:56.308791583 -0600
Birth: 2023-03-29 15:46:51.724596365 -0600
In apache2.conf:
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
And the full config for this vhost (001-ref.conf):
<VirtualHost ref.harad:80>
ServerName ref.harad
ServerAdmin webmaster@localhost
DocumentRoot /var/www/ref/htdocs
<Directory />
Options FollowSymLinks Includes
AllowOverride None
</Directory>
<Directory /var/www/ref/htdocs/>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
#Order allow,deny
#allow from all
Require all granted
</Directory>
<Directory /var/www/ref/htdocs/_flyspray/>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
#Order allow,deny
#allow from all
Require all granted
</Directory>
ErrorLog /var/log/apache2/error_ref.log
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn
CustomLog /var/log/apache2/access_ref.log combined
</VirtualHost>
In this file I commented out the Order and allow lines in favor of the Require line below, which made no difference.
There are no configuration errors:
$ apachectl configtest
Syntax OK
Apache is running:
$ systemctl status apache2.service
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-03-29 17:44:29 MDT; 49min ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 439245 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 439250 (/usr/sbin/apach)
Tasks: 7 (limit: 38327)
Memory: 42.0M
CPU: 917ms
CGroup: /system.slice/apache2.service
├─439250 /usr/sbin/apache2 -k start
├─439251 /usr/sbin/apache2 -k start
├─439252 /usr/sbin/apache2 -k start
├─439253 /usr/sbin/apache2 -k start
├─439254 /usr/sbin/apache2 -k start
├─439255 /usr/sbin/apache2 -k start
└─439896 /usr/sbin/apache2 -k start
Mar 29 17:44:29 harad systemd[1]: Starting The Apache HTTP Server...
Mar 29 17:44:29 harad systemd[1]: Started The Apache HTTP Server.
Permissions leading to knotwork in /home:
$ stat /home
File: /home
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 812h/2066d Inode: 2 Links: 8
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-03-29 10:33:09.874123424 -0600
Modify: 2023-03-25 22:13:38.754784783 -0600
Change: 2023-03-25 22:13:38.754784783 -0600
Birth: 2019-03-03 08:52:01.000000000 -0700
$ stat /home/dracos
File: /home/dracos
Size: 24576 Blocks: 48 IO Block: 4096 directory
Device: 812h/2066d Inode: 103153665 Links: 116
Access: (0750/drwxr-x---) Uid: ( 1000/ dracos) Gid: ( 1000/ dracos)
Access: 2023-03-29 11:03:19.023044685 -0600
Modify: 2023-03-29 11:03:17.423100718 -0600
Change: 2023-03-29 11:03:17.423100718 -0600
Birth: 2023-03-25 22:13:38.750781079 -0600
$ stat /home/dracos/graphics
File: /home/dracos/graphics
Size: 36864 Blocks: 72 IO Block: 4096 directory
Device: 812h/2066d Inode: 94372596 Links: 16
Access: (0755/drwxr-xr-x) Uid: ( 1000/ dracos) Gid: ( 1000/ dracos)
Access: 2023-03-29 15:48:58.536144219 -0600
Modify: 2023-02-25 15:55:31.878748109 -0700
Change: 2023-03-26 02:15:19.817837898 -0600
Birth: 2019-03-03 17:58:31.009967662 -0700
harad /etc/apache2
$ stat /home/dracos/graphics/knotwork
File: /home/dracos/graphics/knotwork
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 812h/2066d Inode: 124257516 Links: 2
Access: (0775/drwxrwxr-x) Uid: ( 1000/ dracos) Gid: ( 1000/ dracos)
Access: 2023-03-29 10:32:38.795209595 -0600
Modify: 2022-04-09 19:03:00.488673788 -0600
Change: 2023-03-10 20:49:52.217436682 -0700
Birth: 2021-01-17 12:55:57.612873620 -0700
I've spent over 3 hours trying to figure this out. The config is clean. Permissions are the same as before. I don't understand why I can't browse symlinks. Once I solve this, all my other vhosts (which all have symlinks) will be functional again.
I've been able to solve this myself several times in the past. What obscure detail have I overlooked after reading through two dozen threads looking for the answer?
