How can I restrict access to `/` and also `/index.php` on my server, but allow nothing else?

I would like to allow users to only be able to visit index.php (and also just /) on my server, and nothing else (returning a 403 if valid and a 404 if an invalid file).

How Can I do this? I've tried the following solutions Denying access to all files except index.html apache

Denying access to all files except index.php but allow access through "/" in htaccess file

but they don't seem to have any effect.

My .htaccess file is this, but it is having no effect and I can still access other files on my server:

Order allow,deny
Deny from all
<FilesMatch index\.php>
        Allow from all
</FilesMatch>

I'm not sure if I need to define anything in my virtualhosts config as well?

My virtualhosts file is:

ServerAdmin webmaster@localhost
ServerName server.mydomain.com
ServerAlias server.mydomain.com
DocumentRoot /var/www/server.mydomain.com
DirectoryIndex index.php
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

SSLCertificateFile /etc/letsencrypt/live/server.mydomain.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/server.mydomain.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

apache2ctl -S output is:

VirtualHost configuration:
*:443                  server.mydomain.com (/etc/apache2/sites-enabled/server.mydomain.com-le-ssl.conf:2)
*:80                   server.mydomain.com (/etc/apache2/sites-enabled/server.mydomain.com.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

IMHO you're going about things the wrong way and are trying to solve a problem that can be avoided completely:

Simply don't store files and directories that don't belong online in your DocumentRoot.

Limit the contents of the directory /var/www/server.example.com to only that index.php file and you can't go wrong.

This question is tagged with the historical apache-2.2 version and syntax from that is not suitable for the current Apache httpd 2.4 release. - See https://httpd.apache.org/docs/2.4/upgrading.html

When you do want to publish something under http://www.example.com/some-path/ use the Alias directive to expose a different directory altogether rather than creating a sub directory /var/www/server.example.com/some-path/ (for example create /var/www//some-path) and use:

<VirtualHost *:80
    ServerAdmin webmaster@localhost
    ServerName server.mydomain.com
    ServerAlias server.mydomain.com
    DocumentRoot /var/www/server.example.com

    Alias /some-path "/var/www/some-path"

    <Directory "/var/www/some-path"> 
       # add settings here, for example
       Order deny,allow
       Deny from all
       Allow From 127.0.0.1
    </Directory>
</VirtualHost>