Apache2 Configuration Issue - Deny from external IP, allow by hostname

I've been messing with this for a couple days now and can't seem to get it to work. I have 2 sites enabled via their own config files. site1.conf & site2.conf

Each sites has rewrite rules from Let's Encrypt that route them to site*-le-ssl.conf

Having an issue where when I enter my public IP into the browser it is loading site1 where I want to either have an error like forbidden or something other than it loading my site1.

Site 2 seems to load only when specified by its hostname.

site1.conf is

<VirtualHost *:80>
     ServerAdmin [email protected]
     ServerName site1.mydomian.com
     ServerAlias site1.mydomian.com
     DocumentRoot /var/www/site1
     <Directory /var/www/site1>
         Options FollowSymLinks
         AllowOverride All
         Require all granted
         Header always set X-Frame-Options "SAMEORIGIN"
     </Directory>

     ErrorLog ${APACHE_LOG_DIR}/site1_error.log
     CustomLog ${APACHE_LOG_DIR}/site1_access.log combined

     RewriteEngine on
     RewriteCond %{SERVER_NAME} =site1.mydomian.com
     RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

site1-le-ssl.conf is

<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerAdmin [email protected]
     ServerName site1.mydomian.com
     ServerAlias site1.mydomian.com
     DocumentRoot /var/www/site1/
     <Directory /var/www/site1>
         Options FollowSymLinks
         AllowOverride All
         Require all granted
         Header always set X-Frame-Options "SAMEORIGIN"
     </Directory>

        ErrorLog ${APACHE_LOG_DIR}/site1_error.log
        CustomLog ${APACHE_LOG_DIR}/site1_access.log combined

        SSLCertificateFile /etc/letsencrypt/live/site1.mydomian.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/site1.mydomian.com/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf

</VirtualHost>

</IfModule>

Even when I disabled site1.conf using a2dissite site1 and reloaded apache, this site still loaded when accessed by public ip.

In my apache2.conf file I changed this to Require all denied, but still was loading site1.

<Directory /var/www/>
        Options FollowSymLinks
        AllowOverride None
        Require all denied
</Directory>

When I renamed /var/www/site1 to /var/www/site1.old and then hit by public ip I then got a Forbidden page which is what I would expect by accessing from ip.

Any ideas how to have site1 enabled and accessed by hostname site1.mydomain.com by not by public ip?