Apache2 configuration to redirect requests with host being an IP

I have the following Apache2 config:

000-default.conf

<VirtualHost _default_:80>
        RewriteEngine on
        RewriteRule ^ https://example.com [END,NE,R=permanent]
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost _default_:443>
        SSLEngine on
        SSLCertificateFile     /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

        RewriteEngine on
        RewriteRule ^ https://example.com [END,NE,R=permanent]
</VirtualHost>
</IfModule>

example.com.conf

<VirtualHost *:80>
        ServerName example.com

        RewriteEngine on
        RewriteCond %{SERVER_NAME} =example.com
        RewriteRule ^(.*)$ https://example.com$1 [END,NE,R]
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerName example.com

        <IfModule mpm_itk_module>
                AssignUserId someuser someuser
        </IfModule>

        <Directory /something>
                <Files "wsgi.py">
                        Require all granted
                </Files>
        </Directory>

        WSGIDaemonProcess example.com python-path=/something:/something/python3.9/site-packages socket-user=someuser user=someuser group=someuser
        WSGIProcessGroup example.com
        WSGIScriptAlias / /something/wsgi.py

        SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

My goal is to have the following (where /XXX is any path):

https://example.com/XXX -> serving by WSGI
https://example.com/XXX -> redirect to https://example.com/XXX
anything else -> redirect to https://example.com

Alternatively, I'd be even happier with anything else just giving HTTP 404 or HTTP 403.

However, this configuration yields the following (assuming the public IP address is 1.2.3.4):

http://1.2.3.4/XXX -> redirect to https://example.com (OK) 
https://1.2.3.4/XXX -> serving by WSGI (WRONG)
http://example.com/XXX -> redirect to https://example.com (WRONG)
https://example.com/XXX -> serving by WSGI (OK)

Why are the redirections working as they are? How can I fix it?