Apache2 SSL certificate error when other website site is enabled , server not picking up the right certification or configuration

I have 2 SSL configuration files for 2 sites. Both SSL are covering www and non-www versions of the sites.

The first SSL one works for SiteA. As soon as I enabled Site B, it looks like my server uses the certificate for Site A and therefore says it is invalid and an insecure connexion,etc.

Below is the config file of site a :

<VirtualHost *:80>
    ServerName site-a.com
    Redirect permanent / https://www.site-a.com/
    DocumentRoot /var/www/site-a.com/
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/site-a.com/>
        Options Indexes FollowSymLinks MultiViews
        Options Indexes FollowSymLinks
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>


<IfModule mod_ssl.c>
    <VirtualHost *:443>

        ServerName site-a.com
        DocumentRoot /var/www/site-a.com/

        SSLEngine on
        SSLCertificateFile /etc/ssl/certs/site_a.crt
        SSLCertificateKeyFile /etc/ssl/private/www.site-a.com.key
        SSLCACertificateFile /etc/ssl/certs/site_a.ca-bundle

        DocumentRoot /var/www/site-a.com/
        ErrorLog ${APACHE_LOG_DIR}/access_site_a_.log
        CustomLog ${APACHE_LOG_DIR}/access_site_a_.log combined
        
        <Directory /var/www/site-a.com/>
            Options Indexes FollowSymLinks MultiViews
            AllowOverride All
            Order allow,deny
            allow from all
        </Directory>
        
        <FilesMatch "\.(cgi|shtml|phtml|php)$">
            SSLOptions +StdEnvVars
        </FilesMatch>
        
        <Directory /usr/lib/cgi-bin>
            SSLOptions +StdEnvVars
        </Directory>
    </VirtualHost>
</IfModule>

I disabled site A to check if that changed something. It made SiteB.com work. If I go to SiteB.com , it now gives the right certificate, no more errors , but only if SiteA is disabled.

And then now , if I enable SiteA again and go to SiteA.com it says the certificate is invalid because it is a certificate for SiteB , etc , just like before.

Is there something I should do in the etc/apache hosts file ? Could it be because of something to do with CFQDN ? It looks like the server checks the name / domain / something , tries to match it and stops ?

I am not using default-ssl.conf (I have renamed it) , I had read that it shouldn't be used/not necessary if the SSL configuration is in the vhost configuration ( ie: sitea.com.conf ) .