Docker containers no internet after adding a static IP, netplan (host)

I am a newbie with netplan.
The context is that I want to add a failover IP (provided by OVH) to one of my server to provide HA (this IP will be failover/moved between all of my servers, only one server will hold this IP).
I followed this guide to configure my IP failover on my servers: https://docs.ovh.com/us/en/public-cloud/configure_a_failover_ip/#ubuntu-2004_1 (my servers are running under ubuntu)

The problem is when I add this configuration to my server AND that the IP Failover is not hold by the server then all docker containers in my server doesn't have internet (but the host does..).
And the only server that has the Failover IP works perfectly (docker has internet)

My servers configuraion looks like this now (with the IP configured):
/etc/netplan/50-cloud-init.yaml

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens3:
            accept-ra: false
            addresses:
            - xxxx:xxxx:xxx:xxx::xxx/56 # maybe sensitive data
            - X.X.X.X/32 # MY FAILOVER IP
            dhcp4: true
            match:
                macaddress: xx:xx:xx:xx:xx:xx # blur my macaddr, idk if its sensitive data
            mtu: 1500
            nameservers:
                addresses:
                - 213.186.33.99 # idk what this is
                search: []
            routes:
            -   to: ::/0
                via: x:x:x:x::1 # maybe sensitive data
            set-name: ens3