We have 2 AKS clusters (not private) deployed in West Europe and East US, without availability zones, Azure CNI networking, Azure network policy (not calico), and everything is provisioned with Terraform.
We are trying to set up an infrastructure for MongoDB replica sets, something similar to this article: https://docs.couchbase.com/operator/2.0/tutorial-remote-dns.html
We need pods in each cluster to be able to access Service IPs (Cluster IPs), but from what we understand, this is not easily achievable, ClusterIPs exist in a virtual network, they are not part of any defined subnet. Is there a way around this? I'm sure this is a common scenario, running Mongo/RabbitMQ/Redis clusters across regions (we're looking to achieve master-master replication of data).
I've also taken a look into a solution using CoreDNS plugin Kubernetai: https://www.cockroachlabs.com/blog/experience-report-running-across-multiple-kubernetes-clusters/ (solution no. 4)
But, this is not built-in the custom CoreDNS image that AKS uses (1.6.6), unfortunately.
The Virtual Networks for the cluster are peered, the pods can access each other via the pod IPs, but we can't access services via their assigned IPs.
VNET West EU: 10.0.0.0/8 Pods subnet: 10.240.0.0/16 Services subnet: 10.0.0.0/16
VNET East US: 20.0.0.0/8 Pods subnet: 20.240.0.0/16 Services subnet: 20.0.0.0/16
Peering is done both ways, allow-vnet-access is set to true.
