AKS Egress Internal via Load Balancer to on-premise service

Ed Boykin

8/17/24, 5:54 PM

I have an AKS cluster and one of the pods, call it "my-service", needs to connect to an on-premises service via VPN and that service requires whitelisting of IPs. Unfortunately, it can only whitelist individual address and not ranges. The connectivity between AKS (Azure) and the on-prem service over the VPN is running and verified working. The AKS pod can communicate with the on-prem service. The difficulty we are running into is the on-prem service is seeing the node IP address as the source IP instead of the private IP of a load balancer we created. I have created an internal load balancer for my pod to handle the egress but the on-prem service continues to report the node IP instead of the LB IP. We are running Kubenet for networking and per MSFT docs, seeing the node Id is the normal behavior. Since we cant whitelist the entire AKS subnet and we pay licensing per whitelist IP address, we need to have a 'static' IP source. Seeing the node ID wont work since we cant guarantee the pod to run on the node and we dont want to pin to a single node. Having a 'static' internal IP is what we where trying to do with the load balancer.

Is there a way to setup AKS or my VNet - VPN - On prem networking so that the on-premise service will see the load balancer or other 'thing' as the source IP?

I've followed quite a bit of the MSFT documentation on this subject here: Internal LB AKS Egress

For reference, here's the Load Balancer manifest. Pretty vanilla.

apiVersion: v1
kind: Service
metadata:
  name: my-service-egress
  namespace: my-internal-service-namespace
  labels:
    name: my-service-egress
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
spec:
  type: LoadBalancer
  selector:
    name: my-service
  ports:
    - protocol: TCP
      port: 8999
      targetPort: 8999

0 + 1

azure-networking

aks

Ed Boykin

8/18/24, 10:16 PM

I've done some tests trying to solve. I have AKS with UDR egress. I have a UDR sending all traffic out of nodes to AZ firewall. I have policies SNAT'ing. I'm using RFC 1918 shared ip ranges so I had to remove them from SNAT exclude list. This works but isnt opimtal. The FW picks a private IP from the firewall subnet. The smallest subnet I cant deploy azure FW in has 64 addresses. I could whitelist all 64 on my API...but, part of the licensing agreeming for this API includes fees for EACH whitelisted IP and of course, it doesnt support whitelisting CIDR or IP ranges. :(

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: AKS Egress Internal via Load Balancer to on-premise service

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.