Istio: How do I exclude unhealthy destination from a VirtualService?

I'm trying to configure load balancing and failover for external services. Each HTTP endpoint for the service needs its own specific headers.

I created a virtual service with two destinations:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: test-external
spec:
  hosts:
  - test-external.com
  http:
  - headers:
      request:
        set:
          test: "true"
    route:
    - destination:
        host: "201.returnco.de"
      weight: 50
      headers:
        request:
          set:
            Host: "201.returnco.de"
            api-key: "xxxxxxxxxx"
    - destination:
        host: "501.returnco.de"
      weight: 50
      headers:
        request:
          set:
            Host: "501.returnco.de"
            api-key: "yyyyyyyyyy"
    retries: {}

The hosts 201.returnco.de and 501.returnco.de are external services, so I created a service entry for them.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: test-external
spec:
  hosts:
  - test-external.com
  - 201.returnco.de
  - 501.returnco.de
  location: MESH_EXTERNAL
  ports:
  - name: http
    number: 80
    protocol: HTTP
  resolution: DNS

What I want is to route requests only to 201.returnco.de. The requests should not be routed to a host which returns 5xx status code. In this case, 501.returnco.de always returns 5xx status code, so it is considered unhealthy.

How should I configure the mesh?

I tried making the following destination rule, but this doesn't work.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: test-external
spec:
  host: "*.returnco.de"
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 15s
      maxEjectionPercent: 100

The mesh considers 201.returnco.de and 501.returnco.de as two separate services. After the unhealthy endpoint for the host 501.returnco.de is evicted, Istio proxy returns 503 error for requests because there are no healthy endpoints.

Configuring multiple endpoints for a single service is not ideal because I need to set different headers for each endpoint.

From your configs and description, it is understood that you are creating multiple endpoints for the same service. In this scenario if you use “*.returnco.de” in your DestinationRule for OutlierDetection, when you receive 5xx errors for the endpoint 501.returnco.de as you mentioned it will evict the pods. Since 201.returnco.de is another endpoint of the same service which inturn relies on the same pods you are getting 503 errors because the pods are already evicted. In order to prevent the pods from getting evicted you can directly mention 201.returnco.de in your DestinationRule as we already know 501.returnco.de returns the 5xx errors and will evict the pods, below is the modified DestinationRule for your reference

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: test-external
spec:
  host: 201.returnco.de
  trafficPolicy:
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      consecutiveGatewayErrors: 1
      interval: 15s
      maxEjectionPercent: 100

Also if you don’t want your requests to be routed to 501.returnco.de you can give less weight such as 0 to 501.returnco.de rule and some high weight such as 100 to 201.returnco.de rule or use the explicit deny option of Istio.