Join Log in
Score:1
M.A. Heshmat Khah avatar Server

Envoy to envoy TLS for TLS proxy

us flag
M.A. Heshmat Khah

Assume this artitecture

                             ................................................
                             .                                              .
                             .                                              .
                             .                                              .
                             .                                              .
                   ┌───────────────────┐                            ┌─────────────────┐
┌──────────────────┤ Customer Firewall ├─┐     ┌────────────────────┤Provider Firewall├─┐
│                  └──────────┬────────┘ │     │Provider Network    └──┬──────────────┘ │
│   Customer Network          │          │     │             ┌─────────┴─┐              │
│                             │          │     │       ┌─────┤ Envoy - B ├───────┐      │
│                             │          │     │       │     └─────┬─────┘       │      │
│                      ┌──────┴─────┐    │     │       │           │             │      │
│       ┌─────────┐    │            │    │     │       │           │             │      │
│       │ Client  ├────┤  Envoy - A │    │     │  ┌────┴────┐  ┌───┴─────┐  ┌────┴────┐ │
│       └─────────┘    │            │    │     │  │ Service │  │ Service │  │ Service │ │
│                      └────────────┘    │     │  └─────────┘  └─────────┘  └─────────┘ │
│                                        │     │                                        │
└────────────────────────────────────────┘     └────────────────────────────────────────┘

The client wants to connect to services at the provider network, but the Customer Firewall has a strict policy on HTTP filtering and SNI filtering.

It will drop any unauthorized traffic, So any unallowed Domain/SNI will be dropped.

The SNI/Domain of Envoy B (envoy-b.example.com) is allowed in the firewall, but they don't Allow any other domain.

All services are HTTP/HTTPS

Here is the configuration for the envoy servers:

Envoy - A:

  • It listens on port 80, and on every HTTP traffic, it will forward the traffic to Envoy - B using HTTP2 + TLSv1.3
  • It listens on port 443, and on every TPC traffic, it will forward the traffic to Envoy - B without any further encryption
static_resources:
  listeners:
    # HTTP
    - name: http_forwarder
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 80
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: http_route
                  virtual_hosts:
                    - name: http_forwarder
                      domains: [ "*" ]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: forwarder_to_envoy_b_http
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

    # HTTPS/TLS
    - name: https_forwarder
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 443
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                cluster: forwarder_to_envoy_b_tls
                stat_prefix: https_passthrough
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog

  clusters:
    - name: forwarder_to_envoy_b_http
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: forwarder_to_envoy_b_http
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: envoy-b.example.com
                      port_value: 8888

      dns_resolution_config:
        resolvers:
          - socket_address:
              address: "1.1.1.1"
              port_value: 53
        dns_resolver_options:
          no_default_search_domain: true
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_params:
              tls_minimum_protocol_version: TLSv1_3
              tls_maximum_protocol_version: TLSv1_3

    - name: forwarder_to_envoy_b_tls
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: forwarder_to_envoy_b_tls
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: envoy-b.example.com
                      port_value: 8889
      dns_resolution_config:
        resolvers:
          - socket_address:
              address: "1.1.1.1"
              port_value: 53
        dns_resolver_options:
          no_default_search_domain: true

Envoy - B:

  • It listens on port 8888, and on every HTTP2 + TLSv1.3 traffic, it will decrypt the traffic and forward the plain HTTP request to services
  • It listens on port 8889, and on every TPC traffic, it will forward the traffic to Envoy - B without any further encryption/decryption
static_resources:
  listeners:
    - name: http_listener_0
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 8888
        filter_chains:
          - filters:
              - name: envoy.filters.network.http_connection_manager
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                  stat_prefix: ingress_http
                  route_config:
                    name: local_route
                    virtual_hosts:
                      - name: local_service
                        domains: [ "*" ]
                        routes:
                          - match:
                              prefix: "/"
                            route:
                              cluster: http_cluster
                  http_filters:
                    - name: envoy.filters.http.router
                      typed_config:
                        "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            
            transport_socket:
              name: envoy.transport_sockets.tls
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
                common_tls_context:
                  tls_params:
                    tls_minimum_protocol_version: TLSv1_3
                    tls_maximum_protocol_version: TLSv1_3
                  tls_certificates:
                    certificate_chain:
                      filename: /certs/proxy-crt.pem
                    private_key:
                      filename: /certs/proxy-key.pem
                  alpn_protocols: HTTP2


    - name: tls_listener_0
      address:
        socket_address:
          protocol: TCP
          address: 0.0.0.0
          port_value: 8889
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                cluster: tls_cluster
                stat_prefix: https_passthrough
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog

  clusters:
    - name: http_cluster
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: http_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: private.service.example.com
                      port_value: 80

      dns_resolution_config:
        resolvers:
          - socket_address:
              address: "1.1.1.1"
              port_value: 53
        dns_resolver_options:
          no_default_search_domain: true


    - name: tls_cluster
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: tls_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: private.service.example.com
                      port_value: 443
      dns_resolution_config:
        resolvers:
          - socket_address:
              address: "1.1.1.1"
              port_value: 53
        dns_resolver_options:
          no_default_search_domain: true

The problem

When the client sends an HTTP request, Envoy - A will encrypt it, and the SNI will be envoy-b.example.com which is allowed in the client firewall (instead of the HOST header of the original request private.service.example.com)

But when the client sends an HTTPS/TLS traffic, Envoy - A is just forwarding a TCP stream, so the SNI doesn't change and stays private.service.example.com -> As this hostname is not allowed in the firewall it will be dropped!

I want the traffic of the TCP proxy to get encrypted again so the SNI changes to envoy-b.example.com and the firewall doesn't drop the request.

I tried to add transport_socket for TLS traffic on both envoy hosts but it didn't work:

Envoy A > clusters > forwarder_to_envoy_b_tls:


      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          common_tls_context:
            tls_params:
              tls_minimum_protocol_version: TLSv1_3
              tls_maximum_protocol_version: TLSv1_3

Envoy B > listener > tls_listener_0:


            transport_socket:
              name: envoy.transport_sockets.tls
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
                common_tls_context:
                  tls_params:
                    tls_minimum_protocol_version: TLSv1_3
                    tls_maximum_protocol_version: TLSv1_3
                  tls_certificates:
                    certificate_chain:
                      filename: /certs/proxy-crt.pem
                    private_key:
                      filename: /certs/proxy-key.pem
                  alpn_protocols: HTTP2

Notes

  1. The client doesn't accept that envoy B converts HTTPS traffic to HTTP, then transfers it (using HTTPS), then again HTTPS encryption happens at Envoy A
  2. We don't have access to service private keys, so if we do what is explained in note 1, the certificates changes
  3. The Envoy B REQUIRES SNI to determine the traffic should forward to which services
648
0 + 0
sni
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.