Need assistance configuring Istio Egress gateway with mTLS for communication to external cluster APIs

[Kalin-the-Builder]

Hello everyone,

We have Istio configured on our Kubernetes cluster, with both Ingress and Egress gateways enabled (see IstioOperator configuration below). Our Ingress gateway is functioning as desired, with mTLS enabled, ensuring secure communication from clients to our Kubernetes cluster.

However, we're encountering difficulties configuring the Egress gateway to establish mTLS communication with another external cluster where our APIs reside. Despite our efforts, we haven't been successful, and we're not seeing any communication activity in the logs of the Egress Controller.

Problem:

We are attempting to implement mTLS for communication from our Kubernetes cluster to an external cluster where our APIs are hosted on Port 8443.
Despite configuring the Egress gateway with mTLS settings, we're not seeing any communication activity in the logs of the Egress Controller.
We have ensured that the necessary certificates are available and properly configured on both ends.

IstioOperator Configuration:

---
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          resources:
            requests:
              cpu: 200m
              memory: 2Gi
            limits:
              cpu: 400m
              memory: 4Gi
          service:
            type: LoadBalancer
            externalTrafficPolicy: Local
    egressGateways:
      - name: istio-egressgateway
        enabled: true
  values:
    global:
      defaultNodeSelector:
        name: sysnodepool
  meshConfig:
    accessLogFile: /dev/stdout
    outboundTrafficPolicy:
       mode: REGISTRY_ONLY

Service Entry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: example
spec:
  hosts:
    - "${env}.example.${api}.net"
  ports:
    - number: 8443
      name: tls
      protocol: TLS
  resolution: DNS
  location: MESH_EXTERNAL

Egress Gateway:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: egress-gateway
spec:
  selector:
    istio: istio-egressgateway
  servers:
    - hosts:
        - "${env}.example.${api}.net"
      port:
        number: 8443
        name: tls-egress
        protocol: TLS
      tls:
        mode: MUTUAL
        credentialName: tls-secret-example-${env}

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: example-internal-mtls-dest-rule
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
    - name: example
      trafficPolicy:
        loadBalancer:
          simple: ROUND_ROBIN
        portLevelSettings:
          - port:
              number: 8443
            tls:
              mode: ISTIO_MUTUAL
              sni: "${env}.example.${api}.net"

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: example-external-mtls-dest-rule
spec:
  host: "${env}.example.${api}.net"
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
      - port:
          number: 8443
        tls:
          mode: MUTUAL
          credentialName: tls-secret-example-${env}
          sni: "${env}.example.${api}.net"

Virtual Service:

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: example-virtual-service
spec:
  hosts:
    - "${env}.example.${api}.net"
  gateways:
    - mesh
    - example-egress-gateway
  http:
    - match:
        - gateways:
            - mesh
          port: 443
      route:
        - destination:
            host: istio-egressgateway.istio-system.svc.cluster.local
            subset: idg
            port:
              number: 443
          weight: 100
    - match:
        - gateways:
            - example-egress-gateway
          port: 443
      route:
        - destination:
            host: "${env}.example.${api}.net"
            port:
              number: 8443
          weight: 100

We would appreciate any assistance or guidance on properly configuring the Istio Egress gateway for mTLS communication with external cluster APIs. Additionally, any insights into troubleshooting steps or common pitfalls would be greatly appreciated.

Thank you in advance for your help!

[abasitt]

the first step is to make sure within a mesh the traffic is hitting the egress gateway. I suggest start from the source sidecar and see if the traffic is routed correctly to the egress gateway via the VS that you defined.

[Kalin-the-Builder]

I can confirm no traffic is hitting the egress gateway. I've followed the the guides (https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/#configure-mutual-tls-origination-for-egress-traffic) and even still I'm not getting anything in my Egress Logs. Am I missing something? I've got two namespace the 'environment' and 'istio-system' do these configuration files need to be applied on specific namespaces?

[rootsongjc]

Since you don't see any communication activity in the logs of the egress controller, it means that the traffic is not being routed to Egress Gateway correctly. I think you can check if there is a network policy or firewall rule organizing the Egress traffic. Also you can increase the log level of the Egress Gateway pod and analyze it more specifically based on the logs.

istioctl pc log <Egress Pod Name> --level=debug

[Kalin-the-Builder]

Definitely not getting rooted correctly.

Would adding this to my MeshConfig help?

outboundTrafficPolicy: mode: REGISTRY_ONLY