mTLS origination from sidecar failing to an egress service (outside the OCP cluster) and giving the 403 forbidden error.

[AkshaySingh-DS]

HI team,

From couple of days I'm facing an issue while trying to make the connectivity with a web server which uses a rest api F5 URL from my application which is deployed into openshift cluster and using service mesh product istio.

Things were working fine when we were on simple TLS, the issue came when we switched to mTLS.

So what we're doing:-

Basically we are trying to make mTLS connectivity from a react application (deployed into opneshift cluster) to IBM FileNet server (which support mTLS and outside the cluster).

I followed the istio docs and trying to originate TLS from sidecar ( https://istio.io/latest/docs/tasks/traffic-management/egress/egress-tls-origination/).

We have created the below mesh resources as per the docs:-

Service entry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: originate-mtls-for-my-service
spec:
  hosts:
  - my-external-host-name
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
    targetPort: external-host-port
  - number: external-host-port
    name: https-port
    protocol: HTTPS
  resolution: DNS

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-my-service
spec:
  workloadSelector:
    matchLabels:
      app: my-app-label
  host: my-external-host-name
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 80
      tls:
        mode: MUTUAL
        credentialName: client-credential # this must match the secret created earlier to hold client certs, and works only when DR has a workloadSelector
        sni: my-external-host-name

However after trying above configs react app (deployed on OCP) can't able to talk to external service and we are getting "403 Forbidden error"

While we hit the curl URL from sidecar proxy we got below response.

curl -iv FileNet_host:FileNet_port

• connected to FileNet_host:FileNet_port
• ALPN offering h2
• ALPN offering http/1.1
• successfully set certificates verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
• CApath: none
• TLS1.3 (OUT), TLS handshake, client hello (1):
• TLS1.3 (IN), TLS handshake, server hello (2):
• TLS1.3 (IN), TLS handshake, [no content] (0):
• TLS1.3 (IN), TLS handshake, Encrypted Extension (8):
• TLS1.3 (IN), TLS handshake, Request CERT (13):
• TLS1.3 (IN), TLS handshake, Certificate (11):
• TLS1.3 (OUT), TLS handshake, unknown CA (560):
• SSL certificate Problem: self signed certificate in Chain
• Closing connection (0)

I tried to see such issue on istio/discuss page however couldn't be able to find anything like what i'm If facing. If somebody please look at on this and suggest me on right direction that would really help, Thanks in advance!!

[howardjohn]

Your curl logs indicate curl itself is doing the TLS. What you want is to send to plaintext port 80, then redirect to port 443, then have Istio add TLS.

Your DR seems wrong:

    portLevelSettings:
    - port:
        number: 80

should be configured on the 'external-host-port' -- I assume this is a number in reality? If not, it needs to be a number.

Your curl should use port 80 and http

[AkshaySingh-DS]

Thanks @howardjohn the issue got resolve now, I'm able to make the connectivity from react app to external FileNet server. The above Istio settings was correct only. And I did not change the port in DR.

So basically the react app was sending the https traffic to sidecar at port 80. And what I wanted was, app should send the plain http traffic to sidecar at port 80 and then sidecar would do the TLS origination.

So when developer changed the endpoint https to http in the API call we were able to get the 201 response from external FileNet server.

The above curl output was misleading as we were hitting the https request from sidecar envoy proxy (This was the catch because in app image we were not having the CURL to test so thought of top hit from sidecar which is inccorrect)