Cannot get Istio Gateway/VirtualService to work with a Let's encrypt certificate #50563
-
Hi, I am relatively new to Kubernetes and Istio. About a month ago I installed an on-prem Cluster with 3 masters and 3 workers. Then I installed the cert-manager from Helm. I have a ClusterIssuer for prod and staging. I have done a deployment that works with Istio and I can connect from my remote HAproxy LBR to the Istio. backend be_nauthilus_prod from defaults_nauthilus
option httpchk
# http-check connect ssl alpn h2,http/1.1
http-check send meth GET uri /ping ver HTTP/1.1 hdr Host login.authserv.me body "pong"
http-check expect status 200
cookie server_used dynamic indirect nocache insert
dynamic-cookie-key *******
# server kn1.roessner-net.de 192.168.0.180:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
# server kn2.roessner-net.de 192.168.0.181:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
# server kn3.roessner-net.de 192.168.0.182:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
server kn1.roessner-net.de 192.168.0.180:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions
server kn2.roessner-net.de 192.168.0.181:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions
server kn3.roessner-net.de 192.168.0.182:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions Istio was installed like this (Version 1.21.1): istioctl install \
--set meshConfig.accessLogFile=/dev/stdout \
--set values.gateways.istio-ingressgateway.type=NodePort Here is a part of my deployment-manifest: apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nauthilus
namespace: auth
spec:
selector:
matchLabels:
app: nauthilus
template:
metadata:
labels:
app: nauthilus
...
ports:
- containerPort: 9080
...
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: login-authserv-tls
namespace: auth
spec:
dnsNames:
- login.authserv.me
issuerRef:
kind: ClusterIssuer
name: letsencrypt-staging
secretName: login-authserv-tls
---
apiVersion: v1
kind: Service
metadata:
name: nauthilus-service
namespace: auth
spec:
selector:
app: nauthilus
ports:
- name: nt-http
port: 9080
targetPort: 9080
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: nauthilus-gateway
namespace: auth
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- login.authserv.me
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- login.authserv.me
tls:
mode: SIMPLE
credentialName: login-authserv-tls
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nauthilus-virtual-service
namespace: auth
spec:
hosts:
- login.authserv.me
gateways:
- nauthilus-gateway
http:
- match:
- headers:
Host:
exact: login.authserv.me
route:
- destination:
host: nauthilus-service
port:
number: 9080
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: nauthilus-permissive
namespace: auth
spec:
selector:
matchLabels:
app: nauthilus
mtls:
mode: PERMISSIVE # Currently there doe not exist any default PeerAuthentication in the cluster! The certificate was created: kubectl -n auth get certificate 15:24:34
NAME READY SECRET AGE
login-authserv-tls True login-authserv-tls 6h49m Listeners call: istioctl pc listeners -n istio-system istio-ingressgateway-d7cd98789-nbgqj
ADDRESSES PORT MATCH DESTINATION
0.0.0.0 8080 ALL Route: http.8080
0.0.0.0 8443 SNI: login.authserv.me Route: https.443.https.nauthilus-gateway.auth
0.0.0.0 15021 ALL Inline Route: /healthz/ready*
0.0.0.0 15090 ALL Inline Route: /stats/prometheus* Secrets: istioctl pc secret -n istio-system istio-ingressgateway-d7cd98789-nbgqj
zsh: correct 'secret' to 'secrets' [nyae]? n
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
default Cert Chain ACTIVE true ae46e57af984b867501b383411056ea0 2024-04-20T13:01:57Z 2024-04-19T12:59:57Z
kubernetes://login-authserv-tls Cert Chain ACTIVE true 2b7caef2b0c086605e60b148463ed3554ebe 2024-07-18T06:51:24Z 2024-04-19T06:51:25Z
ROOTCA CA ACTIVE true 8e5399a24d2d28f618e0771fd4c8abfb 2034-04-16T13:01:48Z 2024-04-18T13:01:48Z Pods in istio-system: kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-6f68dfd8f4-l7mw7 1/1 Running 0 21h
istio-ingressgateway-d7cd98789-nbgqj 1/1 Running 0 24h
istiod-68cb48856d-8vmqm 1/1 Running 0 24h
jaeger-7d7d59b9d-q9htd 1/1 Running 0 21h
kiali-588bc98cd-6hljc 1/1 Running 0 21h
prometheus-7545dd48db-hz7cv 2/2 Running 0 21h Pods in auth: kubectl get pods -n auth
NAME READY STATUS RESTARTS AGE
nauthilus-dlwcd 2/2 Running 3 (6h59m ago) 22h
nauthilus-f7fng 2/2 Running 1 (10h ago) 22h
nauthilus-vw5km 2/2 Running 1 (34m ago) 22h When trying to connect to the service using HTTPS, I see the follwing errors in my istio-gateway logs: kubectl logs istio-ingressgateway-d7cd98789-nbgqj -n istio-system | grep filter_chain_not_found | tail
[2024-04-19T10:13:22.645Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 9 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:1394 - -
[2024-04-19T10:13:23.683Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 6 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:28265 - -
[2024-04-19T10:13:24.708Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:53749 - -
[2024-04-19T10:13:25.737Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 6 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:35022 - -
[2024-04-19T10:13:26.761Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:50896 - -
[2024-04-19T10:13:27.784Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:63985 - -
[2024-04-19T10:13:28.807Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 4 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:39671 - -
[2024-04-19T10:13:29.828Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:21819 - -
[2024-04-19T10:13:30.852Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:63980 - -
[2024-04-19T10:13:31.879Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 7 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:64593 - - Node-Ports for the istio-ingressgateway: kubectl -n istio-system get svc | grep istio-ingressgateway
istio-ingressgateway NodePort 10.107.222.131 <none> 15021:31267/TCP,80:30264/TCP,443:31246/TCP 24h Testing from the LBR: curl -v -H "Host: login.authserv.me" -k https://192.168.0.180:31246/ping
* Trying 192.168.0.180:31246...
* Connected to 192.168.0.180 (192.168.0.180) port 31246 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 192.168.0.180:31246
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 192.168.0.180:31246 Whatever I tried, I can not get HTTPS to work. All examples I found on the internet most often use Kubernetes in a KaaS with external LBR. So maybe I miss something for the NodePort HAproxy-setup for on-prem-systems. Kubernetes nodes information: kubectl get nodes
NAME STATUS ROLES AGE VERSION
km1.roessner-net.de Ready control-plane 53d v1.29.3
km2.roessner-net.de Ready control-plane 53d v1.29.3
km3.roessner-net.de Ready control-plane 38d v1.29.3
kn1.roessner-net.de Ready loadbalancer 53d v1.29.3
kn2.roessner-net.de Ready loadbalancer 53d v1.29.3
kn3.roessner-net.de Ready loadbalancer 53d v1.29.3 If someone does have an idea, what is wrong, I really thank you in advance! If someone thinks, it might be a bug, I will report this question as a bug.
|
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
This is a case of https://istio.io/latest/docs/ops/common-problems/network-issues/#configuring-sni-routing-when-not-sending-sni. Note |
Beta Was this translation helpful? Give feedback.
This is a case of https://istio.io/latest/docs/ops/common-problems/network-issues/#configuring-sni-routing-when-not-sending-sni. Note
-H
on curl only sets the HTTP host header, not the TLS SNI header