Kong breaks cert-manager certificate creation #12267

jakoberpf · 2023-12-28T17:59:54Z

Is there an existing issue for this?

I have searched the existing issues

Kong version (`$ kong version`)

3.4

Current Behavior

When deploying kong/ingress alongside with cert-manager certificate creation failed due to cert-manager/cert-manager#5918

kubernetes ingress manifest

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: vault
  namespace: vault
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  ingressClassName: kong
  tls:
  - hosts:
      - vault.<redacted>
    secretName: vault-cert
  rules:
  - host: vault.<redacted>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: vault
            port:
              number: 8200

kong-controller logs

2023-12-28T17:51:33Z	error	Failed to fetch the secret (vault/gateway-cert)	{"name": "vault", "namespace": "vault", "GVK": "networking.k8s.io/v1, Kind=Ingress", "error": "resource processing failed"}

cert-manager logs

I1228 17:38:11.480947       1 trigger_controller.go:215] "cert-manager/certificates-trigger: Certificate must be re-issued" key="vault/vault-cert" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I1228 17:38:11.481073       1 conditions.go:203] Setting lastTransitionTime for Certificate "vault-cert" condition "Issuing" to 2023-12-28 17:38:11.481064989 +0000 UTC m=+5203.127755535
I1228 17:38:11.481187       1 conditions.go:203] Setting lastTransitionTime for Certificate "vault-cert" condition "Ready" to 2023-12-28 17:38:11.48118162 +0000 UTC m=+5203.127872166
I1228 17:38:11.508725       1 controller.go:162] "cert-manager/certificates-readiness: re-queuing item due to optimistic locking on resource" key="vault/vault-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"vault-cert\": the object has been modified; please apply your changes to the latest version and try again"
I1228 17:38:11.508842       1 conditions.go:203] Setting lastTransitionTime for Certificate "vault-cert" condition "Ready" to 2023-12-28 17:38:11.508835094 +0000 UTC m=+5203.155525641
I1228 17:38:21.707165       1 controller.go:162] "cert-manager/certificates-key-manager: re-queuing item due to optimistic locking on resource" key="vault/vault-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"vault-cert\": the object has been modified; please apply your changes to the latest version and try again"
I1228 17:38:21.738016       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "vault-cert-1" condition "Approved" to 2023-12-28 17:38:21.737989616 +0000 UTC m=+5213.384680163
I1228 17:38:21.775055       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "vault-cert-1" condition "Ready" to 2023-12-28 17:38:21.775031464 +0000 UTC m=+5213.421722008
I1228 17:40:25.283336       1 acme.go:233] "cert-manager/certificaterequests-issuer-acme/sign: certificate issued" resource_name="vault-cert-1" resource_namespace="vault" resource_kind="CertificateRequest" resource_version="v1" related_resource_name="vault-cert-1-3054829319" related_resource_namespace="vault" related_resource_kind="Order" related_resource_version="v1"
I1228 17:40:25.283789       1 conditions.go:252] Found status change for CertificateRequest "vault-cert-1" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2023-12-28 17:40:25.283779662 +0000 UTC m=+5336.930470200
E1228 17:40:35.307438       1 controller.go:167] "cert-manager/certificates-issuing: re-queuing item due to error processing" err="failed to apply secret vault/vault-cert: Patch \"https://172.20.0.1:443/api/v1/namespaces/vault/secrets/vault-cert?fieldManager=cert-manager-certificates-issuing&force=true\": context deadline exceeded" key="vault/vault-cert"
E1228 17:40:46.307796       1 controller.go:167] "cert-manager/certificates-issuing: re-queuing item due to error processing" err="failed to apply secret vault/vault-cert: Patch \"https://172.20.0.1:443/api/v1/namespaces/vault/secrets/vault-cert?fieldManager=cert-manager-certificates-issuing&force=true\": context deadline exceeded" key="vault/vault-cert"
I1228 17:40:50.644476       1 controller.go:162] "cert-manager/challenges: re-queuing item due to optimistic locking on resource" key="vault/vault-cert-1-3054829319-3717849091" error="when updating the status: Operation cannot be fulfilled on challenges.acme.cert-manager.io \"vault-cert-1-3054829319-3717849091\": the object has been modified; please apply your changes to the latest version and try again"
E1228 17:40:51.081930       1 controller.go:208] "cert-manager/challenges: challenge in work queue no longer exists" err="challenge.acme.cert-manager.io \"vault-cert-1-3054829319-3717849091\" not found"
E1228 17:40:55.645200       1 controller.go:208] "cert-manager/challenges: challenge in work queue no longer exists" err="challenge.acme.cert-manager.io \"vault-cert-1-3054829319-3717849091\" not found"
E1228 17:40:58.309581       1 controller.go:167] "cert-manager/certificates-issuing: re-queuing item due to error processing" err="failed to apply secret vault/vault-cert: Patch \"https://172.20.0.1:443/api/v1/namespaces/vault/secrets/vault-cert?fieldManager=cert-manager-certificates-issuing&force=true\": context deadline exceeded" key="vault/vault-cert"

Expected Behavior

Ingress ressources creates certificate request which creates certificate.

Steps To Reproduce

Create EKS cluster
Deploy cert-manager
Create ClusterIssuer with dns01 challenge and route53 IRSA
Deploy kong/ingress
Create ingress as shown in bug report
Observe cert-manager will create CertificateRequest (becomes ready) and Certificate (will stay in unready status due to "context deadline exceeded" while "applying Secret data" cert-manager/cert-manager#5918)
Uninstall kong/ingress
Observe that Certificate becomes ready

Anything else?

No response

The text was updated successfully, but these errors were encountered:

randmonkey · 2024-01-02T07:30:46Z

@jakoberpf According to your original issue in cert-manager repo (cert-manager/cert-manager#5918), The secret may fail on the webhook validation. So I would like to know your KIC version for further investigation.
Would you please tell me:

What was version of KIC?
How did you install KIC,Kong and certManager?

jakoberpf · 2024-01-03T08:27:17Z

@randmonkey Thanks for the feedback. So I installed the kong/ingress chart with the version 0.10.1, which results in the controller image kong/kubernetes-ingress-controller:3.0. cert-manager chart version is v1.13.3 which is also the service version.

Both charts are install somewhat in the same time as they are part of the same terraform module in my case. I did not install the kong/kong chart. My understanding is that the kong/ingress aka. KIC is sufficient to get started?

Hope this is helpful information.

randmonkey · 2024-01-03T15:58:34Z

Yes, I think installing kong/ingress charts includes installing KIC (with admission webhooks) and Kong gateway. Are you using all default values?

jakoberpf · 2024-01-03T20:04:19Z

Mostly default yes, just some service configuration...

gateway:
  proxy:
    type: NodePort
    http:
      enabled: true
      servicePort: 80
      containerPort: 8000
      nodePort: 31080

    tls:
      enabled: true
      servicePort: 443
      containerPort: 8443
      nodePort: 31443

vibhanshu-thakur-dev · 2024-01-15T19:57:22Z

@jakoberpf I am facing the same issue with Kong 3.5 both in DBless and hybrid mode.

Uninstallation of Kong ingress makes all certificates available instantly. I am just passing the below values to kong helm chart:
proxy: annotations: "service.beta.kubernetes.io/aws-load-balancer-type": "nlb" "service.beta.kubernetes.io/aws-load-balancer-internal": "false" "external-dns.alpha.kubernetes.io/hostname": "domain name" "external-dns.alpha.kubernetes.io/ttl": "60" replicaCount: "2"

I have an EKS cluster with Kong 3.5 and cert manager 1.13.2

jakoberpf · 2024-01-19T15:14:09Z

@randmonkey can we support the investigation somehow?

jakoberpf · 2024-02-21T12:44:42Z

@randmonkey sorry for bothering, but can we support in debugging or fixing this? We would love to use Kong as our solution, but this is kinda breaking any automation process.

kaelanspatel · 2024-02-27T22:10:57Z

Also ran into this problem. On my end, this seemed to be an issue with the configuration of the KIC webhook as per: Kong/kubernetes-ingress-controller#2431.

Either fixing the issue with the ValidatingWebhook (i.e. configuration, vpc issues) or removing the secrets rule as per that issue may resolve this for you as it did for me @jakoberpf.

jakoberpf · 2024-02-27T22:18:53Z

@kaelanspatel Thats nice, thanks. Could be a intermediary solution for us.

chronolaw · 2024-03-05T05:20:40Z

@randmonkey , do you think that we have gotten a solution for this issue?

cascadia-sati · 2024-03-26T10:26:02Z

Also running into this, which is blocking our integration of Kong

chrizkim · 2024-05-16T16:28:06Z

If I'm understanding it correctly, looks like Kong/charts#1061 will address the issue when it's released, though the new flag will need to be set explicitly to true (defaults to the existing behavior)

randmonkey added the pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... label Jan 2, 2024

randmonkey removed the pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... label Jan 3, 2024

chronolaw added the area/kubernetes Issues where Kong is running on top of Kubernetes label Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kong breaks cert-manager certificate creation #12267

Kong breaks cert-manager certificate creation #12267

jakoberpf commented Dec 28, 2023 •

edited

randmonkey commented Jan 2, 2024

jakoberpf commented Jan 3, 2024 •

edited

randmonkey commented Jan 3, 2024

jakoberpf commented Jan 3, 2024

vibhanshu-thakur-dev commented Jan 15, 2024 •

edited

jakoberpf commented Jan 19, 2024

jakoberpf commented Feb 21, 2024 •

edited

kaelanspatel commented Feb 27, 2024 •

edited

jakoberpf commented Feb 27, 2024

chronolaw commented Mar 5, 2024

cascadia-sati commented Mar 26, 2024

chrizkim commented May 16, 2024

Kong breaks cert-manager certificate creation #12267

Kong breaks cert-manager certificate creation #12267

Comments

jakoberpf commented Dec 28, 2023 • edited

Is there an existing issue for this?

Kong version ($ kong version)

Current Behavior

kubernetes ingress manifest

kong-controller logs

cert-manager logs

Expected Behavior

Steps To Reproduce

Anything else?

randmonkey commented Jan 2, 2024

jakoberpf commented Jan 3, 2024 • edited

randmonkey commented Jan 3, 2024

jakoberpf commented Jan 3, 2024

vibhanshu-thakur-dev commented Jan 15, 2024 • edited

jakoberpf commented Jan 19, 2024

jakoberpf commented Feb 21, 2024 • edited

kaelanspatel commented Feb 27, 2024 • edited

jakoberpf commented Feb 27, 2024

chronolaw commented Mar 5, 2024

cascadia-sati commented Mar 26, 2024

chrizkim commented May 16, 2024

jakoberpf commented Dec 28, 2023 •

edited

Kong version (`$ kong version`)

jakoberpf commented Jan 3, 2024 •

edited

vibhanshu-thakur-dev commented Jan 15, 2024 •

edited

jakoberpf commented Feb 21, 2024 •

edited

kaelanspatel commented Feb 27, 2024 •

edited