ImageUpdateAutomation: failed to push to remote

[toonvanstrijp]

Describe the bug

Since yesterday our ImageUpdateAutomation controller stopped working. I tried looking up the image automation controller logs, the only message I see is:

{"level":"error","ts":"2024-04-10T07:54:41.594Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"bb111abb-d5a5-4b76-93a2-d58015405f71","error":"failed to push to remote: object not found"}
{"level":"info","ts":"2024-04-10T07:54:42.656Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"2ff0570d-4df5-4d45-929a-0de386172361"}
{"level":"error","ts":"2024-04-10T07:54:42.656Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"2ff0570d-4df5-4d45-929a-0de386172361","error":"failed to push to remote: object not found"}
{"level":"info","ts":"2024-04-10T07:56:08.483Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"6ad9d9a8-0be6-4a9b-a7fb-bfc89a823100"}
{"level":"error","ts":"2024-04-10T07:56:08.483Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"6ad9d9a8-0be6-4a9b-a7fb-bfc89a823100","error":"failed to push to remote: object not found"}

Steps to reproduce

I'm not sure how to reproduce this error consistently, our production environment is set up in the same way but doesn't experiences these problems.

Expected behavior

I expect the ImageAutomationController to push and update our repository

Screenshots and recordings

No response

OS / Distro

AWS Kubernetes

Flux version

v2.2.3

Flux check

► checking prerequisites
✔ Kubernetes 1.29.1-eks-b9c9ed7 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.3
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.4
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

Git over HTTPS

Container Registry provider

AWS

Additional context

We're using bitbucket, in bitbucket we created a repository access token and then used the bootstrap command as followed:

flux bootstrap git \
  --url=https://bitbucket.org/[PATH].git \
  --branch=develop \
  --path=clusters/development \
  --username=x-token-auth \
  --password="..." --token-auth=true --components-extra=image-reflector-controller,image-automation-controlle

Code of Conduct

• I agree to follow this project's Code of Conduct

[toonvanstrijp]

I've enabled debug logging now I'm getting the following logs:

{"level":"debug","ts":"2024-04-10T08:18:26.511Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba","gitrepository":{"name":"flux-system","namespace":"flux-system"}}
{"level":"debug","ts":"2024-04-10T08:18:26.516Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba","gitrepository":{"name":"flux-system","namespace":"flux-system"},"ref":{"branch":"develop"},"working":"/tmp/flux-system-flux-system1440146471"}
{"level":"debug","ts":"2024-04-10T08:18:27.129Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba","count":3,"manifests-path":"/tmp/flux-system-flux-system1440146471"}
{"level":"debug","ts":"2024-04-10T08:18:27.142Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba","working":"/tmp/flux-system-flux-system1440146471"}
{"level":"debug","ts":"2024-04-10T08:18:27.455Z","logger":"events","msg":"failed to push to remote: object not found","type":"Warning","object":{"kind":"ImageUpdateAutomation","namespace":"flux-system","name":"image-updates","uid":"3ed23245-dca2-418d-956d-f41cebcaa450","apiVersion":"image.toolkit.fluxcd.io/v1beta1","resourceVersion":"128560300"},"reason":"error"}
{"level":"info","ts":"2024-04-10T08:18:27.487Z","msg":"Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba"}
{"level":"error","ts":"2024-04-10T08:18:27.489Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"image-updates","namespace":"flux-system"},"namespace":"flux-system","name":"image-updates","reconcileID":"45ba3897-b816-43a0-b9e1-4344aec264ba","error":"failed to push to remote: object not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}