kustomization: hit 27s timeout running '/usr/bin/git submodule update --init --recursive'

[ar-qun]

Describe the bug

Hello everyone,

I am a little bit baffled by what exactly happened but around 3 hours a go I started receiving an error

Warning BuildFailed 6m12s (x146 over 3h33m) kustomize-controller (combined from similar events): kustomize build failed: accumulating resources: accumulation err='accumulating resources from '../../base/infrastructure': read /tmp/kustomization-1762639108/kubernetes/base/infrastructure: is a directory': recursed accumulation of path '/tmp/kustomization-1762639108/kubernetes/base/infrastructure': accumulating resources: accumulation err='accumulating resources from 'reloader/': read /tmp/kustomization-1762639108/kubernetes/base/infrastructure/reloader: is a directory': recursed accumulation of path '/tmp/kustomization-1762639108/kubernetes/base/infrastructure/reloader': accumulating resources: accumulation err='accumulating resources from 'https://github.com/stakater/Reloader/deployments/kubernetes': URL is a git repository': hit 27s timeout running '/usr/bin/git submodule update --init --recursive'

On this customization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: reloader

resources:
  - namespace.yaml
  - https://github.com/stakater/Reloader/deployments/kubernetes

patches:
- target:
    group: apps
    version: v1
    kind: Deployment
    name: reloader-reloader
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/args
      value:
      - --auto-reload-all

Strangely enough if I run flux diff kustomization infrastructure --path kubernetes/overlay/infrastructure it works:

✓  Kustomization diffing...
► Deployment/reloader/reloader-reloader drifted

metadata.generation
  ± value change
    - 12
    + 13

spec.template.spec.containers.reloader-reloader.image
  ± value change
    - ghcr.io/stakater/reloader:v1.0.83
    + ghcr.io/stakater/reloader:v1.0.85

⚠️ identified at least one change, exiting with non-zero exit code

Thank you kindly in advance.

Steps to reproduce

Really didn't do anything here...

Expected behavior

Should just work...

Screenshots and recordings

No response

OS / Distro

WSL2

Flux version

flux: v2.2.3

Flux check

► checking prerequisites
✔ Kubernetes 1.27.10-gke.1055000 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.3
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.37.4
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.37.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.31.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.2.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.2.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.2.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

GitHub

Container Registry provider

N/A

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

We don't recommend using kustomize remote bases as it bypass Flux source-controller and its Git cache, in fact we advise users to disable them https://fluxcd.io/flux/faq/#should-i-be-using-kustomize-remote-bases

To deploy reloader you could use a HelmRelease or create a Flux GitRepository/Kustomization for it.

[ar-qun]

I will use Flux GitRepository/Kustomization then.

There is a bit hesitation to use kustomize.toolkit.fluxcd.io/v1 too often because I suppose it will break the chain of kustomize.config.k8s.io/v1beta1 files. For example if I have set an image update in kustomization.yaml like so

images:
- name: my-service
  newTag: "1234" # {"$imagepolicy": "dev:my-service:tag"}

Then I suppose to not apply to any kustomize.toolkit.fluxcd.io/v1 resources down the line. Does that make sense?

Not that I need that for the reloader.