Helm operator does not update CR child serviceaccount ImagePullSecret when watched CR ImagePullSecret changed.

[lihongbj]

Bug Report

What did you do?

Helm operator does not update CR child serviceaccount ImagePullSecret when operator watched CR ImagePullSecret changed.

I defined a CRD kong for helm chart Kong, and use helm operator to deploy it, and a watches.yaml is also defined to watch kong.

First the helm operator pod is deployed, and then a new CR instance kong/gateway without ImagePullSecret is created, so child serviceaccount/gateway-kong and child pod/ gateway-kong are created accordingly. Because no ImagePullSecret is provided in kong/gateway, so no ImagePullSecret is in serviceaccount/gateway-kong and the pod is hang in state ImagePullBackOff.
Then ImagePullSecret is added for kong/gateway, but the serviceaccount/gateway-kong with owner to kong/gateway is not updated accordingly. And no new text are logged in helm operator pod log for this change.

What did you expect to see?

When ImagePullSecret is added/changed for kong/gateway, the serviceaccount/gateway-kong with owner to kong/gateway is update accordingly by helm operator and then the child pod pod/ gateway-kong is created and running without error ImagePullBackOff.

# oc get -oyaml kong/gateway
kind: Kong
metadata:
  labels:
    app.kubernetes.io/instance: kong-operator
    app.kubernetes.io/managed-by: kong-operator
    app.kubernetes.io/name: kong-operator
  name: gateway
  uid: d42c590f-b60e-4530-834f-ac9d5e097024
spec:
  hook:
    enabled: true
  imagePullSecrets:
  - entitlement-key            # <--- new added

# oc get -oyaml sa/gateway-kong
apiVersion: v1
imagePullSecrets:
- name: gateway-kong-dockercfg-fdnzk
- entitlement-key                # <--- should sync added from owner kong
kind: ServiceAccount
metadata:
  name: gateway-kong
  ownerReferences:
  - apiVersion: management.my.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Kong
    name: gateway
    uid: d42c590f-b60e-4530-834f-ac9d5e097024

What did you see instead? Under which circumstances?

When ImagePullSecret is added/changed for kong/gateway, the serviceaccount/gateway-kong with owner to kong/gateway is NOT update accordingly by helm operator and the child pod pod/ gateway-kong is hang in ImagePullBackOff.

# oc get -oyaml kong/gateway
kind: Kong
metadata:
  labels:
    app.kubernetes.io/instance: kong-operator
    app.kubernetes.io/managed-by: kong-operator
    app.kubernetes.io/name: kong-operator
  name: gateway
  uid: d42c590f-b60e-4530-834f-ac9d5e097024
spec:
  hook:
    enabled: true
  imagePullSecrets:
  - entitlement-key            # <--- new added

# oc get -oyaml sa/gateway-kong
apiVersion: v1
imagePullSecrets:                # <------ not new added/updated from owner kong
- name: gateway-kong-dockercfg-fdnzk
kind: ServiceAccount
metadata:
  name: gateway-kong
  ownerReferences:
  - apiVersion: management.my.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Kong
    name: gateway
    uid: d42c590f-b60e-4530-834f-ac9d5e097024

pod list:

NAME                                READY   STATUS             RESTARTS   AGE
gateway-kong-589cb76-6rsrg           0/2     ImagePullBackOff   0          14m
kong-operator-86bc498-dw4t8          1/1     Running            0          15m

helm operator log:

{"level":"info","ts":"2024-03-05T13:52:53Z","logger":"cmd","msg":"Version","Go Version":"go1.21.7","GOOS":"linux","GOARCH":"amd64","helm-operator":"v1.33.0-dirty","commit":"542966812906456a8d67cf7284fc6410b104e118"}
{"level":"info","ts":"2024-03-05T13:52:53Z","logger":"cmd","msg":"Environment variable OPERATOR_NAME has been deprecated, use --leader-election-id instead."}
{"level":"info","ts":"2024-03-05T13:52:53Z","logger":"cmd","msg":"Watching all namespaces.","Namespace":""}
2024/03/05 13:52:53 Warning: Dependencies are handled in Chart.yaml since apiVersion "v2". We recommend migrating dependencies to Chart.yaml.
{"level":"info","ts":"2024-03-05T13:52:53Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2024-03-05T13:52:53Z","logger":"helm.controller","msg":"Watching resource","apiVersion":"management.my.com/v1alpha1","kind":"Kong","reconcilePeriod":"1m0s"}
{"level":"info","ts":"2024-03-05T13:52:53Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2024-03-05T13:52:53Z","msg":"Starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2024-03-05T13:52:53Z","msg":"Starting EventSource","controller":"kong-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-03-05T13:52:53Z","msg":"Starting Controller","controller":"kong-controller"}
{"level":"info","ts":"2024-03-05T13:52:54Z","msg":"Starting workers","controller":"kong-controller","worker count":16}
2024/03/05 13:52:54 Warning: Dependencies are handled in Chart.yaml since apiVersion "v2". We recommend migrating dependencies to Chart.yaml.
2024/03/05 13:52:54 warning: cannot overwrite table with non table for kong.proxy.stream (map[])
I0305 13:52:55.225334       1 request.go:696] Waited for 1.037869002s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/metal3.io/v1alpha1?timeout=32s
2024/03/05 13:52:56 warning: cannot overwrite table with non table for kong.proxy.stream (map[])
W0305 13:52:59.931382       1 warnings.go:70] would violate PodSecurity "restricted:v1.24": seccompProfile (pod or containers "ingress-controller", "proxy" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0305 13:52:59.947654       1 warnings.go:70] would violate PodSecurity "restricted:v1.24": seccompProfile (pod or container "kubectl" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0305 13:53:00.062520       1 warnings.go:70] would violate PodSecurity "restricted:v1.24": seccompProfile (pod or container "kubectl" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Environment

Operator type:

Kubernetes cluster type:

$ operator-sdk version

$ go version (if language is Go)
go: 1.21.7.

$ kubectl version

# oc $v
Client Version: 4.9.40
Server Version: 4.12.47
Kubernetes Version: v1.25.16+5c97f5b

Possible Solution

Additional context

[acornett21]

@lihongbj Is this related to issue #6692 when using 1.34.0? Or is this a separate issue you see in 1.33.0?

[lihongbj]

@acornett21 , thanks for your quick reply. This is a separate issue found in 1.33.0 and before version.

[jberkhahn]

this could be related to https://kubernetes.io/docs/concepts/security/pod-security-admission/
are the pods in question actually being created?