gha-runner-scale-set-0.9.x: Volume /var/run and /var/run/secrets/kubernetes.io/serviceaccount Conflict

[JohnYoungers]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

When using the gha-runner-scale-set to specify a custom template.spec.containers value containing the defaults from the documentation:

...
serviceAccountName: my-sa,
containers:
- name: runner
  image: ghcr.io/actions/actions-runner:latest
  command: ["/home/runner/run.sh"]
  env:
    - name: DOCKER_HOST
      value: unix:///var/run/docker.sock
  volumeMounts:
    - name: work
      mountPath: /home/runner/_work
    - name: dind-sock
      mountPath: /var/run
...

When this runner type is created, the pod fails to start with the following message:

Error: failed to create containerd task: 
failed to create shim task: OCI runtime create failed: 
runc create failed: unable to start container process: 
error during container init: 
error mounting "/var/lib/kubelet/pods/{id}/volumes/kubernetes.io~projected/kube-api-access-{id}" 
to rootfs at "/var/run/secrets/kubernetes.io/serviceaccount": 
mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/runner/rootfs/run/secrets: read-only file system: unknown

Describe the bug

It looks like the /var/run volume is conflicting with the attempt to mount the serviceaccount token at the /var/run/secrets/... subpath

Describe the expected behavior

The pod to start without error: does the secrets mount need to be specified manually?

Additional Context

The cluster is version 1.28, running in AWS (EKS)

Controller Logs

N/A

Runner Pod Logs

N/A

[danielhabakkuk]

Facing the same issue after upgrading the gha-runner-scale-set to 0.9.0. Did someone find a fix for this?

[AmitBenAmi]

I am having the same issue with version 0.8.3 chart on EKS version 1.27

[AmitBenAmi]

@JohnYoungers I found out the root cause for this issue in my case.
For the runner container, I mounted the /var/run volumeMount with readyOnly: true, which eventually caused the mounting of the ServiceAccount token (on /var/run/secrets) to fail - it is a read-only file-system.

Changing /var/run mount to /run/docker path (or any other) specifically for mounting the docker.sock fixed the issue - just note that you need to change this path on both containers.

I did not dig into the option of only mounting the specific docker.sock file, but eventually that did the trick