[containerMode: kubernetes] K8s Job/Hook implementation does not work with basic settings

[jacksonporter]

I am trying to use the containerMode: kubernetes with the following CRD and get the following error on job execution:

CRD:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
<trimmed>
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                  - amd64
  containerMode: kubernetes
  dockerdContainerResources: {}
  image: ''
  labels:
    - containerJob
  repository: <redacted>
  resources: {}
  serviceAccountName: <redacted>
  tolerations:
    - effect: NoSchedule
      key: cicd_runners
      operator: Equal
      value: 'false'
  workVolumeClaimTemplate:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: gp3

---

Error on Initialize containers:

Error: Error: Missing file at path: /runner/_work/_temp/_runner_hook_responses/<redacted>.json
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.

Follow-up error on Complete runner:

A job completed hook has been configured by the self-hosted runner administrator
Error: Access to the path '/runner/_work/_temp/_github_workflow' is denied.

I am running in EKS with the EBS CSI driver. The job pod gets spun up, the runner fails, and the failed job pod stays running/dangling there. Not sure what I am doing wrong, would love some pointers!

[matthenry87]

I am having the exact same experience. I upgraded my helm installation of the chart, and even re-built my custom runner image from their latest image.

actions/runner#981 (comment)

When my initial runner comes up, it doesn't see like the volume is mounted to it:

  volumes:
  - ephemeral:
      volumeClaimTemplate:
        metadata:
          creationTimestamp: null
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: gp2
          volumeMode: Filesystem
    name: work

    volumeMounts:
    - mountPath: /runner/_work
      name: work

The PVC is there.. but in k9s I see Used By: <none> prior to the first workflow running. And I can cd to and list /runner/_work just fine:

runner@k8s-container-mode:/$ ls -la /runner/_work/
total 24
drwxr-xr-x. 3 runner docker  4096 Apr 15 23:37 .
drwxrwxrwx. 8 runner docker  4096 Apr 15 23:38 ..
drwx------. 2 runner docker 16384 Apr 15 23:37 lost+found

Then, I kick off my workflow, and it does seem to mount the volume for a moment, as I can briefly list it's contents:

runner@k8s-container-mode:/$ ls -la /runner/_work/
total 40
drwxr-xr-x. 7 runner docker  4096 Apr 16 00:34 .
drwxrwxrwx. 8 runner docker  4096 Apr 15 23:38 ..
drwxr-xr-x. 3 runner runner  4096 Apr 16 00:34 _PipelineMapping
drwxr-xr-x. 7 runner runner  4096 Apr 16 00:34 _actions
drwxr-xr-x. 6 runner runner  4096 Apr 16 00:34 _temp
drwx------. 2 runner docker 16384 Apr 15 23:37 lost+found
drwxr-xr-x. 3 runner runner  4096 Apr 16 00:34 <redacted>

But then immediately after:

runner@k8s-container-mode:/$ ls -la /runner/_work/
ls: cannot open directory '/runner/_work/': Permission denied

[matthenry87]

@jacksonporter Did you ever get this figured out?

[jacksonporter]

I did not :/

[jaimehrubiks]

Can you try adding fsGroup?:

  spec:
    securityContext:
      fsGroup: 1000
    nodeSelector:

(Note that I'm not sure if it will work or not because I'm a little bit confused on the differences between actions-runner-controller and gha-runner-scale-set-controller so not sure if you can add securityContect on the former)

[AlexisColes]

FYI: I ran into this same error after it had previously been working. After reverting all the changes that had been made to the set up since it had been working it turned out to be bottlerocket hosts that were the issue.

Just changing the hosts back to vanilla amazon Linux 2 it all started working again.

[barbabbott]

All is good thank you

[amille44420]

Wondering if the issue can be solved on bottlerocket ? We cannot move back to amazon linux as it's not CIS benchmark compliant.