Job random stuck with "Job is waiting for a runner from XXX to come online"

[cheskayang]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.1

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

this error happens randomly

Describe the bug

Job randomly gets stuck with the msg "Job is waiting for a runner from XXX to come online"
cancel the job and rerun will fix it.

Observations:
1: Compared with jobs without issue, for this job gets stuck, there is no job started msgs get received on the listener pod (i.e. it only gets job available, job assigned, and then stuck util cancelled manually..)
2. for the job gets stuck, the EphemeralRunnerSet gets patched to 1 replica and then immediately gets patched to null, see logs below

2024-05-06T13:44:52Z INFO listener-app.worker.kubernetesworker Created merge patch json for EphemeralRunnerSet update {"json": "{\"spec\":{\"patchID\":1,\"replicas\":1}}"}
2024-05-06T13:44:52Z INFO listener-app.worker.kubernetesworker Scaling ephemeral runner set {"assigned job": 1, "decision": 1, "min": 0, "max": 20, "currentRunnerCount": 0, "jobsCompleted": 0}
2024-05-06T13:44:52Z INFO listener-app.worker.kubernetesworker Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "basic-runner-622tx", "replicas": 1}
2024-05-06T13:44:52Z INFO listener-app.listener Getting next message {"lastMessageID": 248}
2024-05-06T13:45:42Z INFO listener-app.worker.kubernetesworker Created merge patch json for EphemeralRunnerSet update {"json": "{\"spec\":{\"patchID\":0,\"replicas\":null}}"}
2024-05-06T13:45:42Z INFO listener-app.worker.kubernetesworker Scaling ephemeral runner set {"assigned job": 0, "decision": 0, "min": 0, "max": 20, "currentRunnerCount": 1, "jobsCompleted": 0}
2024-05-06T13:45:42Z INFO listener-app.worker.kubernetesworker Ephemeral runner set scaled. {"namespace": "arc-runners", "name": "basic-runner-622tx", "replicas": 0}
2024-05-06T13:45:42Z INFO listener-app.listener Getting next message {"lastMessageID": 248}

3. the following log seems to occur on the controller side when this issue happens

Getting runner jit config failed with conflict error, trying to get the runner by name

Describe the expected behavior

job should not stuck

Additional Context

- cancel the job and rerun will fix the issue
- having both issues on 0.9.0, 0.9.1
- running on gke

Controller Logs

see description

Runner Pod Logs

N/A

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[cheskayang]

same issue as mentioned in #3499 and #3420
providing more details on the observed behavior

[nikola-jokic]

Closing this one as a duplicate. Thank you for linking it!