"Error: The operation was canceled" when no job was canceled

[sgandon]

Hello
We are using using the ARController in our EKS cluster and we are having some issues with our runners deployed using RunnerDeployment. We are using Github cloud (and not the GH server).
Sometimes, and to be honest quite often we are getting our job killed somehow.
The last message in the job logs is the following

Error: The operation was canceled.

the messages in the job summary is a bit different

test / helm_testThe operation was canceled.

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
The operation was canceled.

And looking at the pods logs it seems that the pod is recieving a SIGTERM message but we don't know why.

Runner update in progress, do not shutdown runner.
Downloading 2.302.1 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should be back online within 10 seconds.
Runner update process finished.
Runner listener exit because of updating, re-launch runner after successful update
Update finished successfully.
Restarting runner...

√ Connected to GitHub

Current runner version: '2.302.1'
2023-03-13 10:28:39Z: Listening for Jobs
2023-03-13 10:28:43Z: Running job: build_images
time="2023-03-13T10:28:48.432312191Z" level=info msg="parsed scheme: \"\"" module=grpc
time="2023-03-13T10:28:48.432383100Z" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
time="2023-03-13T10:28:48.432404757Z" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2023-03-13T10:28:48.432415758Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2023-03-13 10:29:13.979  NOTICE --- Executing actions-runner-controller's SIGTERM handler.
2023-03-13 10:29:13.980  NOTICE --- Note that if this takes more time than terminationGracePeriodSeconds, the runner will be forcefully terminated by Kubernetes, which may result in the in-progress workflow job, if any, to fail.
2023-03-13 10:29:13.982  NOTICE --- Ensuring dockerd is still running.
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
/runner /
2023-03-13 10:29:14.2  NOTICE --- Waiting for the runner to register first.
2023-03-13 10:29:14.30  NOTICE --- Observed that the runner has been registered.

# Runner removal

Failed: Removing runner from the server
Runner "github-actions-runner-large-scwn9-fzjff" is still running a job"
2023-03-13 10:29:16.130  NOTICE --- Waiting for RUNNER_GRACEFUL_STOP_TIMEOUT=15 seconds until the runner agent to stop by itself.
/
2023-03-13 10:29:31.185  NOTICE --- Sending SIGTERM to the actions runner agent (327).
2023-03-13 10:29:31.187  NOTICE --- SIGTERM sent. If the runner is still running a job, you'll probably see "Error: The operation was canceled." in its log.
2023-03-13 10:29:31.192  NOTICE --- Waiting for the actions runner agent to stop.

Could anyone help us fix this issue please ?

[lucetre]

Are there any updates?

[maeghan-porter]

I am having this exact same issue, also running RunnerDeployments in EKS. I was just going to post about this as well. I'm using summerwind/actions-runner-dind:v2.303.0-ubuntu-22.04 images and SPOT 8vCPU core 32GB memory instances. The instances don't show any significant spike in CPU or memory usage around that time.
My only working theory was that it was because they are SPOT instances and perhaps the instances were getting cycled, however the instance running the container I just had get sigtermed is still around and has been for 5 days, so I'm really at a loss now :(

@sgandon this post was 3 months ago, did you figure out a resolution?

[MPV]

Related to this upstream issue?

• The linux runner does not gracefully shutdown on SIGINT runner#2582

[adefemi171]

I am also stuck on this as well, it works and then it doesn't on other runs. I have increased the memory and cpu hoping things will change but it just does not work at the moment, any updates?

[ViniciusBastosTR]

The Github team should pay more attention to this problem. There are many complaints about this and they are not being resolved.

[MPV]

FYI: If it helps, we've seen this issue when nodes run out of disk (on both GitHub-hosted and self-hosted runners). The user experience is that the job is "cancelled" and no helpful error about the disk being the cause.

[ggprod]

We are seeing this issue as well, could it occur if the associated pod exceeds the cpu or memory limit?

[hasankfz]

we are also experiencing this same issue

[claudiallium]

i am also experiencing this issue

[jonwest]

If you are running on spot instances you will also get this when the nodes are preempted.

[cb-krishnapatel]

I've faced a similar issue for a long time. When I deep-dived into this, I got to know that there is a runner registration API called from pod which failed, the failure was due to token expiry. I'm using a GitHub app and after a few hours, this token gets refreshed. is this the reason why?