GitHub Runner start delays

[sean-stage]

One of my builds is using a standard-8 GitHub runner (8 core, 32GB memory as far as I understand), but today I am seeing long delays before one is started in te ubicloud GitHub Runners dashboard.

My jobs in GitHub actions say: "Waiting for a runner to pick up this job..." and it has been 4-5 minutes before one is started on a few occassions today already. Yesterday these jobs were starting quickly.

There is currently no reported downtime or issues on GitHub's APIs, so I'm led to believe this might be an issue on Ubicloud's side? I am using Hetzner as the provider.

Any ideas, or has anyone else seen this?

[enescakir]

Hello @sean-stage,
Thanks for trying out Ubicloud!

When a new workflow job is triggered, GitHub sends us a webhook event. We keep a pool of runners ready to serve customer requests quickly. However, depending on the load, the runner pool may become depleted, requiring the provision of a new one upon request. We anticipate jobs to start within 15-20 seconds if a runner is selected from the pool; otherwise, it can take 50-60 seconds.

If GitHub sends the webhook on time, the process should not take 4-5 minutes.

Sometimes, GitHub may fail to send webhook events. We have an open support ticket with GitHub, and their engineering team is aware of and working on the issue.

If you share your workflow run details with us at support@ubicloud.com, we can investigate further. You can find the run ID in the URL after opening the details of a specific run.

[sean-stage]

Hi @enescakir ,

I suspect then it may have been a case of GitHub not sending a webhook payload, or it was severely delayed. If I notice it again I'll grab the run ID and send it along to that email address. Thanks again.

[dphuang2]

I'm finding that some of build never even start—is this known by the Ubicloud team? @sean-stage

[sean-stage]

Hey @dphuang2 ,

We're still seeing a few cases of this. The message tends to be waiting for an executor and the job is never started.

@enescakir , do you have any updates on progress of this issue with GitHub? We're seeing great success with using Ubicloud for our github actions runs, but these few isolated cases of jobs not picking up an executor are holding us back a bit.

We would love to change all of our repositories over to use ubicloud but want to wait for this issue to be ironed out first.

Thanks

[enescakir]

Hi @dphuang2 and @sean-stage,
GitHub experienced 6 incidents that affected Actions and Webhooks in the last 7 days: https://www.githubstatus.com/history. These incidents caused some webhook events to either not be sent or to be sent with a delay, which left our backend unaware of any jobs waiting for a runner.
It seems like it will take some time to fix the missing webhook events issue on the GitHub side.

In response to this, we're currently revising our listening logic to use polling instead of solely relying on webhooks. We anticipate that these changes will be deployed within this week.

Additionally, we're experiencing compatibility issues with the max-parallel configuration. If you're using it, that could be the problem. Our polling mechanism should also help resolve this issue.

[sean-stage]

Thanks for the info. I'll check whether the jobs are using max-parallel or not too. It's great that you'll be using a polling mechanism to help offset any issues on github side via webhooks - thanks!

[jakubadamw]

We've been observing this too. Is there anything the Ubicloud team can do to improve the throughput? Increasing the pool size: both the static size for the whole system, but also dynamically based on, say, time of day/week, to match anticipated levels of traffic at a particular point in time?

[byucesoy]

We've noticed an increase in failure rates for API calls to GitHub, along with some delays in webhook deliveries. After we register the runner to GitHub, seems GitHub has issues with job assignments.

Although there's no active incident reported, unofficial data from https://downdetector.com/status/github/ suggests others are experiencing some problems.

[jakubadamw]

Thank you!