Using concurrency in workflows shouldn't cancel previously pending workflows

[TSRBerry]

Better discussion:

https://github.com/orgs/community/discussions/12835

Select Topic Area

Product Feedback, Question

Body

We recently introduced concurrency for our release workflow at Ryujinx , so we could merge PRs without waiting for the previous release job to finish.

We expected all new workflows to be pending until the previous job is done, but that wasn't the case.

As you can see here, the first job gets canceled after a new pending job gets added, which we wanted to avoid.

We never defined a "priority" or anything to make it cancel this workflow, but it still did.

Reading the documentation again we found this sentence:

Any previously pending job or workflow in the concurrency group will be canceled.

Source: Github workflow concurrency

Now this job was already in progress, but was canceled anyway, so we aren't sure if this note applies to our situation.
cancel-in-progress: true is obviously not defined in our workflow file, so this situation is a little confusing.

If this is intended is there a way for us to not cancel previously in-progress workflows? We would prefer to run these workflows in the order they were queued in, so our changelog matches with the actual releases we publish.

---

(cc: @marysaka)

[TSRBerry]

(Seems like this discussion is missing a few labels, but I'm not sure what to do to add them.)

[mgasner]

See also: actions/runner#1523, actions/runner#2227 (folks from GH saying not their problem), https://github.com/orgs/community/discussions/5435 (folks from GH saying they won't implement, a year ago), https://github.com/orgs/community/discussions/33142, https://github.com/orgs/community/discussions/12835, https://github.com/orgs/community/discussions/33948, https://github.com/orgs/community/discussions/32376. Related: https://github.com/orgs/community/discussions/31040, https://github.com/orgs/community/discussions/31544. Plus lots of others, plus others discussing the need for configurable concurrency limits (e.g. limit to 2 concurrent jobs).

[vitalykarasik]

+1 - 'concurrency' implementation lacks a few important features.
Unfortunately, I don't know how we can raise an issue about that.
Vitaly
#46014
https://www.linkedin.com/posts/vkarasik_cicd-jenkins-githubactions-activity-7026518468577366017-X0tt

[ShutdownRepo]

Agreed, this is a very important lack. I find it very surprising that GitHub's pipeline lacks the ability to manage queues

[austinpray-mixpanel]

My use case: I've got expensive e2e tests that I only want to run one master copy of at a time to leave headroom for all the PRs that might need the resources. I want to run on every master commit so we can easily bisect when they break. We have multiple commits per hour during working hours.

[yardensade]

How come pending jobs with concurrency are being cancelled? most of us will want all of them to await running and not be cancelled by new triggers.

This feature is very lacking as is and missing a very strong use case for customers, can we get some insight on this?

[SamBernaerdtOntoforce]

Same here. We need the option
cancel-pending: false

Our use case: We have e2e tests that run against the same server and cannot run in parallel. Any e2e workflow should only start when no other e2e workflows are running. Concurrency would be the solution if there is an option to not cancel pending jobs.

[MerzoukeMansouri]

+1 need a waitting list

[dvasdekis]

Mature CICD solutions (Azure Devops, CircleCI) have this solution, see here for Devops implementation: https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-190-update

+1 to having cancel-pending: false. The expected behaviour is to queue.

[jlongo-encora]

Any updates on this one?

[aquierick]

+1 to having cancel-pending: false

[iamironz]

• for it.
  It would be way more convenient to support workflow runs one-by-one based on the queue instead of canceling, as it might be perhaps important for validating develop after the merge and sending Slack notifications to the appropriate contributor in the case of the failed build. We have exactly this case.

[groeney]

Does anyone have a clever workaround to not having support for cancel-pending: false?

[samirtahir91]

I have worked around by checking the status of last 50 jobs/runs for workflow ID(s), for all those without the status "completed", check the created_at timestamp, and essentially sleep and loop until all of the non-completed runs with an older timestamp are completed. The caveat here is your queued jobs are sleeping and using compute time.

I'm not using the concurrency feature at all now

In my case I have a workflow that does some changes, creates a PR and merges to main. The reason I have to queue is because Github API rate limiting, there is the option for merge queues however even creating a PR is limited by Githubs secondary api rate limit for a single user/account, basically you cannot do any parallel api calls to Github and their documentation strongly advises against it. Which is why i find it bizarre there is no native queuing option/serial run option.

It works ok for small time consuming jobs, but for longer jobs you may not want them to be sleeping for a long time

[vmred]

+1 to having cancel-pending: false or any other naming flag for this

[danlehmann]

+1 for "cancel-pending: false"

This feature would be incredibly useful for us as our automated test infra pretty often skips over tests when using concurrency. While we can come up with a way to enforce concurrency inside the tests themselves, that is pretty involved (and has the risk of running into timeouts). Github's feature is so close to being really useful

[wpride]

For anyone looking for a workaround, check out https://github.com/marketplace/actions/action-turnstyle

[Husnain08]

this does not work as intended. can you provide some examples @wpride ?

[farshad-t]

one workaround is to limit the workflow to only run on a single runner. This way, runner will limit the number of concurrent runs.

[farshad-t]

It probably only works on a single self-hosted runner.

[Husnain08]

I am curious as to why it could work? can you explain it @farshad-t ?

[farshad-t]

If there is only one runner, there will be no need for "concurrency". Workflows will naturally run in order and one at a time.

[ephraim271]

@farshad-t I have only one self-hosted runner, but multiple workflows are still attempting to run at the same time

[vitalykarasik]

"attemping" - yes, i.e. we have a queue. And only one job is running, so it's a working workaround.

[GuillaumeDerivery]

+1 "cancel-pending: false"

[Kruziikrel13]

+1 "cancel-pending:false"

[Husnain08]

+1 "cancel-pending:false"

[Shereef]

I proposed a feature request here
https://github.com/orgs/community/discussions/12835
Please upvote it

[TSRBerry]

Oh I had no idea this existed! My post could have been just a comment to yours.
Thanks for sharing!

[Shereef]

@TSRBerry Are you able to put this link at the top ? I feel like we need to attract more people there

[TSRBerry]

Good idea, I'll do that!

[bcarranza]

Ey Github team, this is an important Lack , we really need this due requirements to work one workflow at once but without cancel , just enqueue.
do you have some information about feature.
+1 "cancel-pending:false"

?

[philn-delphia]

The behavior of concurrency groups is very surprising. This is a much needed feature.

[9999years]

[aromeronavia]

So amazed concurrency doesn't work out of the box. I just got rid of the merge queue in my team because I thought it was not necessary, and all of a sudden, it seems we will need to go back to it because this feature is not complete...

[adamatsolita]

+1

[matheuscscp]

+1

[jmax01]

+1

[cmyllis]

+1. Would be funny if they implement it after 2 years 🥲

[tt-rkim]

👍 +1

[amy-brainlabs]

Hacky workaround we implemented, removing concurrency and adding these steps makes jobs sleep if any are already running and checks every 30 seconds:

jobs:
  build:
    runs-on: ubuntu-20.04
    steps:
      - name: Wait for previous runs to complete
        id: wait_for_previous_runs
        run: |
          IN_PROGRESS_RUNS=$(curl -s -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
                              -H "Accept: application/vnd.github.v3+json" \
                              "https://api.github.com/repos/${{ github.repository }}/actions/runs?event=push&status=in_progress" \
                              | jq -r '.workflow_runs[].id | select(. != ${{ github.run_id }})')
          if [ -z "$IN_PROGRESS_RUNS" ]; then
            echo "No previous runs in progress. Proceeding with deployment."
          else
            echo "Waiting for previous runs to complete..."
            echo "::set-output name=in_progress_runs::$IN_PROGRESS_RUNS"
          fi

      - name: Store in-progress runs
        if: steps.wait_for_previous_runs.outputs.in_progress_runs != ''
        id: store_in_progress_runs
        run: echo "::set-output name=in_progress_runs::${{ steps.wait_for_previous_runs.outputs.in_progress_runs }}"

      - name: Check if stored in-progress runs are completed
        if: steps.wait_for_previous_runs.outputs.in_progress_runs != ''
        id: check_in_progress_runs
        run: |
          while true; do
            STORED_RUNS="${{ steps.store_in_progress_runs.outputs.in_progress_runs }}"
            COMPLETED_RUNS=$(curl -s -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
                              -H "Accept: application/vnd.github.v3+json" \
                              "https://api.github.com/repos/${{ github.repository }}/actions/runs?event=push&status=completed" \
                              | jq -r '.workflow_runs[].id | select(. != ${{ github.run_id }})')
            if [ "$(comm -23 <(echo "$STORED_RUNS" | tr ' ' '\n' | sort) <(echo "$COMPLETED_RUNS" | tr ' ' '\n' | sort) | wc -l)" -eq 0 ]; then
              echo "All stored in-progress runs completed. Proceeding with deployment."
              break
            else
              echo "Waiting for stored in-progress runs to complete..."
              sleep 30
            fi
          done

          .... rest of steps

[Felixoid]

Sorry to say @amy-brainlabs, but it's a very limited approach.

Imagine a fleet of 600+ self-hosted runners sleeping for veeeeeerrrrrrrrrrry long time. It's just burning money for nothing

[amy-brainlabs]

yeah its works for us we have at most 4 or 5 short running processes, would be very costly for 600 and not appropriate!

[krukonshedul]

mine crypto while its waiting - just as wasteful, but spreads the profits 😅