Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The best way to abort the jobs running in the current process and release their locks before exiting #1995

Open
dko-slapdash opened this issue Mar 9, 2021 · 4 comments

Comments

@dko-slapdash
Copy link

Hi. I'm trying to implement a "graceful stop" sequence for a worker process which happens when e.g. a machine is stopping. The idea is to call queue.pause(true), then wait for some reasonable amount of time (say, 10 seconds) letting most of the short-lived jobs to finish on their own. Then, there will always be some remaining jobs (long-lived) for which 10 seconds is not enough; for them, we can do nothing, so I just terminate the process.

The problem is with locks of these remaining jobs. Theoretically, I know for 100% that these jobs will stop running within a second (since the process dies), so I want the locks to be released immediately; this way, the jobs' keys can be rescheduled ASAP and not when the locks expire (due to various reasons, the locks timeouts are set to some high value, like 60 seconds, to eliminate the chances of expiring them during some temporary problems with CPU load or network).

Is there a way to tell Bull to force-release locks for all the jobs which are still running in the current (!) process?

@manast
Copy link
Member

manast commented Mar 17, 2021

Although for BullMQ most of what it says is relevant to this issue: https://docs.bullmq.io/guide/workers/graceful-shutdown

@blankenshipz
Copy link

@manast I came here to ask a similar question; I have a setup where the state of my tasks is being managed by a separate application from my workers (listening to the event stream) and I have several of my worker application running in parallel - if one of my workers is stopped or crashes during the execution of a job does it automatically get reqeueued? I'm running Bull 3.20.1

@manast
Copy link
Member

manast commented Apr 8, 2021

@stale
Copy link

stale bot commented Jul 12, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jul 12, 2021
@manast manast removed the wontfix label Jul 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants