boss.fail(job.id, err) and inconsistent job retries

[pulpdood]

Hi, we've created a boss worker for a job queue, and in the job handler we call boss.fail(job.id, err) in a catch block whenever our handler code in the try block throws an error. We are creating jobs using boss.send(queueName, data, RETRY_CONFIG), where RETRY_CONFIG contains the necessary settings.

We want the job to retry whenever the job handler throws an error and calls boss.fail, but it only retries some of the time and other times the state of the job is just set to completed, and this retry behaviour is therefore inconsistent. Note with this I am testing and making it so it always errors and call boss.fail.

I tried printing out the output of boss.fail, and I get an object with three properties, including one called updated.

When the job correctly retries after the boss.fail, the output is updated: 1

done {
  jobs: [ '132f8d40-69d5-4228-9f70-c940e8f2814b' ],
  requested: 1,
  updated: 1
}

but most of the time the job goes straight to completed and we get updated: 0 from boss.fail

{
  jobs: [ '09b735b4-9230-4513-a3be-9b42bfd34fb4' ],
  requested: 1,
  updated: 0
}

I tried making it so that job.fail is called a second time after a delay if the first on didn't retry, but it still doesn't correctly retry and I get updated: 0 both times.

            let fail = (await boss.fail(job.id, e) as any);
            console.log(fail)
            await new Promise(f => setTimeout(f, 3000));
            if(fail.updated === 0) {
                console.log((await boss.fail(job.id, e) as any))
            }

Just wondering why this behaviour is inconsistent and anyway to resolve? Is there some concurrency issue?

[timgit]

If you are using work(), it will automatically call fail() or complete() internally. Once a job reaches complete state, it can't be marked as failed.

Would you add more code to examples to show how you got this job id?

[pulpdood]

This is the worker for the queue which calls the handler with the job

    boss.work(CoreQueue.WEBFLOW_CHECKOUT, options, (job: any) => {
        webflowCheckout(job);
    });

And this is the handler itself:

    const webflowCheckout = async (job: any) => {
        logger.info(`job '${job.name}' triggered`);

        try {
            // will throw validation error
            await webflowCheckoutHandler(job.data.webflow, boss);
            await boss.complete(job.id);
        } catch (e) {
            await boss.fail(job.id, e); // only retries some of the time
            logJobError(e, job);
        }
    };

Both codes are sharing the same boss object as they are both wrapped in an async function

We use boss.send (in a different container using its own boss instance) to create the job, i.e.

await boss.send(CoreQueue.WEBFLOW_CHECKOUT, webflowJobData, RETRY_CONFIG);

where RETRY_CONFIG is:

export const RETRY_CONFIG = {
    retryLimit: 10,
    retryDelay: 10,
    retryBackoff: true,
};

We would like the job to be in retry state when it fails.

[timgit]

It looks like your work() syntax is malformed. Use the following syntax instead to make sure you're returning the promise from async to the worker.

boss.work(CoreQueue.WEBFLOW_CHECKOUT, options, (job: any) => webflowCheckout(job));

Also, as I mentioned in my previous comment, do not call fail() or complete() inside a handler. The worker does this automatically based on a try ... catch around your function.

[pulpdood]

Does this mean we have to throw an error from webflowCheckout(job) in order to trigger a retry?

[timgit]

It means you won't typically need a try...catch in your handlers unless you want to log the error and re-throw

[pulpdood]

Thank you! I fixed the work() syntax and threw the error from the handler and retries are working properly now. Just wondering if this should be updated in the docs which still says we can use job.done(error).

pg-boss/docs/readme.md

Line 874 in 9938b1a

.catch(error => job.done(error))