Usage in serverless environment

[bentron2000]

I'm working with a next js application deployed on Vercel backed by postgres.
I'm interested to know people's thoughts on using pg-boss for building a queue manager in this setup - considering that there would be no persistent node process running.

Below are some of the things I've been considering...

• Each server request that deals with queues could spin up an instance of pg-boss to interact with the queues.

  • Is starting an instance of pg-boss expensive?
  • Does a dormant instance of pg-boss consume an active connection to a db?
  • Is this what noSupervisor/noScheduling options are for?

• Vercel has cron functions that could be called to perform any cleanup or maintenance tasks - alternatively, pg-cron could achieve similar results.

  • Can cleanup/maintenance functions be run manually? If so, what would be the recommended interval for such tasks?
  • If not - would the idea be to trigger a function that would query for active queues, and if so, setup an instance of pg-boss with monitor-states set to a low number? (this could be set to run for just slightly less than the timeout period of the function?)

• Each job in a queue could call back to an API endpoint to receive data (or handle errors) from a completed job and could then call the next job.

  • It looks like fetch/complete/fail are good for this purpose?

• A limit could be set for the number of concurrent jobs in a specific queue

Any thoughts on this welcome!

Thanks
B

[timgit]

Yes, I would recommend using the fetch/complete/fail API since these aren't polling the database looking for work. Starting an instance of pg-boss is not expensive, especially if you're skipping the maintenance via noSupervisor. If you choose to do that, you would want to start an instance in a cron at some point to ensure things like the retention policy are being enforced. I haven't added a maintenance API yet to run this manually, but that would probably be ideal in order to know when it's done. Until then, you can listen for the maintenance event and then stop it.

[bentron2000]

Thanks for the response @timgit :)

This is great info.

I've taken your advice re the maintenance process - however theres some strange behaviour happening.
I've submitted an issue #405

To summarise for others who might like to achieve the same in a serverless env:

1. Queueing and De-Queueing jobs can be done by spinning up a PgBoss instance with { noSupervisor: true } option, and using send() and fetch(). Jobs are completed using complete() and fail()
2. In the absence of a specific maintenance API, maintenance tasks can be executed periodically in a separate instance by omitting the { noSupervisor: true }. Shutting down the process after maintenance tasks are run can be achieved by listening to the maintenance event - pending Maintenance Interval Issues / config options ignored #405
3. Getting the number of currently active jobs is not something there is a native function for, but can be done with something like this...

  function numActiveJobs() {
    const schema = this.boss.db.config.schema
    const result = await this.boss.db.executeSql(
      `SELECT COUNT(*) as count FROM ${schema}.job WHERE state = 'active'`
    )
    return result.rows[0].count
  }

Thanks again :)

[trompx]

Hey @bentron2000 looks like we are trying to achieve the same thing.
One idea is that each time a background job is added to the queue, save it to a dynamoDB table with a trigger to hit the api route that run the maintenance (so that the processing of the job is done in a new lambda). For scheduled tasks, have an eventbridge rule that run every minute and hit an api route that execute the ready jobs (also in a new lambda).
But maybe you found an easier way? (I wonder if it is possible to trigger an async request without waiting for the response in a next.js app, so we don't have to use aws SQS or dynamoDB for that).