use Transactions for agenda

[harisvsulaiman]

Purpose

Users want the Agenda CRUD operations to be part of an ACID MongoDB transaction. (This excellent blog post gives me a hint to say so.)

Given the following scenario with pseudo code of creating a user in a databse:

Start Transaction.
1. Create User Document.
2. Create Agenda Job Document
Commit Transaction.

... the multi document transaction ensures ACID compliance, so that either both or none of the documents will be created in the DB. If step 2 fails, step 1 will be rolled back/not committed.

Todo

• Add sessions for job creation methods
• Add sessions for job enable, disable, cancel, purge
• Add Internal transactions for run where the session might be pass down to the job processor . This should be optional, because mongodb transactions have timeout of 60 seconds.
  See https://www.mongodb.com/blog/post/performance-best-practices-transactions-and-read--write-concerns
• Add Tests
• Document the new api

Questions

• Are we going to support mongo pre 4.0? if so how?
• Should we pass session to the job processor to make it fully atomic?
• should we introduce any breaking changes to the api or is it perfect the way it is?

Ps: This is based on the stale pr #715
@koresar @simison

Signed-off-by: Haris Sulaiman harisvsulaiman@gmail.com

[harisvsulaiman]

I would appreciate it if someone could help me with tests.

[koresar]

I took a look at the changes. They are great!
Sorry, I can't help with this. Also, looks like you're on your own.
You can try asking people in Agenda Slack maybe?

[harisvsulaiman]

@koresar Sure I will post it there. I will take my time completing this pr then.

[pdfowler]

@harisvsulaiman This is nice work and has pointed me towards some reading that I'm probably overdue on wrt transactions in mongo. I've left a couple of style comments and will take a closer look at testing options

Is there any more context you can give on the tests you have started on and where you see the primary blockers?

On the topic of your questions:

Are we going to support mongo pre 4.0? if so how?

I would vote no support for 4.0, but see some options for handling it, from simplest to more involved: document that sessions require 4.0 (minimum); check DB version on connect, throw error if enabled and <4; Log warning and only enable sessions if >= 4.0.

Should we pass session to the job processor to make it fully atomic?

My gut is to leave that to a separate PR but I don't have sufficient context to say one way or the other

should we introduce any breaking changes to the api or is it perfect the way it is?

In my read tHru of the changes, I think the appending of session as the last arg is a fair implementation and not overly invasive. Do you have specific breaking changes in mind?

[pdfowler]

Could be evaluated to an empty string instead of a Boolean if shouldUseTransactions: ''. Technically no impact since both are falsely, but better IMO to use double negation when interpreting default-false args/flags.

[pdfowler]

Nit: personal preference for eslint's no-undef-init, but I'd have to take a closer look elsewhere in5e project for consistency over preference

[pdfowler]

At a glance, this import doesn't seem to be used. Is this a dev artifact, mistakenly unused, or serve so,e other function?

[harisvsulaiman]

@pdfowler Thank you for the feedback.

I would vote no support for 4.0, but see some options for handling it, from simplest to more involved: document that sessions require 4.0 (minimum); check DB version on connect, throw error if enabled and <4; Log warning and only enable sessions if >= 4.0.

We can check if sessions are supported when we initialize agenda. Then disable it altogether.

My gut is to leave that to a separate PR but I don't have sufficient context to say one way or the other

I think we should make that optional too.

I haven’t gotten around to writing tests for this implementation, I would appreciate it if you could lend a hand.

[simllll]

Hi all, I was looking into this out of curiosity. Am I right that the main use case for this is that a job's database queries are then handled by one "transaction"? First I was thinking that this is a very interesting idea, but I don't really get it now:

• Would that mean agenda starts a new transactions / session for e.g. a new job, and all the queries following should relate to this one? But how can a job itself then access the current session? E.g.
  -> "create new user" .. agenda starts transaction . the create new user creates some db entries (how can they use the session that agenda has started?), .. third one fails.. everythign shuld get rolled back?
• What is if mongodb of agenda and the job's things are happening on different databases?
• What if the job does other stuff too, like sending mails or whatever..this is not part of this transaction then.
• Then there is this 60 second limit, so this limits this use case also, but still struggeling here with the first point anyway.

For the other operations, what would the expected benefit be?
I would see one for creating and deleting jobs, because there the application could pass that in, but this would also assume that there are some pre requesitions that are fullfilled, not sure if this isn't easier to solve with a simple try / catch error handling.

Don't get me wrong, I really want to support your submission, but I don't fully understand it yet?

[davidpetrea]

Hello, any update on this?