Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Request] RAM Cache Middleware for ADOJobStore #2235

Open
da3dsoul opened this issue Jan 6, 2024 · 0 comments
Open

[Request] RAM Cache Middleware for ADOJobStore #2235

da3dsoul opened this issue Jan 6, 2024 · 0 comments

Comments

@da3dsoul
Copy link

da3dsoul commented Jan 6, 2024

Is your feature request related to a problem? Please describe.
Kind of? It would help with the performance issues with ADOJobStore and the RAM requirements of RAMJobStore. In my opinion, it would be a happy middle ground.

Describe the solution you'd like
Basically, it would use write-back caching on a RAM Store. If a job is scheduled for far enough in the future, then it would be passed to the Job Store immediately. If it's set to run immediately, then it runs per the concurrency settings etc etc, and if it takes too long to run, then it's passed to the Job Store for persistence. I particularly would like to run this via SQLite, as there are likely to be many relatively fast jobs scheduled, and (haven't tried it yet tbh, just been reading) that can cause performance issues, especially with the latency of SQLite. This could also help with situations where you have a cloud job store, and the latency to the database isn't great. It'll only pass jobs through if they aren't run "immediately".

Describe alternatives you've considered
An alternative is to obviously use a DBMS that has write-back RAM caching. I prefer not to require users to install a DBMS with such a heavy feature, though.

I kind of expect that I'll need to experiment and see how necessary it is for my use case, and that if it's even necessary, I'd need to try to implement it myself. The performance FAQ was pretty realistic in saying that you should just use the right tool for the job, and that's a better DBMS. My other concern is how it handles SQLite's concurrency issues. I've fought with those before, but I'm hoping I don't need to fork the SQLite Job Store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant