You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Kind of? It would help with the performance issues with ADOJobStore and the RAM requirements of RAMJobStore. In my opinion, it would be a happy middle ground.
Describe the solution you'd like
Basically, it would use write-back caching on a RAM Store. If a job is scheduled for far enough in the future, then it would be passed to the Job Store immediately. If it's set to run immediately, then it runs per the concurrency settings etc etc, and if it takes too long to run, then it's passed to the Job Store for persistence. I particularly would like to run this via SQLite, as there are likely to be many relatively fast jobs scheduled, and (haven't tried it yet tbh, just been reading) that can cause performance issues, especially with the latency of SQLite. This could also help with situations where you have a cloud job store, and the latency to the database isn't great. It'll only pass jobs through if they aren't run "immediately".
Describe alternatives you've considered
An alternative is to obviously use a DBMS that has write-back RAM caching. I prefer not to require users to install a DBMS with such a heavy feature, though.
I kind of expect that I'll need to experiment and see how necessary it is for my use case, and that if it's even necessary, I'd need to try to implement it myself. The performance FAQ was pretty realistic in saying that you should just use the right tool for the job, and that's a better DBMS. My other concern is how it handles SQLite's concurrency issues. I've fought with those before, but I'm hoping I don't need to fork the SQLite Job Store.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
Kind of? It would help with the performance issues with ADOJobStore and the RAM requirements of RAMJobStore. In my opinion, it would be a happy middle ground.
Describe the solution you'd like
Basically, it would use write-back caching on a RAM Store. If a job is scheduled for far enough in the future, then it would be passed to the Job Store immediately. If it's set to run immediately, then it runs per the concurrency settings etc etc, and if it takes too long to run, then it's passed to the Job Store for persistence. I particularly would like to run this via SQLite, as there are likely to be many relatively fast jobs scheduled, and (haven't tried it yet tbh, just been reading) that can cause performance issues, especially with the latency of SQLite. This could also help with situations where you have a cloud job store, and the latency to the database isn't great. It'll only pass jobs through if they aren't run "immediately".
Describe alternatives you've considered
An alternative is to obviously use a DBMS that has write-back RAM caching. I prefer not to require users to install a DBMS with such a heavy feature, though.
I kind of expect that I'll need to experiment and see how necessary it is for my use case, and that if it's even necessary, I'd need to try to implement it myself. The performance FAQ was pretty realistic in saying that you should just use the right tool for the job, and that's a better DBMS. My other concern is how it handles SQLite's concurrency issues. I've fought with those before, but I'm hoping I don't need to fork the SQLite Job Store.
The text was updated successfully, but these errors were encountered: