Quartz.NET 4.0 Roadmap

[lahma]

This is an umbrella issue gathering things that could be done in 4.0 timeframe.

• Refactor persistence API to be more document database friendly, When will the RavenDB job store implementation be ready? #814 probably contributes to this
• Use ValueTask instead of Task, RamJobStore doesn't need heavy machinery
• Use Microsoft.Extensions.Logging.Abstractions instead of liblog
• Add HTTP API for scheduler, maybe via Quartz.AspNetCore package, existing issue HTTP based API #511
• Use the new TimeProvider abstraction Change SystemTime to TimeProvider / Microsoft.Bcl.TimeProvider #2079
• System.Text.Json serialization support

[Sicos1977]

A replacement for .net remoting like a rest api would also be a nice thing

[lahma]

Thanks @Sicos1977 , added #511 to the list

[Pinox]

An option to load persistent data from a cloud provider like Azure Table Storage (No SQL) would be an awesome low cost solution for running quartz.

Currently I have quartz persistent data running on a separate SQL DB and it's such a waste of money especially the fact that I'm not running many jobs on the DB. Table storage where you only pay for the data that you use will make much more sense for many people as I think most Quartz db's are probably very small in size. Table storage is cheap, fast and accessible from anywhere with near infinite scaling. A 1GB solution on Azure Table Storage will cost you $0.13 per GB

https://azure.microsoft.com/en-gb/services/storage/tables/

[lahma]

@Pinox what I've understood the row key is used for fast lookups in table storage. Quartz needs to check next fire time which is changing value for triggers and if you have thousands of triggers might become a problem. Azure table storage might not scale well in this case?

[Pinox]

@lahma I know there is a limit for transactions (20 000 per second) and entities (2000 per second.) per partition. So yes if you do more than thousands per second it will not scale well on Azure Table Storage but you still have general NoSQL DB's as option for such scenario's.

I just think if Quartz was able to work on NoSQL then for smaller type workloads it will be a perfect fit for a NoSQL solution such as Azure Table Storage and if you really need massive record update per second then you can always throw it in bigger NoSQL DB like CosmoDB / MongoDB and then you are can use the most appropriate hardware for your task that is persistent and very cost effective.

SQL Server now supports JSON format , so you can also run a "NoSQL database" on SQL Server without the relational schema that is usually the case.

https://docs.microsoft.com/en-us/sql/relational-databases/json/store-json-documents-in-sql-tables?view=sql-server-ver15

[ghd258]

MYSQL 8.0 now supports JSON format , so you can also run a "NoSQL database" on MYSQL 8.0 without the relational schema that is usually the case.

[DaveLaa]

It would be great to have a sepparation of functionality into specific libraries. There could be a Quartz.Net Core dll without external dependencies and specific dlls für the different extensions. This would reduce the necessity to add libraries to a project that doesn't use them.

[lahma]

There are some libraries that are ubiquitous, like logging tracing etc so I would consider them as the necessary evil. Especially with .NET Core / NET5 it's quite normal that libraries that need to plug-in into larger constructs have these. DI system is a bit on the fence, but building a bridge is also an option.

[Pinox]

FYI - new timer API approved by MS.

dotnet/runtime#31525

[cd21h]

Please add job execution pipeline to allow applying cross-cutting concerns to job handlers.

Something similar to ASP.NET Core middleware
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/write?view=aspnetcore-5.0

[Vandersteen]

Middleware pipeline would be a welcome addition for us.
I know there are listeners, but they do not allow to 'wrap' things like: #1224

[lahma]

Middleware pipeline would be a welcome addition for us.
I know there are listeners, but they do not allow to 'wrap' things like: #1224

I think a PR would be most welcome. We can start breaking things in main as current version can be branched to 3.x maintenance.

[Doomer3D]

Please replace Json.NET with System.Text.Json for .NET 5+

[mkstephenson]

I would also second the replacement of Json.NET if it is possible

[lahma]

If someone wants to contribute a STJ implementation the same way there's a separate project for NSJ integration, that would be nice. We could offer it in 3.x timeline and make it default in 4.0.

[mkstephenson]

I'll have a look at it if I have time, can't promise anything at this stage though.

[tonyqus]

Please replace Json.NET with System.Text.Json for .NET 5+

I don't think System.Text is mature enough so far. JSON.NET is still much better than System.Text. If you have tried JSON Path Selector feature, you will understand what I mean. Don't replace a library just because the other one is newer.

[lahma]

Please replace Json.NET with System.Text.Json for .NET 5+

I don't think System.Text is mature enough so far. JSON.NET is still much better than System.Text. If you have tried JSON Path Selector feature, you will understand what I mean. Don't replace a library just because the other one is newer.

I think we can support both, JSON.NET shouldn't just disappear to ensure existing users keeping the old desired behavior. I can imagine there would be regressions with just changing to STJ, hard to predict things that can go wrong.

[tonyqus]

Please consider replacing MS DI with AutoFac. We did see a few unexpected issues happen in MS DI and Scoped lifecycle are not reliable at all. I believe it's kind of design issue of MS DI.

[tonyqus]

I'm reading some comparison between Quartz.NET and Hangfire. The major complaint is about how Jobs are created. Hangfire supports any method while Quartz.NET only supports IJob based tasks.

For example,

BackgroundJob.Enqueue(
    () => Console.WriteLine("Fire-and-forget!"))

I haven't dive deep into the code. Do you think it's possible to support any method with automatic IJob task creation?

[tonyqus]

Another major complaint is about automatic creation of database schema. Since Quartz.NET use sql script instead of code-first ORM (such as Entity Framework), the initialization setup experience is not as good as Hangfire.

Do you think we can add Entity framework based database schema creation feature? If you agree adding this to 4.0, I'm willing to work on that.

[lahma]

Please consider replacing MS DI with AutoFac. We did see a few unexpected issues happen in MS DI and Scoped lifecycle are not reliable at all. I believe it's kind of design issue of MS DI.

Or Lamar, or DryIoc.. the list goes on. There's already an Autofac integration that you can use.

I haven't dive deep into the code. Do you think it's possible to support any method with automatic IJob task creation?

I don't personally like the Hangfire way of things when we start to talk about persisted jobs. Clean and simple for in-memory jobs but complexity starts to hit with things like () => someService.Method(counter++), how to handle all context and captures, makes things harder to test too IMO.

Do you think we can add Entity framework based database schema creation feature? If you agree adding this to 4.0, I'm willing to work on that.

I don't think we need EF to just handle schema. Because schema doesn't change that often I think it would be enough to detect via switch whether intitial schema creation is allowed and the schema is missing. Maybe add support for detecting database schema version and if it's different from what Quartz requires, give a startup error.

[Cricle]

Will support nosql like MongoDB, LiteDB store?

#1019 I try but there is no suitable one

[lahma]

Will support nosql like MongoDB, LiteDB store?

#1019 I try but there is no suitable one

The core project don't have resources to support such specialized one, you can always improve the existing ones though.

[Cricle]

1. Add support for Redis.
   There is a project that adds Redis and has a PR that allows support for Quartz 3 (Ported to Quartz.net 3 icyice80/QuartzRedisJobStore#4) but official support would be awesome. Especially that the project has some limitations - https://github.com/RedisLabs/redis-quartz#limitations
2. Add documentation and sample on how to configure and use clustering - I have 3 instances of .NET 6 REST API behind Traffic, so I'd like to be able to easily configure clustering in Quartz.
3. Simplify how we register and set up Job Stores. Take a look at https://github.com/glucaci/mongodb-quartz-net#basic-usage, you must set properties (NameValueCollection). Ideally we should be able to do the same using fluent API.

I think 3. can be a PR(Although the project has been passively updated), if it display or I solved it, I will merge in temp Nuget package

[s3YwCf2ZbfJG4SHAfjQMAjtsf]

I don't think we need EF to just handle schema. Because schema doesn't change that often I think it would be enough to detect via switch whether intitial schema creation is allowed and the schema is missing. Maybe add support for detecting database schema version and if it's different from what Quartz requires, give a startup error.

Hangfire provides additional nugets to handle schemas with NHibernate. Yesterday, I started with Quartz.Net because of the issues I was having with Hangfire and Oracle and .Net 6. After a few hours, it was great that the schema script was available and saved me a great deal of time. Almost all of the requirements have been handled in just a few hours. I need to go back and handle load balancing with the AdoJobStore.

[michaeljon]

Has anyone done any work getting System.Text.Json as a supported serializer? I'd be willing to pick up and help. We have some really hairy JSON structures and I would not want to have to go back in and write Json.Net serializers for them.

[mkstephenson]

I started to look at it and got some stuff down, however I got stuck around some of the structures (and figuring out some of the tests), then some other stuff came up and I didn't get round to finishing it. I just checked the repo now and there appears to be some stuff surrounding System.Text.Json? However I don't know how complete it is.

[jafin]

In relation to the DI/container work for 4.0 , there appears work has been done since opening this ticket at improving the stance. Are there still outstanding goals that have not been met?

[Almaroo]

Section How do I chain Job execution? Or, how do I create a workflow?
mentions that

In the future, Quartz will provide a much cleaner way to do this, but until then, you'll have to use one of the above approaches, or think of yet another that works better for you.

Any chances of seeing it in the nearest future?

Cheers

[lahma]

Any chances of seeing it in the nearest future?

Would you be willing to contribute such a feature?

[Almaroo]

Provided my design got accepted, sure thing(:

I did not mean to be disrespectful in any way, sorry if it sounded that way(:

[Khitiara]

to clarify my previous statement, I am well aware that injecting the logger from DI is supported in 3.x; the option appears to be missing in 4.x; I was asking if that option is planned to be re-added (the checklist entry for further DI support in this issue does not have an associated more specific issue where i could ask this instead)

[lahma]

to clarify my previous statement, I am well aware that injecting the logger from DI is supported in 3.x; the option appears to be missing in 4.x; I was asking if that option is planned to be re-added (the checklist entry for further DI support in this issue does not have an associated more specific issue where i could ask this instead)

So it's no longer working or what is the problem? There's no plan to remove the support as far as I know.

[Khitiara]

to clarify my previous statement, I am well aware that injecting the logger from DI is supported in 3.x; the option appears to be missing in 4.x; I was asking if that option is planned to be re-added (the checklist entry for further DI support in this issue does not have an associated more specific issue where i could ask this instead)

So it's no longer working or what is the problem? There's no plan to remove the support as far as I know.

on 4.0, there is no way to pull the ILoggerFactory from the DI container when setting it up in the configure delegate to AddQuartz as the only option is to call SetLoggerFactory and within that delegate the final IServiceProvider is not yet available so only premade ILoggerFactory instances are available

[Khitiara]

at least as of the current state of the main branch that is

[lahma]

to clarify my previous statement, I am well aware that injecting the logger from DI is supported in 3.x; the option appears to be missing in 4.x; I was asking if that option is planned to be re-added (the checklist entry for further DI support in this issue does not have an associated more specific issue where i could ask this instead)

So it's no longer working or what is the problem? There's no plan to remove the support as far as I know.

on 4.0, there is no way to pull the ILoggerFactory from the DI container when setting it up in the configure delegate to AddQuartz as the only option is to call SetLoggerFactory and within that delegate the final IServiceProvider is not yet available so only premade ILoggerFactory instances are available

If you can create a public repository showing the problem, that would be nice. Ideally of course a pull request against the main branch fixing the thing as you are quite familiar with the problem domain - it would speed things up a lot.

[Khitiara]

https://github.com/Khitiara/Quartz4LoggerInjectionIssue/blob/develop/Program.cs shows a very simplified example of the problem; I would create a pull request but I'm not really sure how to actually solve this issue in an idiomatic way and I'm not intimately familiar with the quartz codebase - I just noticed the issue when trying out the preview out of curiosity

[SeanFarrow]

Would adding an EFCore IJobStore implementation be accepted for version 4?
I've already started writing one.

Thanks,
Sean.

[lahma]

I've created a PR against main (v4 development), which drops support for full framework and makes Quartz require NET 6.0 as minimum target platform: #2226

See poll results: #2080

[remcoros]

@Khitiara you can work around it by doing this:

var app = await builder.Build();
Quartz.Logging.LogProvider.SetLogProvider(app.Services.GetRequiredService<ILoggerFactory>());
await app.RunAsync();

[Khitiara]

@remcoros That would work but seems inelegant - thanks for the workaround! I might look into a way to PR support into quartz later for a more elegant solution.

[izanhzh]

Looking forward to the release of 4.0, I want HttpApi very much, and I can foresee that the community will easily extend various styles of background management UI