Lock 'STATE_ACCESS' wrongful returner

[Suchiman]

Describe the bug

Ever since upgrading from ASP.NET Core 7 to ASP.NET Core 8, we occassionally see the following warning in the log:

[13:42:36 WRN] Lock 'STATE_ACCESS' attempt to return by: f0bfa366-4955-4c00-a9d1-bfe11c884af8 -- but not owner!
[13:42:36 WRN] stack-trace of wrongful returner:    at System.Environment.get_StackTrace()
   at Quartz.Impl.AdoJobStore.DBSemaphore.ReleaseLock(Guid requestorId, String lockName, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.ReleaseLock(Guid requestorId, String lockName, Boolean doIt, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.ReleaseLock(Guid requestorId, String lockName, Boolean doIt, CancellationToken cancellationToken)
   at Quartz.Impl.AdoJobStore.JobStoreSupport.DoCheckin(Guid requestorId, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()
   at Quartz.Impl.AdoJobStore.JobStoreSupport.ClusterRecover(ConnectionAndTransactionHolder conn, IReadOnlyList`1 failedInstances, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Quartz.Impl.AdoJobStore.StdAdoDelegate.DeleteFiredTriggers(ConnectionAndTransactionHolder conn, String instanceName, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c.<OutputWaitEtwEvents>b__12_0(Action innerContinuation, Task innerTask)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Threading.Tasks.UnwrapPromise`1.TrySetFromTask(Task task, Boolean lookForOce)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessInnerTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.ProcessCompletedOuterTask(Task task)
   at System.Threading.Tasks.UnwrapPromise`1.InvokeCore(Task completingTask)
   at System.Threading.Tasks.UnwrapPromise`1.Invoke(Task completingTask)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()

We've not yet noticed adverse affects from this so we've been ignoring it but i thought it might be best to report it eventually.

Version used

3.8.0

To Reproduce

No clear way to reproduce is known, the only hint we have is that this happened without upgrading Quartz (so unlikely to be a regression in Quartz) and we've never seen this before ASP.NET Core 8 and that we're using the following configuration:

services.AddQuartz(q =>
{
    var builder = new SqlConnectionStringBuilder(connectionstring);
    builder.ApplicationName = "QuartzNet";

    // default max concurrency is 10
    q.UseDefaultThreadPool(x => x.MaxConcurrency = 5);
    // this is the default
    // .WithMisfireThreshold(TimeSpan.FromSeconds(60))
    q.UsePersistentStore(x =>
    {
        // force job data map values to be considered as strings
        // prevents nasty surprises if object is accidentally serialized and then
        // serialization format breaks, defaults to false
        x.UseProperties = true;
        x.UseClustering();
        // there are other SQL providers supported too
        x.UseSqlServer(builder.ToString());
        // this requires Quartz.Serialization.Json NuGet package
        x.UseNewtonsoftJsonSerializer();
    });
});

services.AddQuartzServer(options =>
{
    options.WaitForJobsToComplete = true;
    options.AwaitApplicationStarted = true;
    options.StartDelay = TimeSpan.FromMinutes(5);
});

services.AddQuartzHostedService();

Expected behavior

No warnings to be printed