Trigger state "BLOCKED", never gets unblocked and job stalls out in Quartz .NET 3.8

[xantari]

Describe the bug

TRIGGER_STATE in QRTZ_Triggers is shown as "BLOCKED". Job never resumes. How am I supposed to recover from a "BLOCKED" trigger?

Version used

Quartz .NET 3.8 - .NET Core 8.0

To Reproduce

Have no idea how it became blocked. It was processing a long list of data and just stalled out.

Expected behavior

Job should go back to Waiting status....

Additional context

From what I can tell, the job begin at 11:31 AM PST, ran for a very long time. It is a every 5 minute recurring schedule. The 11:31 AM PST job was still running at 11:35:01 AM when it tried to fire again and it tried to start again. Which is wierd, because [DisallowConcurrentExecution] is on the job. I think this is where it got deadlocked with itself.

Must not be monitoring the DisallowConcurrentExecution tag anymore I suspect.

Reverting back to 3.5.0 and will monitor the situation more.

[danports]

I just ran into the same issue today and am trying to figure out how to recover the stalled job too. Just setting the TRIGGER_STATE back to WAITING wasn't enough - I saw Handling 1 trigger(s) that missed their scheduled fire-time. in the logs, but the stalled job didn't actually fire, and its TRIGGER_STATE was just set back to BLOCKED. 🤔

[danports]

OK, what worked for me to fix the stalled job was:

DELETE FROM QRTZ_FIRED_TRIGGERS WHERE TRIGGER_NAME = 'TriggerName';
UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = 'WAITING' WHERE JOB_NAME = 'JobName';

@xantari Did you try any of the releases between 3.5.0 and 3.8.0?

[xantari]

Nope didn’t try those versions. Went back to 3.5 for now.

[gtbuchanan]

We have a similar problem occur intermittently in 3.8, but the TRIGGER_STATE ends up as ERROR. No error actually occurred in the job that we can find logs for. This causes the scheduler to hang on shutdown. Manually updating the TRIGGER_STATE back to WAITING resolves the problem.

[danports]

That might be #707, which I've run into too - I commented there but didn't get any response, but given the age of that issue, it might be worth opening a new issue if you think that's the problem you're having too.

[gtbuchanan]

For what it's worth, we eventually started encountering the BLOCKED state as well. Strangely enough, we have been on 3.8 since December 4th and 3.8.1 since April 4th but we only started encountering the problem on April 10th after we upgraded from the .NET 7 Runtime to the .NET 8 Runtime on the server (and a few other minor transitive upgrades). This could obviously just be a coincidence but I wanted to mention it anyway.

EDIT: We actually found our problem to be in CsvHelper. We switched from CsvWriter.WriteRecords(IEnumerable, CancellationToken) to CsvWriter.WriteRecordsAsync(IEnumerable, CancellationToken) and a few of our jobs started hanging unexpectedly. This caused them to enter the BLOCKED state, which now seems the reasonable thing to do 😁. My apologies for the red herring.