Question: handling complex dependency graphs

[bh2]

I just installed Cronicle and love it - everything works so well and is easy to understand.

I was curious if Cronicle can support one particular use case though, which (and I think is pretty common for users of job scheduling systems like this) involves more complex dependency management.

I see that there is the concept of chains, and I got that working, but what about:

1. more complex dependencies? For example: Event C depends on (i.e. should only be run upon successful completion of) Events A and B, which should be run in parallel.
2. dependent jobs that run on a scheduled time? For example: Event Y depends on Event X. Event X runs at 8am and typically takes 2 minutes to run. Event Y should run at 8:30am, but only if Event X was successful. (Using chain reactions, I only see a way to make Event Y run immediately after Event X successfully completes).
3. visualization of chain reactions? Say I have 100 events, where each one depends on the successful completion of the previous job. It seems like I can achieve this by creating Event A and scheduling it to run at a particular time, and adding a chain reaction event (Event B). Event B would have a chain reaction event (Event C), and so on. However the only event that would be "officially scheduled" (and thus the only one that would appear in the "Upcoming Events" list) would be Event A. Is that right? Is there a better way to configure and or visualize the full job list of what I'm describing?

Thank you!

[jhuckaby]

Thank you for the compliment!

This feature will be included in Cronicle v2 (Orchestra), slated for release this year (2023). We're introducing the concept of "Workflows", which are groups of events that can run in sequence, in parallel, or combinations of the two. You can also configure things like "delay from start", so an event can run at an offset (say 2 hours) from the start of a workflow, and you can choose to abort a workflow on a single job failure.

All events in a workflow can exchange data between jobs, so you can pass data (and FILES!) from one job to the next.

I believe this should cover all three of your use cases above.

[ovizii]

Is there a way to stay updated on your progress on Orchestra? i.e. a newsletter or something? I looked at your profile to find the Orchestra repo but I either failed to find it or its not public yet.

[jhuckaby]

@ovizii I'm so sorry, I haven't released anything like that yet. I prefer to work offline, as doing it in public adds way too much pressure and stress. I can tell you that progress is coming along nicely, though. I'm still on schedule for a 2023 release.

[ovizii]

Just out of curiosity, what is the progress on Orchestra looking like?

[jhuckaby]

Ugh, I'm so sorry that I missed my 2023 deadline for Orchestra. Life keeps getting in the way. I'm still working on it every chance I get. I am just now, today, working on a real-time network monitoring system for active jobs, so you can see network bandwidth throughput across your job processes. It's super cool!

Progress is being made -- still a lot to complete tho. I don't want to make any more promises that I can't keep, so I'll just say, 2024 is the year! 😃

[ovizii]

Never apologize, it's your hobby, right? If you have to stick to any timelines and report your progress I guess it wouldn't be as much fun for you so happy coding, and it'll be done when it is done. :-)

[jhuckaby]

I've moved this to a discussion, because many people consider the number of "open issues" to be a sign of trouble on a repo.

[ovizii]

Sounds great, thanks for taking the time to reply and for this awesome tool :-)

[StephenAtKinetiq]

It would be awesome to see flows visualized with run times, last status, etc.

[dheise-rheinenergie-trading-com]

For the visualization of cronicle chains I can recommend my self-written tool. 😆: https://github.com/dheise-rheinenergie-trading-com/graphicle