Add option to prevent most recent run on DAG enable with catchup=False

[tambulkar]

Description

As mentioned in AIRFLOW-1156, I want to add the option to not run the most recent scheduled DAG on enabling.

Use case / motivation

We want to be enable the DAG for future runs, but don't want it to run the most recent task automatically, since there are tasks e.g. sending an email, that needs to be done manually on occasion. We do not want to repeat this task multiple times.

Related Issues

AIRFLOW-1156

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[22quinn]

I think if start_date is set properly, it will not be triggered immediately.

For example, for a DAG that is scheduled to run daily, if you enable it today (2020-07-22) and do not want to trigger the most recent run (execution_date = 2020-07-21) immediately, simply set start_date to 2020-07-22 or a later date whenever you want it to start.

[tambulkar]

I think if start_date is set properly, it will not be triggered immediately.

For example, for a DAG that is scheduled to run daily, if you enable it today (2020-07-22) and do not want to trigger the most recent run (execution_date = 2020-07-21) immediately, simply set start_date to 2020-07-22 or a later date whenever you want it to start.

That would only work for the first time you enable it. I don't want to have to keep updating the start date if I disable it and enable it at some point after the start date.

[22quinn]

I think if start_date is set properly, it will not be triggered immediately.
For example, for a DAG that is scheduled to run daily, if you enable it today (2020-07-22) and do not want to trigger the most recent run (execution_date = 2020-07-21) immediately, simply set start_date to 2020-07-22 or a later date whenever you want it to start.

That would only work for the first time you enable it. I don't want to have to keep updating the start date if I disable it and enable it at some point after the start date.

Right. Now I get the point.

I think this is an interesting case and requires some change to part of the scheduling logic. I'll work on it.

[22quinn]

Just want to write down my exploration and thoughts which might help if anyone wants to pick it up or discuss it further.

I looked into the scheduling logic and it turns out that the behaviour of catching up the most recent run was not designed deliberately. Rather, it is a "side effect" of the fundamental scheduling logic.

Airflow scheduler runs like a batch job. For every few seconds, it parses the DAGs and determines the next dag_run (if any) for every DAG. For example, for a DAG scheduled to run hourly 0 * * * *, a scheduler cycle starting at 2020-07-22T02:00:09 will schedule the "next" dag_run next_execution_time = 2020-07-22T02:00:00 (in other words, execution_time = 2020-07-22T01:00:00). Note that although I'm using the word "next", for the scheduler, it is doing something like a look-back to determine and schedule the "next" dag_run. If a DAG is paused at the beginning and is enabled at 2020-07-22T02:30:00, the next scheduler cycle will look back and still find the "next" dag_run to be next_execution_time = 2020-07-22T02:00:00. So it appears to be a catchup but it is really just a longer-delayed scheduling.

Therefore, if we want to prevent the most-recent catchup, we might have to add some more complex logic and modify the DAG model. For example, adding a timestamp to record when a DAG is enabled so that scheduler can skip most-recent run. I'm not sure if it's worth it for this feature.

[eladkal]

For anyone who is interested in this feature check #38168