Review usage of Cosmos with Airflow 2.3 & older version of dbt

[tatiana]

Context

We are aware there are conflicts between Airflow/dbt dependencies. And there is a cost involved with supporting older versions of Airflow, as seen in #944.

Acceptance criteria

• Check which versions of Airflow are being used by Astronomer customers who use Cosmos
• Make a survey with the community to check which versions of dbt & versions of Airflow they are using
• Share the survey with the community
• Make a decision about what is the oldest version of Airflow we want to support (the survey will be open for a while, we should close this ticket for now, and have a follow up task to check on this - Review dbt in Airflow Survey results #1024 )

[tatiana]

Initially I thought about doing a "/poll" in the #airflow-dbt channel. However, since it will disappear from Slack in 90 days, I decided to have this in a Google Form.

We already planned to have two questions, and this could be an opportunity to add more. I drafted a survey:

• https://docs.google.com/forms/d/e/1FAIpQLSdlFdh3UkZNlTNX2_7V0Wo0tZdVHMzGZsY3Q4j_0_e6JCzQhw/viewform
• https://docs.google.com/forms/d/1xYKsClnedr-e_d5BHEvVc0xsTOJgL7bfW6GlZM66gKs/edit

So far, I have shared with the following people for initial feedback:

• @cmarteepants
• @jlaneve
• @kaxil
• @pankajkoti
• @pankajastro

[tatiana]

I queried Cosmos task runs since 1 January 2024 by Astronomer Customers, and identified which versions of Airflow and Runtime (that maps to Python versions), they used:

• Airflow versions: 2.4.3 - 2.9.1
• Runtime versions: 6.2.1 - 11.3.0 (Python 3.8 - Python 3.12, based on the docs)

Based exclusively on this, we could drop Airflow 2.3 from Cosmos test matrix. I also checked the usage since 1 May 2024, and unfortunately, Airflow 2.4.3 was still there.

Since the Astro Runtime will be moving to support every Airflow version for two years, 2.4 would still be in there. So, it is probably worth supporting it for the time being.

These diagrams show the percentages of tasks run with Cosmos operators per Airflow/Runtime version:

[tatiana]

Feedback from @cmarteepants:

• Open question: How relevant is to know job titles?
• How could we make more clear the granularity question - we agreed to give more options ("I don't know")

Feedback from @kaxil:

• In future, some of this data could be automatically obtained by emitting metrics from Airflow deployments (e.g. versions), if the user consents

[tatiana]

I addressed all the feedback and we announced the survey: https://bit.ly/dbt-airflow-survey-2024
Considering this ticket done 🎉