Airflow fully supports multi-tenancy #39419
Replies: 3 comments 1 reply
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
Please do not use Github Issues for the questions. For this kind of requests please use Github Discussions and Apache Airflow Community Slack Workspace |
Beta Was this translation helpful? Give feedback.
-
|
Also this question asked couple times in discussions: https://github.com/apache/airflow/discussions?discussions_q=is%3Aopen+multi-tenancy+ |
Beta Was this translation helpful? Give feedback.
-
|
Close this as duplicated |
Beta Was this translation helpful? Give feedback.
-
Description
Does Airflow now fully support multi-tenancy now?
Use case/motivation
We have a requirement to share one Airflow cluster among different teams within a single organization.
Currently, we provision a dedicated Airflow cluster for each team to achieve multi-tenancy. However, teams within the same organization often develop independently but collaborate on the entire business data flow. For instance, Team A's pipelines may serve as the upstream for Team B, requiring Team B's pipelines to wait for Team A's pipelines to finish before starting. Achieving this coordination across separate Airflow clusters is challenging without external signals. Additionally, potential version conflicts may arise among Python dependencies used by different teams if they share one Airflow cluster.
Furthermore, teams have varying requirements for computing resources such as CPU, GPU, and memory. Some teams need worker nodes with ample memory, while others delegate heavy tasks to remote systems and require minimal resources.
Our multi-tenancy requirements are as follows:
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions