Airflow Multi tenancy question

[kimkihoon0515]

Hey, I just checked https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components.
So the multi tenancy is not supported for 2.9.0 version??

[potiuk]

There is no yes/no answer to your simple question because "multi-tenancy" can be understood differenetly - depends who you ask and what you are considering as multi-tenancy.

Depends what level of multi-tenancy you talk about. You might already today deploy airflow in some forms of tenancy support. In the document you can see various levels of muti-tenancy you can achieve even now in Airflow. This document fairly accurately explains today's state of it and you can see if that fits your case.

Even if the proposal is approved and implemented, it implements a very specific "team" variant of "multi-tenancy" - so you might want to consult that.

[kimkihoon0515]

@potiuk
thx for the reply.
I was asking about this tenancy option

C. Using a single instance of Airflow by different tenants

In the Airflow as we have it today (upcoming version 2.9) the execution and parsing environment “per-team” could be separated for different teams.

You could have separate set of DAG file processors per each folder in Airflow DAG folder, and use different set of execution environments (libraries, system libraries, hardware) - as well as separate queues (Celery) or separate Kubernetes Pod Templates used to separate workload execution for different teams. Those are enforceable using Cluster Policies. UI access “per team” could also be configured via custom Auth Manager implementation integrating with organization-managed authentication and authorization proxy.

In this mode, however, the workloads still have access to the same database and can interfere with each other’s DB including Connections and Variables - wmanhich means that there is no “security perimeter” per team, that would disallow DAG authors of one team to interfere with the DAGs written by DAG authors of other teams. Lack of isolation between the team group is the main shortcoming of such a setup that compared to A. and B. this AIP aims to solve.

[potiuk]

It's possible and various users are doing it. One of the ways is roughly outlined here https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#using-multiple-docker-images-and-celery-queues but this is something those users do completely on their own. There is no "flag" to switch or easy configuration - it's completely up to deployment manager to implement it in the way that will be good for them. Some people do it by telling their DAG authors to use KPO only for example. some might implemented custom cluster policies and separate celery queues for different "Tenants".

[potiuk]

I just clairfied it in this chapter in the Draft AIP - just to make sure others will not assume it's more than it is.

[kimkihoon0515]

@potiuk
Can't find

Multi-tenancy of scheduler and webserver is controlled by a "core/multi-team" bool flag (default False).

multi-team in configuration file :(
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#core

[potiuk]

This is a draft "Proposal". Not even approved yet. It's quite far from being implemented

[TakawaAkirayo]

Hi @potiuk
We have a requirement to share one Airflow cluster among different teams within a single organization.

User cases/Motivations
Currently, we provision a dedicated Airflow cluster for each team to achieve multi-tenancy. However, teams within the same organization often develop independently but collaborate on the entire business data flow. For instance, Team A's pipelines may serve as the upstream for Team B, requiring Team B's pipelines to wait for Team A's pipelines to finish before starting. Achieving this coordination across separate Airflow clusters is challenging without external signals. Additionally, potential version conflicts may arise among Python dependencies used by different teams if they share one Airflow cluster.

Furthermore, teams have varying requirements for computing resources such as CPU, GPU, and memory. Some teams need worker nodes with ample memory, while others delegate heavy tasks to remote systems and require minimal resources.

Our multi-tenancy requirements are as follows:

1.Isolation of:

• Execution environments
• Workers
• DAG resources
• etc...

2.Visibility of all pipelines from different tenants
3.Granular access control lists (ACLs) at the tenant level

Questions
I have some thoughts and I would like to seek your advice

1. Should we introduce the concept of 'tenant'

• The tenant is a group of users with roles (exclude admin), the tenant is created by the admin, and each tenant has independent sub DAG folder for isolation. By default, there is one default tenant.
• Everything are managed independently and protected (like users and roles) under each tenant, and there should be no cross tenant operations.

2. Seamless execution environment isolation between tenant

• Currently we can isolate execution environment by docker regarding: https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#using-multiple-docker-images-and-celery-queues but it need the Airflow cluster admin to be involved for configuration and setup for each onboarding of new tenant. So if the concept of 'tenant' is introduced, can we try to make those setup transparent for new tenant's onboarding?
• Also as the tenant have their sub DAG dir, so can I consider that the DAG resources and python dependencies are isolated? Maybe the computing resource like CPU/GPU can be shared on the same worker node among tenants.

[potiuk]

See if the AIP here responds to your need https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-67+Multi-team+deployment+of+Airflow+components - we started voting on it, but currently it is put on hold as we need to decide to what extend it should be implemented in Airlfow 2 vs Airflow 3 - but any comments and review of the proposal there is what you need.