The individual components should be deployable without setting the executor first

[karakanb]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Motivation

I am trying to transition from KubernetesExecutor to CeleryKubernetesExecutor gradually. My goal is to do this transition as safe as possible without any disruptions, and do every change incrementally.

What I want to do is:

• Enable workers, deploy, ensure they can connect to Redis and they are working fine.
• Enable flower, deploy, ensure all is good.
• Change all my tasks to run on kubernetes queue, so that they'll behave the same way.
• Change the executor config to CeleryKubernetesExecutor
• Test individual tasks piece by piece by changing the queue.
• Finally, if all is well, roll it out fully.

I would like to be able to do this, because at each step I want to make sure all is good before I transition the pipelines, and I have a mixed set of workloads that I would like to keep on Kubernetes, while ensuring that after the rollout everything will keep running the same way.

As of now, I am not able to do this gradual switch because of the simple check on the helm chart.

Error: UPGRADE FAILED: execution error at (airflow/templates/_helpers/validate-values.tpl:43:5): If `airflow.executor=KubernetesExecutor`, then all of [`workers.enabled`, `flower.enabled`, `redis.enabled`] should be `false`!

Implementation

Remove the check for disabling workers / redis / flower.

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[thesuperzapper]

@karakanb The values validations are intended to ensure people don't deploy non-sensible combinations of values in the chart (as there are MANY options).

Is there a reason why you can't set your airflow.executor to CeleryKubernetesExecutor at the same time as your other changes to enable flower/redis/etc?

I believe you can change your DAG tasks to use the kubernetes queue while still using the KubernetesExecutor, so that you can enable CeleryKubernetesExecutor without some of the tasks unexpectedly swapping to the Celery workers, either by:

1. Setting the AIRFLOW__OPERATORS__DEFAULT_QUEUE to kubernetes
2. Manually set queue="kubernetes" in ALL your DAG task definitions

[karakanb]

I can do that, but that means that I have to deploy all of the changes at once, including potentially blocking the dag runs in case something is wrong, e.g. a task being forgotten to moved to a default queue, or a task setting an explicit queue somewhere; therefore, it seems to be the safer option to be able to incrementally deploy things and ensure they work step by step without breaking the current flow.

Would you be interested in a solution like skip-checks or sth similar that would allow skipping those checks for cases like this?