GCS & 1.10.2

[zifeo]

Using dag provision through gcs and the latest release continously crash loop.

airflow.exceptions.AirflowConfigException: In kubernetes mode the following must be set in the `kubernetes` config section: `dags_volume_claim` or `dags_volume_host` or `dags_in_image` or `git_repo and git_branch and git_dags_folder_mount_point`

Is there anything config specific to bypass this? I've tried setting manually the dags_volume_host which does not seem to be passed to the actual pods.

[barney-s]

Thanks for the PR. Would you elaborate and share the CRs ?

[zifeo]

@barney-s This is more a question than a PR. Is there anything I can add?

[Pritesh-Patel]

Running into this problem as well.

[Pritesh-Patel]

@zifeo I got past it by adding config AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"

eg:

apiVersion: airflow.k8s.io/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-cluster
spec:
  config:
    airflow:
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "100"
      AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"

Im syncing using GCS - so using dags_in_image seems to work for me, but im assuming for the dags_volume_host you would use something similar.

This was not the easiest to find might make sense to put it in the k8s executor examples? @barney-s

[zifeo]

@Pritesh-Patel Could you share the full config file (except the bucket name of course)? I have tried the same without success.

[Pritesh-Patel]

@zifeo

apiVersion: airflow.k8s.io/v1alpha1
kind: AirflowCluster
metadata:
  name: airflow-cluster
spec:
  config:
    airflow:
      AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "100"
      AIRFLOW__KUBERNETES__DAGS_IN_IMAGE: "True"
  executor: Kubernetes
  ui:
    replicas: 1
    version: "1.10.2"
  scheduler:
    version: "1.10.2"
  worker:
    version: "1.10.2"
  dags:
    subdir: ""
    gcs:
      bucket: "airflow-dags"
  airflowbase:
    name: airflow-base

Unfortunately after this I ran into SQL errors and decided to give up on this and use the helm chart. I would preferably like to use this - but i dont have the time to debug the issue. Let me know if you get further!

[max-sixty]

This is somewhat an aside - but if you want an config file which runs Airflow-in-Kubernetes on version 1.10.3 successfully, ours is open source: https://github.com/SixtyCapital/infrastructure/blob/master/docker/airflow/airflow.cfg

Our Kubernetes setup is more basic than this repo, unfortunately; though it works well

[zifeo]

@max-sixty Looks interesting although it does not seem to support the Kubernetes executor?

@Pritesh-Patel Thanks. I will retry soon. I am curious to see whether you managed to run the Kubernetes executor with the helm chart. It seems to be unstable and was the reason we switched to the operator.

[max-sixty]

Yes @zifeo - we would love to use the Kubernetes Executor but haven't found it developed enough to use practically, yet

[zifeo]

@barney-s is there something on the roadmap about this?