This setups the infrastructure for Airflow, in Docker, as close as possible to a deploy in a Kubernetes/Helm environment: having containers for the airflow-scheduler, airflow-web, airflow-triggerer, and airflow-worker (with the CeleryExecutor)
1. Start setting up the infrastructure in Docker with:
Airflow with LocalExecutor:
docker compose up -dAirflow with CeleryExecutor:
docker compose -f docker-compose.celery.yml up -d2. Airflow WebUI can be accessed at:
open http://localhost:80803. Airflow DAGs:
To deploy Airflow DAGs, just move them inside the dags folder and Airflow should pick it up soon enough
- PEP-517: Packaging and dependency management with PDM
- Run Airflow DAGs on Docker
- Code format/lint with Ruff
- Complete Astronomer Academy's Airflow 101
- Deploy Airflow to Kubernetes with Helm
- Run/Deploy Airflow DAGs on Kubernetes with KubernetesPodOperator