This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!
-
Updated
May 22, 2024 - Python
This repository includes data engineering projects using Apache Airflow. I hope to add more projects using different technologies soon!
Airflow pipeline to finetune LLM on Kubernetes
Simple python script for easy local airflow deployment with docker. Packed with additional components. Will be adding more going forward.
Sync DAG changes from Git to Airflow
MLOps, haciendo un ETL sencillo usando Docker y Airflow y Google Cloud
An open-source project dedicated to constructing robust data pipelines and scalable software infrastructure. We leverage industry-standard tools favored by developers to enhance efficiency and reliability. Uniquely, these pipelines are field-tested on farms across Sumatra, Indonesia, ensuring real-world applicability and resilience.
Apache Airflow For Data Engineers Tutorial
🎵 LyricWave - Your AI Music Composer 🎶 Compose Unique MP4 Songs Effortlessly! LyricWave uses AI to create personalized music by harmonizing lyrics with captivating melodies and synthetic vocals. Unleash your musical creativity today! 🚀🎶
ETL (Extract, Transform, Load) pipeline to integrate sales data from various sources into a central data warehouse
Welcome to my Apache Airflow learning journey repository! 🚀 This repository serves as a comprehensive documentation of my exploration and understanding of Apache Airflow, an open-source platform for orchestrating complex workflows.
Source code of the Apache Airflow Tutorial for Beginners on YouTube Channel Coder2j (https://www.youtube.com/c/coder2j)
En este repositorio se encuentran algunos dags realizados siguiendo el entrenamiento de data engineer con el fin de aprender y practicar airflow
Apache airflow packed in docker compose
Automating Data Scrapers With Python and Airflow
This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.
An end-to-end pipeline that ingests raw data from CSV files through Airflow DAGS into BigQuery. From there, it uses dbt to normalize and clean the data and afterwards to make the transformations and come up with relevan reports.
GlassdoorETL automates ETL job for company data from Glassdoor into a PostgreSQL database. Utilizing Airflow and Docker, it ensures timely updates and consistency. A flexible tool for data engineers, it provides easy deployment and management for insightful perspectives from Glassdoor data.
Automated Indeed Job Offer Scraper: Airflow Orchestrated and Scheduled, Data Loaded into PostgreSQL Database
Add a description, image, and links to the airflow-docker topic page so that developers can more easily learn about it.
To associate your repository with the airflow-docker topic, visit your repo's landing page and select "manage topics."