An ETL Data Pipelines Project that uses AirFlow DAGs to extract employees' data from PostgreSQL Schemas, load it in AWS Data Lake, Transform it with Python script, and Finally load it into SnowFlake Data warehouse using SCD type 2.

ETL Pipeline for Twitter Data using Apache Airflow

ETL Pipeline (postgres, bigquery, csv, json, google storage)

Keywords: Python, Airflow, AWS, S3, Redshift, ETL

Python ETL Data Pipeline with AWS Glue and Athena

This repository showcases an end-to-end data engineering pipeline built with Airflow and Docker. The pipeline extracts data from a .tsv file, performs transformation operations, and loads the transformed data into a CSV file.

extract transform and load and transfrom

Udacity project within the Data Engineer Nanodegree

Implementation ETL with Python for data integration workflows.

We Build an ETL pipeline using Airflow that accomplishes the following: Downloads data from an AWS S3 bucket, Runs a Spark/Spark SQL job on the downloaded data producing a cleaned-up dataset of delivery deadline missing orders and then Upload the cleaned-up dataset back to the same S3 bucket in a folder primed for higher level analytics

ETL using application streaming and creating a Data Lake

Using all embedded tools & database SQLite+ DuckDb+ Schedule lib to build a data system for local.

Data Pipelines with Airflow

Building an ETL Pipeline using Airflow

Ruby client library for accessing google analytics feeds

Script interpreter to small ETL tasks

mvn, servlet 3.0, hadoop 1.0.2, cassandra 2.0

Takes geo shp files and migrates them up to the Socrata Geo API

This is a repo that holds a working example of multi-thread programming in python 2.7

db etl