Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

One framework to develop, deploy and operate data workflows with Python and SQL.

Data pipeline performing ETL to AWS Redshift using Spark, orchestrated with Apache Airflow

Project demonstrating how to automate Prefect 2.0 deployments to AWS ECS Fargate

Code examples showing flow deployment to various types of infrastructure

Classwork projects and home works done through Udacity data engineering nano degree

Let your pipe lines flow thru the Python code in xonsh.

Deploy a Prefect flow to serverless AWS Lambda function

Apache Spark Guide

F1 Data Pipeline

Analysis of 311 Service Requests for the City of NYC (from 2010 to 2023) Tech: Prefect cloud, dbt core, BigQuery, Compute Engine, CloudRun, Artifact Registry, Terraform, Docker

ETL pipeline combined with supervised learning and grid search to classify text messages sent during a disaster event

Learning from multiple companies in Silicon Valley. Netflix, Facebook, Google, Startups

A end-to-end real-time stock market data pipeline with Python, AWS EC2, Apache Kafka, and Cassandra Data is processed on AWS EC2 with Apache Kafka and stored in a local Cassandra database.

Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database

Challenge to job: Data Scientist

A batch Data Pipeline that retrieves data from a user purchase table and a movie review table and is transformed to form a user behaviour metric table.

Marshmallow serializer integration with pyspark

A data engineering pipeline for digital marketers.