DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.

Proceso ETL

The ADF Universal Framework is an open-source project designed to provide a comprehensive and flexible solution for building scalable and efficient data integration workflows using Azure Data Factory (ADF).

This project leverages Azure Cloud services like Azure Data Factory, Azure Databricks, and Synapse Analytics to execute a data engineering workflow. Utilizing data sourced from the Olympic API on GitHub, it involves extracting raw data into Azure Data Lake Storage, transforming it with PySpark on Azure Databricks, and analyzing the transformed data

O objetivo deste projeto foi aplicar os conhecimentos nas ferramentas de extração e tratamento de dados da plataforma Azure.

Generic Pipelines / Templates for Data Factory / Synapse Pipelines w.r.t Different MSFT Offering Integrations / Use Cases

Explore the Tokyo Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted advanced analytics in Azure Synapse, and visualized insights in Synapse or Power BI.

Tokyo Olympic Azure Data Engineering Project

DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.

Short guide as to how to set up the ADF Project :)

This project showcase the following: 1. REST Connection to CHAT GPT 2. Logging system to Azure file system, storage app and insights. 3. Azure Data Factory Rest implementation using MS Graph client.

Terraform script to deploy almost all Azure Data Services

End to end data reporting project using Azure services like Azure Data Factory for data orchestration, Azure Synapse Analytics for data warehousing, Databricks for data transformations, and Power BI for intuitive data visualization and reporting.

Threat Detection and Visualization

Azure Data Landing Zone - IAC , Config, Code

Datas Engineer Exploration of the 2021 Tokyo Olympic data using a variety of tools and technologies, with a primary focus on Azure services, Leveraging Azure Data Factory, Data Lake Gen 2, Synapse Analytics, and Azure Databricks to perform data engineering tasks, analysis, and visualization.

Data engineering with Azure services

This is to demo how to use Azure Data Factory with AKS/KEDA to run batch jobs in Azure

End-to-End data engineering project with Azure Databricks as cloud service and Tokyo olympic data