etl

Star

Here are 3,649 public repositories matching this topic...

apache / airflow

Star

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Updated Apr 27, 2024
Python

airbytehq / airbyte

Star

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Updated Apr 27, 2024
Python

apache / doris

Star

Apache Doris is an easy-to-use, high performance and unified analytics database.

bigquery real-time sql database spark hive hadoop etl snowflake olap query-engine redshift dbt elt iceberg hudi delta-lake lakehouse

Updated Apr 27, 2024
Java

dagster-io / dagster

Star

An orchestration platform for the development, production, and observation of data assets.

python metadata workflow data-science etl analytics scheduler orchestration data-engineering data-integration data-pipelines workflow-automation mlops dagster data-orchestrator

Updated Apr 27, 2024
Python

benthosdev / benthos

Star

Fancy stream processing made operationally mundane

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue data-engineering streaming-data stream-processor data-ops

Updated Apr 27, 2024
Go

mage-ai / mage-ai

Star

🧙 Build, run, and manage data pipelines for integrating and transforming data.

python data-science data machine-learning sql spark pipeline etl pipelines orchestration artificial-intelligence data-engineering data-integration dbt elt transformation data-pipelines reverse-etl

Updated Apr 27, 2024
Python

turbot / steampipe

Star

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Updated Apr 26, 2024
Go

kestra-io / kestra

Star

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

workflow data pipeline etl workflow-engine scheduler orchestration data-engineering data-integration elt data-pipeline data-quality low-code data-orchestration data-orchestrator reverse-etl

Updated Apr 26, 2024
Java

cloudquery / cloudquery

Star

The open source high performance ELT framework powered by Apache Arrow

Updated Apr 27, 2024
Go

apache / flink-cdc

Star

Flink CDC is a streaming data integration tool

mysql real-time kafka etl postgresql distributed batch data-integration schema-evolution elt flink cdc data-pipeline change-data-capture paimon

Updated Apr 27, 2024
Java

orchest / orchest

Star

Build data pipelines, the easy way 🛠️

python docker kubernetes data-science machine-learning airflow cloud deployment jupyter etl ide pipelines self-hosted jupyterlab notebooks data-pipelines dag etl-pipeline orchest

Updated Jun 6, 2023
TypeScript

rudderlabs / rudder-server

Star

Privacy and Security focused Segment-alternative, in Golang and React

Updated Apr 27, 2024
Go

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Updated Apr 24, 2024
Python

blockchain-etl / ethereum-etl

Star

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

export bigquery aws csv sql etl ethereum transaction gcp google-cloud erc20 erc20-tokens blockchain-analytics erc721

Updated Apr 11, 2024
Python

quadratichq / quadratic

Star

Quadratic | Data Science Spreadsheet with Python & SQL

python webgl data-science data sql etl spreadsheet wasm data-engineering data-analysis quadratic

Updated Apr 27, 2024
TypeScript

apache / incubator-devlake

Star

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.