Ceph is a distributed object, block, and file storage platform

DBM，数据库管理

A distributed storage benchmark for file systems, object stores & block devices with support for GPUs

The Universal Storage Engine

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

Scalable data processing pipelines in JavaScript

Hadoop FSImage Analyzer (HFSA)

Exports Hadoop HDFS content statistics to Prometheus

25+ DevOps CLI Tools - Anonymizer, SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles), Hadoop HDFS & Hive tools, Solr/SolrCloud CLI, Nginx stats & HTTP(S) URL watchers for load-balanced web farms, Linux tools etc.

Storage Benchmark Kit

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

WASP is a framework to build complex real time big data applications. It relies on a kind of Kappa/Lambda architecture mainly leveraging Kafka and Spark. If you need to ingest huge amount of heterogeneous data and analyze them through complex pipelines, this is the framework for you.

Forex Data Pipeline

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Analysis scripts for log data sets used in anomaly detection.

Storage connector for Trino

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.

Python interface to the TileDB storage engine