Artie Transfer

⚡️ Blazing fast data replication between OLTP and OLAP databases ⚡️

Learn more »

Artie Transfer is a real-time data replication solution for databases and data warehouses/data lakes.

Typical ETL solutions rely on batched processes or schedulers (i.e. DAGs, Airflow), which means the data in the downstream data warehouse is often several hours to days old. This problem is exacerbated as data volumes grow, as batched processes take increasingly longer to run.

Artie leverages change data capture (CDC) and stream processing to perform data syncs in a more efficient way, which enables sub-minute latency.

Benefits of Artie Transfer:

• Sub-minute data latency: always have access to live production data.
• Ease of use: just set up a simple configuration file, and you're good to go!
• Automatic table creation and schema detection: Artie infers schemas and automatically merges changes to downstream destinations.
• Reliability: Artie has automatic retries and processing is idempotent.
• Scalability: handle anywhere from 1GB to 100+ TB of data.
• Monitoring: built-in error reporting along with rich telemetry statistics.

Take a look at this guide to get started!

Architecture

Pre-requisites

As you can see from the architecture diagram above, Artie Transfer is a Kafka consumer and expects CDC messages to be in a particular format.

The optimal set-up looks something like this:

• Debezium or Artie Reader depending on the source
• Kafka
  • One Kafka topic per table, such that you can toggle the number of partitions based on throughput.
  • The partition key should be the primary key for the table to avoid out-of-order writes at the row level.

Please see the supported section on what sources and destinations are supported.

Examples

To run Artie Transfer's stack locally, please refer to the examples folder.

Getting started

Getting started guide

What is currently supported?

Transfer is aiming to provide coverage across all OLTPs and OLAPs databases. Currently Transfer supports:

• Message Queues

  • Kafka (default)
  • Google Pub/Sub

• Destinations:

  • Snowflake
  • BigQuery
  • Redshift
  • Microsoft SQL Server
  • S3

• Sources:

  • MongoDB
  • DocumentDB
  • PostgreSQL
  • MySQL
  • DynamoDB

If the database you are using is not on the list, feel free to file for a feature request.

Configuration File

• Artie Transfer configuration file guide
• Examples of configuration files

Telemetry

Artie Transfer's telemetry guide

Tests

Transfer is written in Go and uses counterfeiter to mock. To run the tests, run the following commands:

make generate
make test

Release

Artie Transfer is released through GoReleaser, and we use it to cross-compile our binaries on the releases as well as our Dockerhub. If your operating system or architecture is not supported, please file a feature request!

License

Artie Transfer is licensed under ELv2. Please see the LICENSE file for additional information. If you have any licensing questions please email hi@artie.so.