Real-Time Streaming ETL Pipeline / Streaming Analytics: Real-Time Data Insights

In this project we will create a simple streaming data pipeline to continuously capture the changes in a MySQL database and replicate them in near real-time into ClickHouse.

_{Built with ❤︎ by Anantha Raju C and contributors}

Explore the docs »

Report Bug · Request Feature

Service	Badge	Badge	Badge	Badge	Badge
GitHub
GitHub

Details

• Technology stack
• Installation
• 01 - Project Setup
• 02 - Initial MySQL Preparation
• 03 - Debezium CDC Configuration
• 04 - ksqlDB Streams
• ksqlDB Tables(Optional)
• 05 - ClickHouse
• 06 - dbt (Optional)
• Reference

Reporting Issues/Suggest Improvements

This Project uses GitHub's integrated issue tracking system to record bugs and feature requests. If you want to raise an issue, please follow the recommendations below:

• Before you log a bug, please search the issue tracker to see if someone has already reported the problem.
• If the issue doesn't already exist, create a new issue
• Please provide as much information as possible with the issue report.
• If you need to paste code, or include a stack trace use Markdown +++```+++ escapes before and after your text.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

Kindly refer to CONTRIBUTING.md for important Pull Request Process details

1. In the top-right corner of this page, click Fork.

2. Clone a copy of your fork on your local, replacing YOUR-USERNAME with your Github username.

   git clone https://github.com/YOUR-USERNAME/Streaming_ETL_pipeline_MySQL.git

3. Create a branch:

   git checkout -b <my-new-feature-or-fix>

4. Make necessary changes and commit those changes:

   git add .

   git commit -m "new feature or fix"

5. Push changes, replacing <add-your-branch-name> with the name of the branch you created earlier at step #3. :

   git push origin <add-your-branch-name>

6. Submit your changes for review. Go to your repository on GitHub, you'll see a Compare & pull request button. Click on that button. Now submit the pull request.

That's it! Soon I'll be merging your changes into the master branch of this project. You will get a notification email once the changes have been merged. Thank you for your contribution.

Kindly follow Conventional Commits to create an explicit commit history. Kindly prefix the commit message with one of the following type's.

build : Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
ci : Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
docs : Documentation only changes
feat : A new feature
fix : A bug fix
perf : A code change that improves performance
refactor: A code change that neither fixes a bug nor adds a feature
style : Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
test : Adding missing tests or correcting existing tests

License

Distributed under the MIT License. See LICENSE.md for more information.

The End

In the end, I hope you enjoyed the application and find it useful, as I did when I was developing it to learn about Streaming ETL pipeline/ infrastructures for real-time data processing and analytics.

If you would like to enhance, please:

• Open PRs,

• Give feedback,

• Add new suggestions, and

• Finally, give it a 🌟.

• Happy Coding ...* 🙂

Contact

Anantha Raju C - @anantharajuc - arcswdev@gmail.com

Project Link: https://github.com/AnanthaRajuC/Streaming_ETL_pipeline_MySQL