Skip to content

πŸŒπŸ“ˆ Unlock IoT potential with this Event Streaming Architecture: Apache Kafka and MQTT power real-time data handling. Visualize, analyze, and scale effortlessly. πŸš€πŸ’»πŸ“Š

License

Notifications You must be signed in to change notification settings

sergio11/iot_event_streaming_architecture

Repository files navigation

QuantumFlow IoT: Smart IoT Event Streaming. Scaling with Apache Kafka and MQTT πŸŒπŸ“Š

Unlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. πŸš€

Ingest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. πŸŒ‘οΈπŸ’§πŸ“ˆ

The technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. πŸ’»πŸ“ˆπŸ”

With Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. πŸ³πŸš€πŸ“Š

Check out the detailed Medium article for a comprehensive overview. πŸ“–

Thank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. πŸŒπŸ“ˆπŸ’‘

Project developed to practice what I have learned in the Udemy course Apache Kafka Series - Kafka Connect Hands-on Learning and Apache Kafka Series - Kafka Streams for Data Processing.

Architecture Overview

The main goals of this architecture are the following:

  • Ingest and store real-time data from IoT sensors located in various locations.
  • Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.
  • Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.
  • Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.

Used technology

  • Spring Boot 2.3.3 / Apache Maven 3.6.3.
  • Spring Boot Starter Actuator.
  • Kafka Streams.
  • Spring Kafka.
  • Micrometer Registry Prometheus.
  • Eclipse Paho MQTT Client.
  • Kafka Connect.
  • Kafka Rest Proxy
  • lombok.
  • Jackson.
  • NodeExporter (Exporter for machine metrics).
  • Prometheus.
  • Grafana.
  • Eclipse Mosquitto.
  • MongoDB.
  • Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).
  • Cadvisor (Analyzes resource usage and performance characteristics of running containers).
  • kafka-exporter (Kafka exporter for Prometheus).

Running Applications as Docker containers.

Rake Tasks

The available tasks are detailed below (rake --task)

Task Description
check_deployment_file_task Check Deployment File
check_docker_task Check Docker and Docker Compose Task
cleaning_environment_task Cleaning Evironment Task
deploy Deploys the IoT Event Streaming Architecture and laun...
login Authenticating with existing credentials
start Start Containers
status Status Containers
stop Stop Containers
undeploy UnDeploy IoT Event Streaming Architecture

To start the platform make sure you have Ruby installed, go to the root directory of the project and run the rake deploy task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.

Containers Ports

Container Port
kafka-topics-ui localhost:8081
kafka-connect-ui localhost:8082
zoonavigator-web localhost:8083
mongo-express localhost:8084
grafana localhost:8085
prometheus localhost:8086
kafka-rest-proxy localhost:9999

Some screenshots

Deploy with Docker Compose.

Viewing topics through Landoop Kafka Topics UI

Viewing Connect Topology through Landoop Kafka Connect UI

Viewing Zookeeper Nodes through ZooNavigator

Viewing information consolidated and processed in MongoDB through Mongo Express.

Viewing the metrics of the IoT sensors simulated on the platform.

Viewing metrics about Kafka's performance.

Viewing platform container metrics.

Visitors Count

Please Share & Star the repository to keep me motivated.