IoT with MQTT + Apache Kafka (Arduino + Raspberry Pi)

Motivation

Internet of Things always fascinated me because of the sheer no of people talking about it and the no of projects coming up related to it. So I thought I should give it a go myself and put up a small demonstration to learn the concepts of IoT projects.

This project is about implementing a IoT system to measure illumination inside of a house and act upon when there is low illumination by illuminating a light.

Design

Architecture

Arduino + Photoresistor Schematic

Raspberry Pi + LED Schematic

Implementation Decisions

I could have easily connected the Arduino and Raspberry Pi directly and get the thing done but I wanted to learn the correct and hard way to do as that is the whole purpose of this project. After some research on the internet I found that the best way to connect CPU and Memory constraint devices to IoT system is by using a protocol called MQTT. MQTT is a pub/sub protocol with a very small (2 byte) header which is ideal for small devices such as Arduino and reduces bandwidth usage when sending data across networks. 

But my aim was not fulfilled with this, what I wanted to implement is a system where more and more devices can be connected both as publishers of parameters (Illumination, Temperature, Humidity, etc.) and as consumers of those parameters in a scalable, decoupled manner. So after some more research I found that there are messaging queues used to enable this decoupling and scalability concerns. During my research I found that there is a upcoming tool named Apache Kafka which is capable of delivering what I am expecting in a more secure fault tolerant manner, so I decided to have it a go and see.

Implementation

Hardware

1. 1 Arduino UNO Development Board
2. 1 Raspberry Pi Model B
3. 1 Ethernet Shield
4. 1 Wifi Adaptor (Dongle)
5. 1 Photoresistor 
6. 1 LED
7. 1 Resistors
8. 1 Wireless Router with LAN Port

Software

All Software used in this project is available at https://github.com/shazin/iot

Prerequisites

Apache Kafka must be downloaded and configured to be started on a PC/Laptop, MQTT Server mosquitto must also be running.

For Apache Kafka refer this url Kafka Getting Started
For Mosquitto refer this url Mosquitto

Demonstraton

The Terminal Tabs in the video have the following started;

1. Mosquitto Server 
2. Zookeeper Server for Apache Kafka
3. Apache Kafka Server
4. Apache Kafka Consumer consuming on topic illumination
5. SSH session to raspberry pi which is running illumination.py python script

And the Spring Source IDE is running Mqtt Kafka Bridge.

When the Photoresistor in Arduino is cover the illumination parameter drops below 300 which will trigger the LED in Raspberry Pi to illuminate.

Lessons Learnt

The approach I tried is very responsive and is easy to scale because of the decoupling I have done by using Mosquitto and Apache Kafka. I didn't see any lags in Raspberry Pi consumer or Arduino Producer. This is may be required for System with lower latency SLAs. 

Future Work

As this is my first attempt at IoT systems, I have left out security and device identification aside. But these are highly required in a fully fledged IoT system. Planning to work on them in the future.

References

1. Samza in LinkedIn: How LinkedIn Processes Billions of Events Everyday in Real-time - Neha Narkhede
2. Federated Identity for IoT with OAuth - Paul Fremantle
3. Python Kafka
4. Kafka Java API 
5. MQTT Java API
6. Arduino Pub/Sub Client

Trackbacks/Pings

1. InfoQ China under Technology Point 2 : http://www.infoq.com/cn/news/2016/06/iot-weekly-46