Sri Lanka's First and Only Platform for Luxury Houses and Apartment for Sale, Rent

Tuesday, December 23, 2014

IoT with MQTT + Apache Kafka (Arduino + Raspberry Pi)


Internet of Things always fascinated me because of the sheer no of people talking about it and the no of projects coming up related to it. So I thought I should give it a go myself and put up a small demonstration to learn the concepts of IoT projects.

This project is about implementing a IoT system to measure illumination inside of a house and act upon when there is low illumination by illuminating a light.



Arduino + Photoresistor Schematic

Raspberry Pi + LED Schematic

Implementation Decisions

I could have easily connected the Arduino and Raspberry Pi directly and get the thing done but I wanted to learn the correct and hard way to do as that is the whole purpose of this project. After some research on the internet I found that the best way to connect CPU and Memory constraint devices to IoT system is by using a protocol called MQTT. MQTT is a pub/sub protocol with a very small (2 byte) header which is ideal for small devices such as Arduino and reduces bandwidth usage when sending data across networks. 

But my aim was not fulfilled with this, what I wanted to implement is a system where more and more devices can be connected both as publishers of parameters (Illumination, Temperature, Humidity, etc.) and as consumers of those parameters in a scalable, decoupled manner. So after some more research I found that there are messaging queues used to enable this decoupling and scalability concerns. During my research I found that there is a upcoming tool named Apache Kafka which is capable of delivering what I am expecting in a more secure fault tolerant manner, so I decided to have it a go and see.


  1. 1 Arduino UNO Development Board
  2. 1 Raspberry Pi Model B
  3. 1 Ethernet Shield
  4. 1 Wifi Adaptor (Dongle)
  5. 1 Photoresistor 
  6. 1 LED
  7. 1 Resistors
  8. 1 Wireless Router with LAN Port

All Software used in this project is available at


Apache Kafka must be downloaded and configured to be started on a PC/Laptop, MQTT Server mosquitto must also be running.

For Apache Kafka refer this url Kafka Getting Started
For Mosquitto refer this url Mosquitto


The Terminal Tabs in the video have the following started;

  1. Mosquitto Server 
  2. Zookeeper Server for Apache Kafka
  3. Apache Kafka Server
  4. Apache Kafka Consumer consuming on topic illumination
  5. SSH session to raspberry pi which is running python script
And the Spring Source IDE is running Mqtt Kafka Bridge.

When the Photoresistor in Arduino is cover the illumination parameter drops below 300 which will trigger the LED in Raspberry Pi to illuminate.

Lessons Learnt

The approach I tried is very responsive and is easy to scale because of the decoupling I have done by using Mosquitto and Apache Kafka. I didn't see any lags in Raspberry Pi consumer or Arduino Producer. This is may be required for System with lower latency SLAs. 

Future Work

As this is my first attempt at IoT systems, I have left out security and device identification aside. But these are highly required in a fully fledged IoT system. Planning to work on them in the future.

  1. Samza in LinkedIn: How LinkedIn Processes Billions of Events Everyday in Real-time - Neha Narkhede
  2. Federated Identity for IoT with OAuth - Paul Fremantle
  3. Python Kafka
  4. Kafka Java API 
  5. MQTT Java API
  6. Arduino Pub/Sub Client
  1. InfoQ China under Technology Point 2 :


Narendran said...

Wonderful POC

Narendran said...

Can you please justify why Kafka is required? I could understand scaling will be an issue. But why not Mosquitto Implementation does not thinks about it. What will happen when a large amount of data load is put upon Mosquitto receiver end?

Unknown said...

Nice POc Can you suggest me the jar files used for Kafka Bridge code especially for java.util.Properties

Thank you can mail me links at