1. Real World Applications of MQTT
Manoj Gudi
Ex-R.A. IIT Bombay — CTO — Focus Analytics
27th August 2017
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 1 / 45
2. Content & Scope
What is MQTT?
MQTT Characteristics and Features
Who are we and what do we do?
Real world use for indoor positioning- I
Real world use for indoor positioning- II
Code samples
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 2 / 45
3. What is MQTT..
MQ Telemetry Transport: lightweight pub/sub messaging
protocol
Designed for constrained devices (low bandwidth — high
latency — unreliable networks)
Built to minimize network bw and device resource
requirements
Offers various reliability levels of message delivery
MQTT v3.1.1 is OASIS standard (2014)
Initially developed by IBM and Eurotech for M2M
communication for embedded devices
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 3 / 45
4. ..What is MQTT
Pub/Sub model
Publishers: sensors like temperature sensors
Brokers
Open source: Mosquitto(Eclipse Project), EMQ(Apache v2
license)
Enterprise services: HiveMQ, Azure IoT Hub, IBM Websphere
etc.
Subscribers: entities which consume data, mobile phones etc.
A MQTT client can be publisher or a subscriber or both
Topics allow clients to exchange information with definedManoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 4 / 45
5. Why MQTT?
Designed App with killer features
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 5 / 45
6. Why MQTT?
By the time it hits production
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 6 / 45
7. MQTT Characteristics
Network model:
Lightweight: 2 byte header overhead per packet
Three main components: data producer(publisher) — data
consumer (subscriber) — queue like structure (topics)
MQTT requires TCP/IP Stack (i.e. way to deliver packets in
order reliably). Can’t be used over UDP.
MQTT-SN for sensor networks, doesn’t require TCP/IP stack.
Works directly over transport layer (like ZigBee)
Suitable for asynchronous communication model with events
(messages)
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 7 / 45
8. MQTT Message format..
Message format:
2 byte fixed-length header
Optional message-specific variable length header and message
payload
Very lean message format. Tries to save and reuse bytes as
much as possible.
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 8 / 45
9. ..MQTT Message format..
Message Type (Description/Values):
0: Reserved 5: PUBREL 10: UNSUBSCRIBE
1: CONNECT 6: PUBCOMP 11: UNSUBACK
2: PUBLISH 7: SUBSCRIBE 12: PINGREQ
3: PUBACK 8: SUBACK 13: PINGRESP
4: PUBREC 9: UNSUBSCRIBE 14: DISCONNECT
15: RESERVED
DUP: Duplicate flag(==1) to indicate re-delivery of an earlier
PUBLISH packet(as of 3.1.1)
QoS Level: Possible values are 0, 1, 2
Remaining Length: encodes(length(variable length header) +
length(payload))
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 9 / 45
10. ..MQTT Message format..
RETAIN:
For a PUBLISH message instructing server to keep the
message for this topic
Last known good value.
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 10 / 45
12. MQTT QoS levels..
MQTT provides 3 levels of QoS (Quality of Service)
Although uses TCP/IP, but data loss can still occur if network
is disrupted (lost messages)
Automatic retransmission and delivery guarantees for certain
QoS by MQTT
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 12 / 45
13. MQTT QoS 0
At most once delivery
Minimal guarantee. Receiver doesn’t acknowledge reception
of message
Sender doesn’t store message for redelivery
Same guarantee as underlying TCP protocol
Also has the least overhead in comparison
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 13 / 45
14. MQTT QoS 1
At least once delivery
QoS1 guarantees that a message will be delivered at least
once to receiver.
Duplicate messages maybe received (identified by DUP = 1)
Sender stores the message until it gets PUBACK
PUBLISH and PUBACK associated by packetId
Re-transmission of packet by sender if PUBACK not receivedManoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 14 / 45
15. MQTT QoS 2
Exactly once delivery
Highest QoS, guarantees each message is received only once
Slowest QoS because of two flows between sender-receiver.
Critical to your application to receive all messages exactly once
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 15 / 45
16. ..MQTT QoS levels
Net-Net:
QoS0: when you have stable connection, once a while packet
drop is OK.
QoS1: when you need to get every message and your app can
handle duplicates
QoS2: when it’s critical to your app to receive all messages
exactly once.
QoS levels defined when creating a connection by the client.
QoS Downgrade possible if subscribing client has lower QoS
then publishing client
What if publisher is QoS1 and subscriber is QoS2?
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 16 / 45
17. Message Persistence..
Non Persistent Session:
Defined by client to broker in CONNECT packet
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 17 / 45
19. ..Message Persistence
What’s stored in a session?
All messages in a QoS 1,2 flow
PUBACK while client was offline
All received QoS2 messages, which are not yet confirmed to
the client
Session identified uniquely to a client by clientID in
CONNECT
A client identifies a stored session by session flag in
CONNACK
Useful if client is in patchy networks and has to consume all
packets
Max number of messages usually limited by memory of
broker/client
Mosquitto can write persistent messages to file, reload when
broker restarts
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 19 / 45
20. Topics
A UTF-8 string used by broker to filter messages for each
client
Hierarchical. Can contain one or more topic levels
Supports wildcards
A ”+” is single-level wildcard
myhome/groundfloor/+/temperature
A ”#” is multilevel-level wildcard (Must be at end)
myhome/groundfloor/#
There can be null sub-topics myhome//kitchen/
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 20 / 45
21. Other Features
Last Will and Testament: used to notify other clients by
broker when a client is disconnected abruptly
MQTT supports TLS, but you lose low overhead benefits
For a long-living TCP connections with MQTT, TLS overhead
negligible
MQTT broker typically runs at 1883(TCP) and 8883(TLS)
Free test broker at https://test.mosquitto.org/
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 21 / 45
22. Who are we?
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 22 / 45
23. What do we do?
Outdoor: GPS — Indoors: ?
Directly lifted from marketing pitch deck
I didn’t make this
I don’t know why there is a magnifying glass
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 23 / 45
24. We work with
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 24 / 45
25. Indoor Positioning in a Large Warehouse..
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 25 / 45
26. ..Indoor Positioning in a Large Warehouse..
Challenges
Warehouse usually situated in very remote places
Metal mesh floors (Weak cellular network)
Operates 24X7
XXL: One misplaced item in inventory takes about a month to
find
Idea is to able to quickly trace people movements in realtime
and retrospectively.
We need to precisely position a warehouse worker up to a foot
level.
With users expecting faster delivery times, misplaced
inventory problem is critical
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 26 / 45
27. Solution # 1
Upside: Very fast and 110% accurate!
Downside: Very complicated and available only in Sirsa(now
in Rohtak)
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 27 / 45
28. ..Indoor Positioning in a Large Warehouse..
UWB (Ultra Wide Band) based positioning system:
Low energy, high-bandwidth communication over a large
spectrum(3.1 to 10.6 GHz)
Minimum 3 Anchors(known position) and 1 Tag(position to be
determined)
Leverages Time of Flight (TOF) method to come up with
position of tag
Accuracy 1 foot and less susceptible to interferenceManoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 28 / 45
29. ..Indoor Positioning in a Large Warehouse..
UWB (Ultra Wide Band) based positioning system:
Accuracy 1 foot and less susceptible to interference
Anchors can talk up to 60M distance in LOS, spread across the
entire warehouse section
Each worker carries a tag with their inventory tracking gadget
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 29 / 45
30. ..Indoor Positioning in a Large Warehouse
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 30 / 45
31. Indoor Positioning in Retail Spaces..
A simple android SDK. Integrate, and done.
We send you location of your users. On phone and if required
on your server
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 31 / 45
32. ..Indoor Positioning in Retail Spaces..
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 32 / 45
33. ..Indoor Positioning in Retail Spaces
Whenever a big client comes on board, with in 24 hrs 30-40%
of their users come on board as well.
A chunk of the location data comes when users are in non
retail places
What if there’s a way where other users coarsely predict other
users?
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 33 / 45
34. Small Confession
That network traffic graph you saw was of a VM, running
HAProxy
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 34 / 45
35. Location Prediction of Users in a distributed way..
This is in a prototype stage.
The idea is to leverage large number of mobile phones in same
area help predict each other while
Avg 140 smart phone per sq km in India. Density increases in
city.
Save us $$$ on network and server costs
It should work even when our prediction engine is down
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 35 / 45
36. ..Location Prediction of Users in a distributed way..
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 36 / 45
37. ..Location Prediction of Users in a distributed way..
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 37 / 45
38. ..Location Prediction of Users in a distributed way
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 38 / 45
39. Enough of talk, show me some code!
def getDataAndPubish (sensorData , client , ):
"""
Get sensor data and publish to broker
"""
topic = "client/warehouse/f1/s1"
sensorData["timestamp"] = int(time.time () * 1000)
client.publish(topic ,
json.dumps(sensorData), qos=1, retain=False)
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 39 / 45
40. Enough of talk, show me some code!
def onConnect(client , userData , msg):
’’’
subscribe when (CONNACK)
’’’
print("Successfully Connected")
client.subscribe("client/warehouse /#", qos =1)
def onMessage(client , userData , msg):
"""
call locationEngine main and get prediction
"""
anchorCoordinates = userData
print("Got Message ..")
data = json.loads(msg.payload.decode("utf -8")
### DO SECRET STUFF
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 40 / 45
41. Enough of talk, show me some code!
def main ():
anchorCoordinates = [[1, 12, 123], [32, 123, 32],
# Get MQTT connection paramers
brokerIP = "localhost"
brokerPort = 1883
defaultTimeout = 60
client = mqtt.Client ()
# Pass baseInstance and session tuple to callbacks
client.on_connect = onConnect
client.on_message = onMessage
# Set anchorCoordinates
client.user_data_set( anchorCoordinates )
client.connect(brokerIP , brokerPort , defaultTimeou
client.loop_forever ()
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 41 / 45
42. Some caveats
If implementing on MQTT on battery constrained devices,
select keep-alive intervals carefully
Too short keep-alive can impact battery profile because of
PINGREQ/PINGRESP
Although Mosquitto (MQTT broker) allows horizontal scaling
between two brokers by bridging. It can be tricky if you need
more brokers or a cluster with Mosquitto
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 42 / 45
43. References/Links
Eclipse Paho Project www.eclipse.org/paho
Hive MQTT Essentials
http://www.hivemq.com/mqtt-essentials/
IoT For Tiny Devices (MQTT-SN), Stefan Roth, Zuehlke
Power profiling MQTT on android:
http://stephendnicholas.com/posts/power-profiling-mqtt-on-
android
Mosquitto Bridging http://www.steves-internet-
guide.com/mosquitto-bridge-configuration/
An Introduction to MQTT, A Protocol for M2M and IoT
Applications. Peter R. Egli INDIGOO.COM
Mosquitto broker docs mosquitto.org
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 43 / 45
44. Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 44 / 45
45. This seems cool, can I join?
Sure!
manoj@getfocus.in — +91-9920909145
careers@getfocus.in
Thanks!
Manoj Gudi Ex-R.A. IIT Bombay — CTO — Focus Analytics
Real World MQTT 45 / 45