IoT Sensor Analytics with Apache Kafka, KSQL, TensorFlow and MQTT => Kafka-Native End-to-End IoT Data Integration and Processing. Large numbers of IoT devices lead to big data and the need for further processing and analysis. Apache Kafka is a highly scalable and distributed open source streaming platform, which can connect to MQTT and other IoT standards. Kafka ingests, stores, processes and forwards high volumes of data from thousands of IoT devices. The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is the streaming SQL engine on top of Apache Kafka which simplifies all this and make stream processing available to everyone without the need to write source code. This talk shows how to leverage Kafka and KSQL in an IoT sensor analytics scenario for predictive maintenance and integration with real time monitoring systems. A live demo shows how to embed and deploy Machine Learning models - built with frameworks like TensorFlow, DeepLearning4J or H2O - into mission-critical and scalable real time applications.

1. 1Confidential
IoT Sensor Analytics with
Apache Kafka, KSQL, TensorFlow and MQTT
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Kafka-Native End-to-End IoT Data Integration and Processing

2. 3
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

3. 4
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

4. 6
Connected Intelligence (Cars, Machines, Robots, …)

5. 7
Smart Cities

6. 8
Smart Retail and Customer 360

7. 9
Intelligent Applications (Early Part Scrapping, Predictive Maintenance, …)

8. 10
?
Architecture (High Level)
Kafka BrokerKafka BrokerStreaming
Platform
Connect
w/ MQTT
connector
GatewayDevicesDevicesDevicesDevice
Device Tracking
(Real Time)
Predictive
Maintenance
(Near Real Time)
Log Analytics
(Batch)
Edge Data Center / Cloud
How to integrate?

9. 13
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

10. 14
MQTT - Publish / subscribe messaging protocol
• Built on top of TCP/IP for constrained devices and unreliable networks
• Many (open source) broker implementations
• Many client libraries
• IoT-specific features for bad network / connectivity
• Widely used (mostly IoT, but also web and mobile apps via MQTT over WebSockets)

11. 17
MQTT Architecture (large scale)
Load
Balancer
MQTT
Server 1
MQTT
Server 2
MQTT
Server 3
MQTT
Server 4
topic: [deviceid]/car
...
Processor
1
Processor
2
Processor
3
Processor
4

12. 18
MQTT Trade-Offs
Pros
• Lightweight
• Simple API
• Built for poor connectivity / high latency scenario
• Many client connections (tens of thousands per MQTT server)
Cons
• Queuing, not stream processing
• Can’t handle usage surges (no buffering)
• No high scalability (true for most MQTT brokers)
• Very asynchronous processing (often offline for long time)
• No good integration to the rest of the enterprise
• No reprocessing of events

13. 19
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
4) End-to-End IoT Integration Architecture(s)
5) IoT Data Processing
6) Live Demo: End-to-End Sensor Analytics

14. 20
Apache Kafka – The Rise of a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

15. 21
Log and Pub/Sub

16. 23
Apache Kafka == Distributed Commit Log with Replication

17. 25
Apache Kafka at Scale
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921
https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture
(2018)
(2018)

18. 26
Kafka Trade-Offs (from IoT perspective)
Pros
• Stream processing, not just queuing
• High throughput
• Large scale
• High availability
• Long term storage and buffering
• Reprocessing of events
• Good integration to the rest of the enterprise
Cons
• Not built for tens of thousands connections
• Requires stable network and good infrastructure
• No IoT-specific features like keep alive, last will or testament

19. 27
(De facto) Standards for Processing IoT Data
A Match Made In Heaven
+ =

20. 28
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

21. 29
TensorFlow
TensorFlow is an open source software library for high
performance numerical computation. Its flexible architecture
allows easy deployment of computation across a variety of
platforms (CPUs, GPUs, TPUs), and from desktops to clusters of
servers to mobile and edge devices. Originally developed by
researchers and engineers from the Google Brain team within
Google’s AI organization, it comes with strong support for
machine learning and deep learning and the flexible
numerical computation core is used across many other scientific
domains.
https://www.tensorflow.org/

22. 30
The First Analytic Models
How to deploy the models
in production?
…real-time processing?
…at scale?
…24/7 zero downtime?

23. 31
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

24. 32
Apache Kafka’s Open Source Ecosystem as Infrastructure for ML

25. 33
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go/.NET /Python
Kafka Producer
KSQL
Kafka
Streams

26. 37
Replayability — a log never forgets!
Time
Model B Model XModel A
Producer
Distributed Commit Log
Different models with same data
Different ML frameworks
AutoML compatible
A/B testing
Google Cloud Storage HDFS

27. 38
Analytic Model (Autoencoder for Anomaly Detection)

28. 39
Model Deployment #1: RPC Communication to do Model Inference
Streams
Input Event
Prediction
Request
Response
Model Serving
TensorFlow Serving
gRPC

29. 40
Model deployment #2: Model interference natively in the App
Streams
Input Event
Prediction

30. 41
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

31. 42
?
Architecture (High Level)
Kafka BrokerKafka BrokerStreaming
Platform
Connect
w/ MQTT
connector
GatewayDevicesDevicesDevicesDevice
Device Tracking
(Real Time)
Predictive
Maintenance
(Near Real Time)
Log Analytics
(Batch)
Edge Data Center / Cloud
How to integrate?

32. 43
?
Architecture (High Level) – Machine Learning Perspective
Kafka BrokerKafka BrokerStreaming
Platform
Connect
w/ MQTT
connector
GatewayDevicesDevicesDevicesDevice
Edge Analytics
Real Time
Model Serving
Predictive
Maintenance
Near Real Time
Model Serving
Model Training
(Batch)
Edge Data Center / Cloud

33. 46
Kafka-Native Integration Options between MQTT and Apache Kafka
Kafka Connect
MQTT Proxy
REST Proxy

34. 47
Kafka-Native Integration Options between MQTT and Apache Kafka
Kafka Connect
MQTT Proxy
REST Proxy

35. 49
?
Integration with Kafka Connect (Source and Sink)
Kafka BrokerKafka BrokerKafka Broker
MQTT
Broker
Connect
w/ MQTT
connector
Connect
w/ MQTT
connectorMQTT
DevicesDevicesDevicesDevice
Kafka
Consumer
MQTT Broker
Persistent + offers MQTT-specific features
Consumes push data from IoT devices
Kafka Connect
Kafka Consumer + Kafka Producer under the hood
Pull-based (at own pace, without overwhelming the source or getting overwhelmed by the source)
Out-of-the-box scalability and integration features (like connectors, converters, SMTs)
?
Connect
w/ MQTT
connector
Connect
w/ MQTT
connector

36. 56
Kafka-Native Integration Options between MQTT and Apache Kafka
Kafka Connect
MQTT Proxy
REST Proxy

37. 57
MQTT Proxy
Kafka BrokerKafka BrokerKafka Broker
MQTT
ProxyMQTT
DevicesDevicesDevicesDevices
Kafka
Consumer
MQTT Proxy
MQTT is push-based
Horizontally scalable
Consumes push data from IoT devices and forwards it to Kafka Broker at low-latency
Kafka Producer under the hood
No MQTT Broker needed
Kafka Broker
Source of truth
Responsible for persistence, high availability, reliability

38. 59
Kafka-Native Integration Options between MQTT and Apache Kafka
Kafka Connect
MQTT Proxy
REST Proxy

39. 60
Confluent REST Proxy
REST Proxy
IoT Applicatons
Native Kafka
Applications
(Java, C, Go, …)
REST / HTTP(S)
TCP
The „simple alternative“ for IoT
• Simple and understood
• HTTP(S) Proxy à Push-based
• Security ”easier”
• Scalable with standard load balancer
(still synchronous HTTP)
• Not for very high throughput
• Implement Kafka Connect features in
your client app

40. 62
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

41. 6363
Processing Options for MQTT Data with Apache Kafka
Streams
Kafka native vs. additional big data cluster and technology
(or others, you name it …)

42. 6464
IoT Data Processing
Kafka Client
Batch
System
AnalyticsKafka Cluster Kafka Connect
Kafka Streams
/
KSQL
MQTT Device
Kafka Ecosystem
Other Components
Real Time
System
All Data
Alerting
Process
Data
Continuously
Forward
Processed
Data
On premise DC / CloudAt the edge

43. 6868
KSQL – Continuous Queries for Streaming ETL / Anomaly Detection
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTES)
GROUP BY card_number
HAVING count(*) > 3;

44. 6969
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

45. 7070
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge
5858
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge

46. 7171
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras

47. 7272
Model Deployment with Apache Kafka, KSQL and TensorFlow
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)

48. 73
Live Demo
End-to-End Sensor Analytics…
Python, Jupyter Notebook, TensorFlow, Keras, Apache Kafka, KSQL and MQTT

49. 74
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras

50. 75
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot

51. 77
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Please contact me!
Come to our booth
to find out more about
Kafka and Confluent