Demostration on Architecture of IoT

1. EVENT DRIVEN STREAMING ANALYTICS
Demonstration on Architecture of IoT
Lei Xu 2016.09

2. 1.DEMO SCENARIO

3. Taxi/Devices
Coordinates &
Payment Info
Devices generate transactions after payment

4. Taxi/Devices
Data Flow
Coordinates &
Payment Info
IoT Gateway
Applications &
Services
Data Management
Cloud
Infrastructure
IoT Connectivity Platform
Online Analytics
• Dropoff-district Count during the
last 30 min
• Real-time Taxi Dropoff Events
• 10 Most Frequent Routes(Pickup-
area Dropoff-area) during the last
30 minutes
• 10 Most Profitable areas within the
last 15 minutes
Offline Analytics
• 10 Most Frequent Routes by week day
• 10 Most Profitable Areas by week day
• Most Profitable Hours at the 10 Most
Profitable Areas by week day
Transaction being sent to IoT Platform for Analytics
Thousands of Browsers Hundreds of BI Clients

5. Data Management
Multi-Channel Data Integration
Structured & Unstructured Data
Analytics & Machine Learning
Cloud Infrastructure
Cloud Native Platform
Scalability & High Availability
Security & Governance
Visualize Data
Reporting, BI & User Apps
History Data & Real-time Data
Multiple Devices & Platforms
IoT Gateway
Device Management
Authentication & Authorization
Gateways & Interfaces
Apps & Services
Rule Engine & Process Mgmt.
Complex Event Processing
Business & Data MicroServices
IoT Platform
Components

6. 2.REQUEST DRIVEN FLOW

7. Storing
Enriched Data
Time Based
Enrichment &
Filtering
Receiving &
Storing Raw Events
Time Based Batch
Aggregation
Storing
Aggregated Data
Sending Raw
Events
Polling Requests
for Latest Data
Taxi/Devices
Traditional Request Driven Flow
Event-driven Flow
Request-driven Flow
Time-based Flow
Analytics
BI Visualization

8. DRAWBACKS & CHALLENGES
• Polling + Request/Response Style => Overhead
• Time Based Batch Process => High Latency
• Architecture - Monolith or SOA will work well with HA?
• How to scale & how to maintain system performance with lots of user requests?
• How to scale & how to maintain system performance with huge amount of data?
• Portable for both Cloud and On-Premise environments?

9. REQUEST DRIVEN & EVENT DRIVEN
Request Driven Event Driven
How is an action being
taken?
As a response to a specific
request
Triggered by the fact of a
specific situation
When is an action being
taken?
When the request is being
processed
Determined by the context
of the situation
What happens when the
event/request occurs?
A response is produced
The event can be ignored,
increment the state, trigger
an internal derive event, or
trigger a solution
http://epthinking.blogspot.jp/2012/12/more-on-request-driven-vs-event-driven.html

10. 3.EVENT DRIVEN FLOW

11. Publishing/
Subscribing
Enriched Events
Enrichment
&
Filtering
Receiving Raw
Events
Real-time
Aggregation/
Analytics
Publishing/
Subscribing
Aggregated Events
Sending Raw
Events
Events
Visualization
Online Analytics
Taxi/Devices
Expectations
• Event Driven
• Low Latency
• Scalability
• High Availability
• Portability
• Light Weight
Event-driven Flow

12. Publishing/
Subscribing
Enriched Events
Enrichment
&
Filtering
Receiving Raw
Events
Real-time
Aggregation/
Analytics
Publishing/
Subscribing
Aggregated Events
Sending Raw
Events
Events
Visualization
Online & Offline Analytics
Storing Data for
Analytics
Analytics
BI Visualization
Taxi/Devices
Event-driven Flow
Request-driven Flow

13. Publishing/
Subscribing
Enriched Events
Enrichment
&
Filtering
Receiving Raw
Events
Real-time
Aggregation/
Analytics
Publishing/
Subscribing
Aggregated Events
Sending Raw
Events
Events
Visualization
Machine Learning Loopback to Streaming
Storing Data for
Analytics
Analytics &
Machine Learning
BI Visualization
Taxi/Devices
Event-driven Flow
Request-driven Flow

14. 4.EVENT DRIVEN ARCHITECTURE

15. Binding Abstraction Layer
Messaging Options
Data Pipeline
Data Microservices
Data
Spring Cloud Stream
App
Spring Cloud
Stream App
Spring Cloud
Stream App
Spring Cloud Stream/Data Flow as Event Driven Architecture
Data
Filter, Enrich, Transform, Aggregation, Rules, CEP, Analytics

16. Publishing/Subscribing
Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time
Aggregation/Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP
Source
MongoDB
Processor
GemFire
Sink
GemFire
Source
WebSocket
Sink
WebSocket
Sink
Reverse GeocodingReal-time
Aggregation/Analytics
Messaging
Apps & Services – Online Analytics

17. Publishing/Subscribing
Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time
Aggregation/Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP
Source
MongoDB
Processor
GemFire
Sink
GemFire
Source
WebSocket
Sink
WebSocket
Sink
Reverse GeocodingReal-time
Aggregation/Analytics
Messaging
BI Visualization
Analytics & Machine Learning
Storing Data for Analytics
HDFS
Sink
SQL Query
Apps & Services – Online & Offline Analytics
MPP SQL on HDFS

18. Publishing/Subscribing
Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time
Aggregation/Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP
Source
MongoDB
Processor
GemFire
Sink
GemFire
Source
WebSocket
Sink
WebSocket
Sink
Reverse GeocodingReal-time
Aggregation/Analytics
Messaging
BI Visualization
Analytics & Machine Learning
Storing Data for Analytics
MPP SQL on HDFS
HDFS
Sink
SQL Query
Machine
Learning
Apps & Services – Machine Learning Loopback to Streaming
Machine Learning

19. 5.ONLINE ANALYTICS
LOCAL DEPLOYMENT

20. MongoDB
Processor
Real-time
Aggregation/Analytics
Spring Boot
App
Spring Cloud
Stream App
Spring Cloud
Stream App
HTTP
Source
GemFire
Sink
GemFire
Source
WebSocket
Sink
Simulator
Spring Cloud
Stream App
Spring Cloud
Stream App
Spring Cloud
Stream App
Java CLI
App
Messaging
WebSocket
Sink
Spring Cloud
Stream App
MVW App
Data Flow
Deployment
Deployment
Reverse Geocoding

21. MongoDB
Processor
Real-time
Aggregation/Analytics
Spring Boot
App
Spring Cloud
Stream App
Spring Cloud
Stream App
HTTP
Source
GemFire
Sink
GemFire
Source
WebSocket
Sink
Simulator
Spring Cloud
Stream App
Spring Cloud
Stream App
Spring Cloud
Stream App
Java CLI
App
Messaging
WebSocket
Sink
Spring Cloud
Stream App
MVW App
Data Flow
Deployment
Scale Out
When NecessaryReverse Geocoding

22. MongoDB
Processor
Real-time
Aggregation/Analytics
Spring Boot
App
Spring Cloud
Stream App
Spring Cloud
Stream App
HTTP
Source
GemFire
Sink
GemFire
Source
WebSocket
Sink
Simulator
Spring Cloud
Stream App
Spring Cloud
Stream App
Spring Cloud
Stream App
Java CLI
App
Messaging
WebSocket
Sink
Spring Cloud
Stream App
MVW App
Data Flow
Deployment
Runtime
Reverse Geocoding

23. 6. PRODUCTION LEVEL DEPLOYMENT

24. MQTT
Source
Enrichment
Processor
IMDG
Sink
Machine
Learning
Processor
WebSocket
Sink
MPP/HDFS
Sink
Real-time
IMDG
Sink
MPP/HDFS
Source
Data Pipelines
IMDG
Source
Broker
by RabbitMQ
WebSocket
by RabbitMQ
Front-end Services
Spring Cloud
Stream Apps
MVW App Spring Cloud
Stream Apps
Data MicroServicesBusiness MicroServices
……
Containerized Applications
Taxi/Devices
• Event Driven
Spring Cloud Stream
• Low Latency
Messaging by RabbitMQ
Real-time by GemFire
• Scalability
Apps Scale-Out by Containers
Governance by Data Flow
Services Scale-out
• High Availability
Platform HA by Cloud Native
Platform or PaaS
• Portability
Apps & Services portable to
other Cloud Native Platforms
• Light Weight
Loosely Coupled MicroServices
or or etc.
Back-end Services
RabbitMQGeode
MongoDB HAWQ
or
Spring Cloud Data Flow

25. 7.DEPLOYMENT & RUNTIME
SCREENSHOTS

30. • Real-time Taxi Dropoff Events
• Dropoff-district Count

31. • 10 most frequent routes
• 10 most frequent routes

32. 8. LOCAL DEPLOYMENT DEMOVIDEO