SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation

Copyright © 2014 – Proprietary and Conﬁdential Information of SQLstream Inc.
1
!
Damian Black, CEO SQLstream
Strata + Hadoop World, 2014!
October 15-17 2014, New York

!SmartCity
StreamApp:
An
Internet
of
Things
Service
for
Real-‐
;me
Traﬃc
Management

2
!
Product:
SQLstream Blaze Stream Processor
Focus:

Powering smart services for the IoT/IoE
Located:
San Francisco

!
Real-time Action in the Internet of Everything

3
“We predict that the Internet of Everything will be a $19 trillion dollar market over the
next several years.”

John Chambers, CEO, Cisco

“When you look at the Internet of Things, it is clear 2014 will be a tipping point in the
evolution of the Internet.”

Marissa Mayer, CEO and President, Yahoo

“We are equipping our products with sensors that constantly measure performance
so our customers see major productivity gains and minimize unplanned downtime.”

Jeﬀ Immelt, Chairman and CEO, GE

A world transformed by an explosion of sensors

4
A World of Big Data
§  Big Data – datasets (3V) beyond the ability of traditional data management systems
§  Driven by Hadoop: massively scalable, multi-server systems on low cost hardware
§  Designed for semi-structured and unstructured data – ideal for sensors
§  Driving expectation for real-time responses and action
4
Internet
Wireless
Sensors
Smartphones

5
0!
45!
90!
135!
180!
225!
2014
2015
2016
2017
2018
2019
2020
Analytics will be dominated by IoT Data
IoE (net IoT)
9.5% CAGR [IDC]
IoT (real-time)
44% CAGR [Gartner]
$Billions
Characteris;cs
of
Internet
of
Things
sensor
data

> 
Very
high
data
volumes

> 
Systems
must
react
to
each
and
every
record

> 
Mostly
‘business
as
usual’
–
no
need
to
warehouse
every
record

6
Stream Processing!
Relational Databases!
Hadoop !
In-memory
Databases!
Big Data Technology: From Human Time to Real-time
6
High
Low
Speed of
Response
Throughput
Capacity
Low
High
Store First
Process First

7
Stream Processing versus Stored Data Queries
Databases store then query.
Stream Processors query then store.
Query
Query
Databases
The query traverses the table.
With every record, the table is
updated and the process must
start again.!
Stream Processing
The incoming streams of data)
move through a continuous,
windowed SQL query. Unlimited
capacity and continuous, real-time
results.!
Storage 
(Optional — RDBMS, Hadoop, NoSQL …)
Performance Metrics
§  1 million EPS/core
§  Multi-server scale-out
§  Latency < 10 milliseconds

8
A real-time data hub for the Internet of Things
IoT
Smart
Srvcs
Smart
Srvcs
Smart
Srvcs
Smart
Srvcs
Smart
Srvcs
Mobile
Apps
Apps
Apps
Apps
Server
Server
Server
Server
Internet
Switch
Proxy
Router
Billing
Fraud
SLA

9
SmartCity StreamApp"
Roads & Maritime Services Case Study

10
We live in a Modern World
§  800 million vehicles on the world’s roads today, estimated to increase to 4 billion by 2050
§  Environmental concerns and air pollution driving innovation, but just the beginning
§  The expectation of on-demand, real-time and accurate information has not been realized
§  The Quality of Life for travelers and commuters is deteriorating
10
Conges4on
Environment
Poor
Informa4on
Quality
of
Life?

11
The Internet of Things
Smart People to Smart Cities with Smart Services
§  Step 1: Real-time traffic flow and congestion for transportation agencies
§  Step 2: Extend for multi-modal Travel Time with public transportation
§  Step 3: Smart City Smart Services – dyanamic traffic management
§  Step 4: Traveler and Commuter Smartphone App access
11
Traffic
Analy4cs
Mul4-‐Modal
Smart
City
Real-‐4me
Informa4on

12
From Big Data to real-time actions
§  Stream processing means high throughput with low latency actions at Big Data scale
§  A real-time data hub delivers multiple applications on a single stream processing platform
§  The platform for the real-time SmartCity StreamApp
§  Delivering a better traveler experience
12
Stream
Processing
Real-‐4me
Data
Hub
Smart
City
PlaIorm
A
BeKer
Experience

13
Case Study: Big Data Innovation for ITS
TT5
:
Real-‐4me
SmartCity
StreamApp
Traffic
App
Hackathon

(Powered
by
SQLstream,
Sept
20-‐21
2014)

•  GPS
fleet
data
–
low
solu;on
cost,
faster
;me
to
value

•  Real-‐;me
traffic
flow
&
conges;on
predic;on

•  All
classified
roads
in
NSW
networks

•  Opera;onal
dashboards
based
on
Google
Maps

•  Public
API
(web
service
feed
and
JDBC
app
connect)

•  Encourage
innova;on
and
app
development

•  Focus
on
conges;on
hotspots

•  Funding
for
the
winning
team

•  Building
on
SQLstream
TT5
API

14

15
Calculate Road Segment (10m) Mean Travel Time

Vehicle
1

Vehicle
2

Vehicle
3

Vehicle
4

Vehicle
N

Aggregate
Mean
Travel

Time
per
Road
Element

Vehicle
GPS
record

Vehicle
Travel
Time
(green
light)

Vehicle
Travel
Time
(red
light)

16
Travel Time Calculation using Vehicle GPS Data Streams

Calculate Vehicle Travel Time along speciﬁed route bounded by REi and REj:
•  RE: Road Element (10 meter segments)
•  l: Length of RE (typically 10 meters)
•  D: Distance traveled by vehicle between GPS reports
•  T: Time taken by vehicle to travel between GPS reports
•  n: Number of GPS reports
TTREi−REj =
1
nRE
Tv,rlREs
Dv,rr(reports)
∑
#
$
%%
&
'
((
v(vehicles)
∑
#
$
%
%
&
'
(
(
REs
∑

17
Update average speed incrementally across any subset of network over rolling time windows
Code Example #2: Continuous Average Speed Calculation
CREATE OR REPLACE VIEW "SpeedZoneStats"
DESCRIPTION ‘Rolling averages for multiple windows partitioned by zone' AS
SELECT STREAM
"zone", -- zone id
"segmentid", -- parent road segment
"speedlimit", -- speed limit for zone
AVG("Speed")OVER last1Min AS "avgSpeed1", -- 1-min running average
AVG("Speed")OVER last5Min AS "avgSpeed5", -- 5-min running average
AVG("Speed")OVER last10Min AS "avgSpeed10" -- 10-mn running average
FROM "RoadPositionInfo"
WINDOW
last1Min AS (PARTITION BY "zone" RANGE INTERVAL '1' MINUTE PRECEDING),
last5Min AS (PARTITION BY "zone" RANGE INTERVAL '5' MINUTE PRECEDING),
last10Min AS (PARTITION BY "zone" RANGE INTERVAL '10' MINUTE PRECEDING);

18
Improve System Accuracy"
Join streaming analytics with historical trends from the data warehouse
0:00

1:00

2:00

3:00

4:00

5:00

6:00

7:00

8:00

9:00

10:00

11:00

12:00

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

0:00

Average
(Trend)
Real-‐;me

>  Trend data comparisons help
to differentiate exceptional
event from normal behavior
>  Predicting extent and
duration of an event can still
be difficult
>  Assessing the traveler
sentiment from Social Media
analytics offers additional
insight
Average Travel Time over a 24 hour period

19
SmartCity StreamApp"
Architecture

20
SQLstream
Blaze

Distributed
SQL
stream

processing
engine
with

intelligent
guide
discovery

and
real-‐;me
dashboards.

s-Visualizer
real-time dashboards for
Enterprise Power Users
s-Server
Distributed SQL Stream
Processor
s-Dashboard
HTML5 real-time
dashboards for Developers
Storm
Adapter
s-Studio
Developer & Admin
StreamLab
Intelligent guided data stream discovery,
analytics and visualization without coding
Enterprise-Class Real-time Data Hub 
Stream Processing for Operational Intelligence and the Internet of Things
Industry

StreamApps

Pre-‐built
libraries
for
rapid

development
of
industry

stream
processing
solu;ons.

Telecoms
Smart
City

Emergency

Services

Oil
&
Gas

21
Control

Systems

SQL Optimizer

Parallel Scheduler

Real-time Indexing

RT Memory Manager

Dynamic Java Analytics (UDX)

Streaming Data Protocol

Interac4ve
Stream
Discovery
and
Visualiza4on

Stream Processing Engine

(HTML5)!
Discovery

API Connect

SmartCity StreamApp – Core Platform Architecture
Devices
&
Apps

Hadoop Ingestion (MB/s)

Events/Sec/Core)

Native
Tables

Repor4ng

Tools

JDBC

Web
Agent

Web Sockets
REST

(HTML5)!
Dashboards

(Flash)!
Dashboards

JDBC

Remote
Systems

Enterprise

Systems

Agents

Adapters

Devices
&
Apps

JDBC

Machine
Data

Enterprise

Systems

Agents

Adapters

Hadoop
&
NoSQL
Enterprise
BI

Data Warehouse

SQL Database

Predictive Analytics

Hadoop / HDFS

HBase

Storm & Kafka

22
SmartCity StreamApp – Software Architecture
Data
Stream
Acquisi4on
Toolbox

GIS & Geospatial Library

Data Quality & Density Analysis

Data Parsing & Transformation

Data Enrichment

StreamApp
Analy4cs
Toolbox

Congestion Prediction Library

Congestion Detection Algorithms

Real-time Trafﬁc Flow

Object Tracking

Integra4on
&
Delivery
Toolbox

Alerting Module

Control System Metadata

KML & Ruby/Rails Library

Email & Workforce Activation

Customer Extensions
Customer Extensions
Customer Extensions

Travel
Time
App

Travel Time Subsystem

Customer Extensions

Conges4on
App

Congestion Subsystem

Customer Extensions

Network
KPIs
App

Network KPIs Subsystem

Customer Extensions

Custom
App

Custom App Subsystem

Distributed SQL & UDX(Java) Execution Engine

Query Management
Distributed Query Optimization
Query Scheduler

Platform - Distributed Server Cluster

JDBC Driver
Adapter Plug-ins
UDX (Java) Libraries
Agent Framework

StreamApp End-
user Applications
StreamApp Toolkits
& External Libraries
StreamApp
Execution Platform

23
Real-‐4me
Data
Hub
for
SmartCity
StreamApp

SmartCity StreamApp – Solution Architecture
Roadside
Furniture

Traffic !
Lights

Speed
Signs

Agents

Vehicle
GPS

Bus &
Train
Roads
Agents

Weather
Feeds
Agents

Social
Media

Twitter Feeds

Agents

Historical
Trends

Data!
Warehouse

Predictive
Analytics

Adapters

Logs
XML, JSON
XML, JSON
JDBC, API
Write API Live
Road
Updates

Traffic !
Lights

Speed
Signs

HTTP RT
Travel
Time
Apps

HTTP RT
Conges4on
Maps

Smart
phone
Device

Website

KML/Ruby RT
Traffic
Opera4ons

Real-time Dashboards

KML/Ruby Planning

KPI Reports

Real-time updates to
traffic light phasing,
speed signs and dynamic
lane configuration
Commuters
on the move
Commuters
travel planning
Network
Operations
Network
Planning
SmartCity StreamApp

Streaming Integration Layer

Distributed Stream Processing
Execution Environment

Operational
Servers

Internet!
Servers

Historical
Trends

Data!
Warehouse
Predictive Analytics

Adapters

JDBC, API

24
>  Minimum Time to Value
Combination of subscription-based GPS vehicle feeds with pre-built StreamApp means operational system for
first requirements available in a few weeks.
>  Strong Return on Investment
Estimated $20million saving on initial project, with > $100 million over project’s lifetime.
>  Traffic congestion reduction
Real-time operational displays and greater accuracy for long term forecasting means a significant reduction in
congestion at key hotspots.
>  Improved commuter experience
Accurate and reliable Travel Time estimates with real-time updates means commuters are better prepared and
better able to plan re-routes.
SmartCity StreamApp – Operational Results

SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation

Semelhante a SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation (20)

Último

Último (20)

SmartCity StreamApp Platform: Real-time Information for Smart Cities and Transportation