Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus StreamAnalytix Webinar

Real-time Streaming Analytics
Based on Apache Storm
Big Data Solutions and Services Partner for
Enterprises
1
© 2014 1 Impetus Technologies

Our Speakers Today
2
Dr. Vijay Agneeswaran, Ph.Director, Big Data
R&D
Anand Venugopal
Sr. Director,
Business
Development
Recorded version available at http://bit.ly/1wb9SZg

Agenda
Introductions
and
Background
Streaming
Analytics
Options
StreamAnalytix
Introduction
StreamAnalytix
Demo
Q&A

Real-time Streaming Analytics Outlook
A 2014 survey data revealed a 66% increase in firms’ use of
streaming analytics in the past two years
70% of the most profitable companies will manage their business
processes using real-time predictive analytics or extreme
collaboration by 2016
Analysts believe, that it has taken 15 years or so for
companies to harness about 50% of the productivity
potential of the Internet, and the next 50% of
productivity gains likely requires connecting things

Business Value of Analytics
Diminishes with the age of data
$$$ ?
Befor
e
• Predictive analytics based on current
events
• Value depends on accuracy
$$
NOW
• Real-time
• Certainty is high – REAL
• Value based on quick
response
$$$
Later
• Descriptive
• Diagnostic
• Least value
5
The drop is non-linear
Value of Data
Age of Data

Business Value of Real-time Streaming
Analytics

SECTION 1
Open Source Options for
Stream Processing

Stream Processing Open Source Options
• S4 from Yahoo
• MillWheel from Google
• Samza from LinkedIn
• Storm
• Spark Streaming

Storm and Spark Positioning
FEATURE STORM SPARK STREAMING
Processing
Methodology
Processes and
dispatches
messages as soon
as they are
received
Treats streaming computations
as a series of deterministic batch
computations on small time
intervals

Processing Latency Lower Latency Higher Latency,
Higher Throughput

Availability Available through
the support of
YARN and MESOS
Available through the support of
YARN and MESOS

Complex Event
Processing
Run SQL-like
commands using
Esper
Spark SQL works on top of
Spark Streaming, still in Beta

Intermediate Data
Storage
Uses ZeroMQ /
Netty for exchange
of data amongst
different Storm
topology tasks
Stores the intermediate data in-memory
in the form of RDDs

Sliding Window
Concept
Achievable through
Esper
Built-in support

Lambda
Architecture
May need separate
batch pipeline
Can have same pipeline for
batch and stream computations

Message
Processing
Semantics
Exactly once
messaging
achieved with
Trident
Exactly once messaging
achieved through RDD lineage

Fault Tolerance Highly fault tolerant
through the use of
Zookeeper cluster
that stores the
cluster and
topology state
Maintains state information by
writing metadata information of
the DStreams to HDFS directory
and periodic check-pointing

Production
Deployments
Groupon, Twitter,
The Weather
Channel,
Infochimps, Aeris,
The Ladders,
Yahoo and many
more.
Sharethrough, Yahoo, Ooyala,
Conviva

+ Storm + Spark Neutral
Feature Storm Spark Streaming
Processing Methodology
Processes and dispatches messages as
soon as they are received
Treats streaming computations as a series of
deterministic batch computations on small time
intervals
Processing Latency Lower Latency Higher Latency, Higher throughput
Availability
Available through the support of YARN and
MESOS
Available through the support of YARN and MESOS
Complex Event Processing Run SQL-like commands using Esper
Spark SQL works on top of Spark Streaming, still in
Beta
Intermediate Data Storage
Uses ZeroMQ / Netty for exchange of data
amongst different Storm topology tasks
Stores the intermediate data in-memory in the form of
RDDs
Sliding Window Concept Achievable through Esper Built-in support available
Lambda Architecture May need separate batch pipeline
Can have same pipeline for batch and stream
computations
Message Processing
Semantics
Exactly once messaging achieved with
Trident
Exactly once messaging achieved through RDD
lineage
Fault Tolerance
Highly fault tolerant through the use of
Zookeeper cluster that stores the cluster and
topology state
Maintains state information by writing metadata
information of the DStreams to HDFS directory and
periodic check-pointing
Production Deployments
Groupon, Twitter, The Weather Channel,
Infochimps, Aeris, The Ladders, Yahoo and
many more.
Sharethrough, Yahoo, Ooyala, Conviva

Poll

StreamAnalytix – gives you a future proof option
STORM SPARK OTHERS
NOW
Time

SECTION 2
Introduction to StreamAnalytix

"Default" Approaches to Streaming Analytics
Proprietary Platforms
• No leverage of Open
Source
• Vendor lock-in
• Could be high cost
• Limited flexibility
" Do it yourself "
• Native Open source
• No vendor support
• Integration & maintenance
nightmare
• Significant delays in time-to-market

The 3rd Approach: Best of Both Worlds
StreamAnalytix mitigates the disadvantages of the "default" approaches and
offers the benefits of both worlds to enterprises for streaming analytics.

StreamAnalytix Platform Benefits
An “App Server” for real-time apps
Focus on your business logic - leave infra to
us
Handle all the 3V’s of Big Data on one
platform
12-18 months of time to market acceleration
Seamless integration with Hadoop and
NoSQL

Block Diagram

Key Features
High Speed
Data
Ingestion
Elastic
Scaling –
Volume,
Velocity
Data Parsing
- Variety
Pluggable
Persistence
Real-time
Index and
Search
Dynamic
Message
Routing
Rule Based
Alert
Pluggable
Workflow
Management
Fault
Tolerance
and Data
Integrity
Optimized for
High
Performance

SECTION 3
Demo

Recap of Key StreamAnalytix Functions
Read
• Ready-to-use connector
• Support for AMQP, KAFKA connectors
• Supports XML, JSON, DELIMITED, etc.
Analyze
• Analyze data in near real-time
• Rule based data routing and alerts
• Workflow management
Persist
• Cross vendor persistence abstraction
• Support for HBase, Cassandra, Oracle NoSQL DB
• Support of indexing and distributed caching
Reports
• Near real-time data visualization
• Historical data visualization
• Search on moving and historical data

Value Adds over Open Source – 1 of 3
Full System Integration, Testing,
Benchmarking and Technical Support
• Kafka and/or Rabbit MQ
• Apache Storm
• Data-storage and Indexing layer -
abstraction and integration
• Alerting and CEP Framework
• Real-time Web based visualization
framework
• Distributed and off-heap caching
for reliability and shared-state
solutions
• BPM / Workflow integration
System
Level
Performance
Data compression and
other optimizations to
minimize
latency of event
processing
Admin Tool - Automated
Installation, Provisioning,
Management and Monitoring tool
for all components and
infrastructure with multi-cluster
management from one station
Monitoring and
Automation of full
system availability

Visual definition of
incoming message
formats, fields
Storm
Workspaces – the
beginning of multi-tenancy
PMML integration over
Storm – UI to import
and run PMML models
on Storm
Visual definition
of Alerting,
regular
expressions
Visual topology creation
and integration with
Alerting, custom-logic,
web UI
Multi-topology creation,
linking and dynamic “run-time”
data routing
Application and pipeline
monitoring and management
(visual interface)

• Web-socket framework for custom
real-time visualization
• Alerts, Raw Data, Enriched data
results
Persistence and Indexing
Real-time Visualization Support
• Abstraction layer supporting any
NoSQL database as persistence store
• Abstraction layer supporting Elastic
search (Solr support soon)
• Indexing and Persistence policy
definition at system level and per field
• Encryption support configurable at
column level
• Fast search support by orchestration
between Index and Storage
• ‘oData’ interface support or 3rd party
BI tools for offline analytics

Licensing Model and Controls
TIME
• Perpetual license + Annual support and maintenance from second
year
OR
• Annual Subscription (includes support)
QUANTITY
• Total number of cores
OR
• Total number of nodes (plus a max cores per node)
FUNCTIONALITY
• Only Indexing / Only Persistence/ Indexing + Persistence
• Admin tool – Yes/ No

Recorded version available at
http://bit.ly/1nMw8nQ
For general inquiries about the StreamAnalytix platform and related
services reach us at inquiry@streamanalytix.com

Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus StreamAnalytix Webinar

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (10)

Semelhante a Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus StreamAnalytix Webinar

Semelhante a Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus StreamAnalytix Webinar (20)

Mais de Impetus Technologies

Mais de Impetus Technologies (20)

Último

Último (20)

Real-time Streaming Analytics for Enterprises based on Apache Storm - Impetus StreamAnalytix Webinar

Notas do Editor