Streaming Architecture for Hybrid Cloud Applications

© 2016 MapR Technologies 1© 2016 MapR Technologies 1MapR Confidential © 2016 MapR Technologies
Architecting a hybrid cloud application using a
global publish-subscribe streaming message
system
Mathieu Dumoulin (MapR Technologies)
Strata Singapore 2016

Streaming Architecture to Connect Everything
(including Hybrid Cloud)
Mathieu Dumoulin (MapR Technologies)
Strata Singapore 2016

© 2016 MapR Technologies 3© 2016 MapR Technologies 3MapR Confidential
Mathieu Dumoulin, Data Engineer
• Master’s degree in text
classification on Hadoop at Fujitsu
Canada’s Innovation Lab and Laval
University
• In Tokyo, I’ve worked as a Data
Scientist, Search Engineer and
Data Engineer
• Working on streaming, complex
event processing and machine
learning

The new rule for the future is going
to be, “Anything that can be
connected, will be connected.”
Jacob Morgan, Forbes - May 2014

Talk Summary
• Clouds: private vs. public vs. hybrid
• It’s all about that streaming
– Streaming for IoT
– Publish-subscribe messaging systems (Kafka)
– Stream Processing (Apache Spark Streaming,
Apache Flink)
– Microservices
• Streams-based Architecture in the hybrid cloud
– Design goals
– Examples
• Recap, Q&A

Weather today for IT:

Public Cloud - Low Upfront Cost and Flexibility
The Good
• Right size instances for
application
• Grow with the business
• “Forever” extensible
• Global in a few clicks
The Bad
• New complexity, no
magic
• Costs can run away
The Ugly
• Local data is far from
processing
• Severe lock-in without
huge in-house expertise

Private Clouds - The Benefits of Ownership
The Bad
• Harder to scale vertically &
horizontally
• Cost of multiple datacenters
The Ugly
• Pay for spike, wasted
resources
• Never right size in a growing
organization
The Good
• Direct access to data
• Security, privacy and legal
compliance
• Hardware certainty
• Low running cost

Private Cloud
- Europe
Private Cloud - Tokyo
Hybrid = Public vs. + Private
Spans at least one public and
one private cloud.
• Test new ideas with low
up-front capital cost
• Cloudbursting
• High Availability and Disaster
Recovery
• Regulatory Requirements
IT infrastructure agility

It’s all about that streaming

Streaming Architecture the Norm for Data Driven
Organizations
“Stream-based computing is becoming the norm for data-driven organizations”
- Friedman & Dunning, Streaming Architecture
• Build flexible systems
– more efficient and easier to build
– Decouples dependencies between data source and processing
• Better model the way business processes take place.
• More value now… and later
– Aggregates data from many sources once
– Serves data to one or many projects immediately
– More efficient and high performance
– Run batch analytics, reprocess data

IoT is a Natural Use Case for Streaming
Connected devices produce data as real-time events that are
modelled naturally as event streams.
Event
Some actions have value only if taken immediately
– Navigation updates from traffic conditions, accident reports, disasters, …
– Slowing down or stopping a factory line in response to quality issues
– Re-routing items mid-way during shipping to increase efficiency
– Continuous engine tuning

IoT is Happening Right Now!

Streams Make the Hybrid Cloud Practical
Streams can serve for inter-cloud
communication in the exact same way they
support any other scenario.
● Abstracts the differences between
on-premise and cloud
● Standardize the expected flow of data
between modules
● Reuse data many times, break down
data silos

What Streaming Requires from a Messaging System
● The producer and consumer are fully independent
● Very high throughput 1,000+/s → 1,000,000+/s
● Persistence
○ Fault-tolerance
○ Data is kept as a replayable sequence
○ Strong ordering of events
● Naming of topics (consumers pick the data they need )
● Geo-distributed replication (for Hybrid Cloud use cases)
It’s very hard to get full isolation of producer and consumers while also keeping
very high speed, but we must have both.

What Streaming Requires from Stream Processing
Frameworks
Desirable features for real-time analytics frameworks:
• Open Source, active development and developer community
• Supports “exactly once” guarantee, stream reprocessing
• How much real-time? Microbatch vs. record-at-a-time
• Performance (latency, throughput)
• Other: Easy to use, compatibility, talent availability
To Know more: https://www.mapr.com/blog/stream-processing-everywhere-what-use
Jim Scott - Stream Processing Everywhere - What to Use? Strata San Jose 2015
Also see Data Artisan’s Blog on Stream Processing Framework Myths

Which Stream Processing Frameworks?

Summing up: Technology to support Streaming
1. Lightweight messaging system
2. Stream Processing Framework
You can get an Introduction to Flink in this Free Book published by O’Reilly

Key Ideas For Effectively Using Streams
Real-time
Analysis
Persist to
Disk
Geo-distributed
Replication
Core part of
Architecture

Streaming Architecture: Ideal Platform for Microservices
Microservices are a modern distributed architecture that realizes the
promises of SOA, Service Oriented Architecture
• Scale up from a test use case to a global deployment
• Decouples components, more modular
• Modern, agile development, testing and deployment
• More robust and responsive
See Krystal Valentine’s “The keys to an event-based microservices application”
presentation, Strata New York 2016

Monolithic to Microservices Architecture
See Fowler’s blog about microservices:
http://www.martinfowler.com/articles/microservices.html

Microservices are Truly Decoupled

When to Use Streaming Architecture

Connect Clouds with Streams:
Streams-based Architecture

Switch from thinking of computer
programs as state-oriented to thinking
of them in terms of flows”
Ted Dunning & Ellen Friedman, Streaming Architecture - O’Reilly - 2016

An End-to-End Streaming Architecture
Japan North
Data Center
Stream
GW
Global Data
Center
Stream

Example Architecture: Log Analysis

Example Architecture: The MapR Blueprint
Download the Finserve app from Github!
https://github.com/mapr-demos/finserv-application-blueprint

Conclusion
• The hybrid cloud matters for IT agility
• Use streams for communication between elements
• Streaming-based systems can be arbitrarily complex
– Still fast, responsive, reliable and easier to develop!
• In a streaming architecture world, a converged platform
(built-in streaming, storage and DB) makes a difference.

Suggested Reading And Video Links
Get Ted & Ellen’s book: Read it Online for Free!
New content presented by Ted Dunning:
1. Big Data in the Cloud (blog):
www.mapr.com/big-data-cloud
a. Direct video link:
https://youtu.be/90KrQAb1_Cw
2. Converged Advantages in the Cloud (blog):
www.mapr.com/converged-cloud
a. Direct video link: https://youtu.be/yjfBXNcmAHA

Q & A
@mapr
mdumoulin@mapr.com
@lordxar
Engage with us!
mapr-technologies

Key Ideas for Microservices
• Services are opaque - API only
• They communicate with only a few other services using
lightweight, flexible protocols.
– HTTP+REST - Synchronous (frontend)
– Messaging Systems (Kafka, MapR Streams) - Asynchronous (backend)
• Data formats should be future-proofed
– JSON - Human readable, easy to use, low efficiency
– Binary (Avro, Protobuf, Thrift) - Efficient but (somewhat) harder to use
{RESTful}

Spark Streaming or Flink: Case by Case
Micro-batches. Time-based
window. Latency: seconds
Continuous flow model.
Record-based window. Latency: ms
Both provide exactly once guarantee, high throughput and low
overhead of fault tolerance. Both streaming and batch supported.

The Hybrid Cloud for IoT Infrastructure
• IoT is a new use case - Need to Test
• Built-in need for baseload capacity and bursting data spikes
• Global marketplace requires geographically dispersed
datacenters
• Increasingly strict compliance requirements
• IoT Security issues need to be taken seriously
Why do IoT applications call out for the flexibility of Hybrid Cloud?

Streaming Architecture for Hybrid Cloud Applications

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Streaming Architecture for Hybrid Cloud Applications

Semelhante a Streaming Architecture for Hybrid Cloud Applications (20)

Último

Último (20)

Streaming Architecture for Hybrid Cloud Applications