Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2btGUjQ. Pat Patterson and Ted Malaska talk about current and emerging technologies. They evaluate each and understand how they are useful in solving problems related to large scale data processing, joining and combining streams. They also talk about the various ways of achieving "at least once" and "exactly once" processing and how we can make sure that data is processed in a timely fashion. Filmed at qconnewyork.com. Pat Patterson has been working with Internet technologies since 1997, building software and working with communities at Sun Microsystems, Huawei, and Salesforce, and now he is a community champion at StreamSets. Ted Malaska is a solutions architect at Cloudera. He has 18 years of professional experience working for start-ups, the US government, banks, commercial firms, bio firms and retail ones.

1. 1© Cloudera, Inc. All rights reserved.
13 June2016
Ted Malaska| Principle Solutions Architect @ Cloudera,
Pat Patterson| Community Champion @ StreamSets
Ingest and Stream Processing -
What will you choose?

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
ingest-stream-processing

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. 2© Cloudera, Inc. All rights reserved.
About Ted and Pat
Ted Malaska
• Principal Solutions Architect
@ Cloudera
• Apache HBase SparkOnHBase
Contributor
• Contact
• ted.malaska@cloudera.com
• @TedMalaska
Pat Patterson
• Community Champion @
StreamSets
• Formerly Developer Evangelist at
Salesforce
• Contact
• pat@streamsets.com
• @metadaddy

5. 3© Cloudera, Inc. All rights reserved.
Streaming Patterns
•Ingestion
•Low Millisecond Actions
•Near Real Time Complex Actions

6. 4© Cloudera, Inc. All rights reserved.
Parts Of Streaming
Producer Kafka Engine Destination

7. 5© Cloudera, Inc. All rights reserved.
Parts Of Streaming
Producer Kafka Engine Destination
At Least once
Ordered
Partitioned
At Least Once Depends
Depends

8. 6© Cloudera, Inc. All rights reserved.
Destinations
• File Systems: example HDFS
• Batch is good
• Only can do exactly once is a file is closed in a single ack.
• Good for Scans
• Solr
• Everything is Document based making exactly once
• Batch is still good
• Good for Search Queries

9. 7© Cloudera, Inc. All rights reserved.
Destinations
• NoSQL: example HBase
• Everything has a row key making exactly once for writes
• Increments can be applied twice is so be careful
• Good for gets and puts
• Kudu
• Everything has a row key making exactly once for writes
• Good for gets, puts, and scans

10. 8© Cloudera, Inc. All rights reserved.
Ingestion Destinations
• File Systems: example HDFS
• Flume
• Kafka Connect
• Solr
• Flume
• Any Streaming Engine

11. 9© Cloudera, Inc. All rights reserved.
Ingestion Destinations
• NoSQL: example HBase
• Flume
• Any Streaming Engine: Storm and Spark Streaming Tested
• Kudu
• Flume
• Kafka Connect
• Any Streaming Engine: Spark Streaming Tested

12. 10© Cloudera, Inc. All rights reserved.
Tricks With Producers
• Send Source ID (requires Partitioning In Kafka)
• Seq
• UUID
• UUID plus time
• Partition on SourceID
• Watch out for repartitions and partition fail overs

13. 11© Cloudera, Inc. All rights reserved.
Streaming Engines
• Consumer
• Flume, KafkaConnect, Streaming Engine
• Storm
• Spark Streaming
• Flink
• Kafka Streams

14. 12© Cloudera, Inc. All rights reserved.
Consumer: Flume, KafkaConnect
• Simple and Works
• Low latency
• High throughput
• Interceptors
• Transformations
• Alerting
• Ingestions

15. 13© Cloudera, Inc. All rights reserved.
Consumer: Streaming Engines
• Not so great at HDFS Ingestion
• But great for record storage systems
• HBase
• Cassandra
• Kudu
• SolR
• Elastic Search

16. 14© Cloudera, Inc. All rights reserved.
Storm
• Old Gen
• Low latency
• Low throughput
• At least once
• Around for ever
• Topology Based

17. 15© Cloudera, Inc. All rights reserved.
Spark Streaming
• The Juggernaut
• Higher Latency
• High Through Put
• Exactly Once
• SQL
• MlLib
• Highly used
• Easy to Debug/Unit Test
• Easy to transition from
Batch
• Flow Language
• 600 commits in a month
and about 100 meetups

18. 16© Cloudera, Inc. All rights reserved.
Spark Streaming
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count Print
Source Receiver RDD
RDD
RDD
Single Pass
Filter Count Print
First
Batch
Second
Batch

19. 17© Cloudera, Inc. All rights reserved.
DStream
DStream
DStream
Single Pass
Source Receiver RDD
Source Receiver RDD
RDD
Filter Count
Print
Source Receiver
RDD
partitions
RDD
Parition
RDD
Single Pass
Filter Count
Pre-first
Batch
First
Batch
Second
Batch
Stateful
RDD 1
Print
Stateful
RDD 2
Stateful
RDD 1
Spark Streaming

20. 18© Cloudera, Inc. All rights reserved.
Flink
• I’m Better Than Spark Why Doesn’t Anyone use me
• Very much like Spark but not as feature rich
• Lower Latency
• Micro Batch -> ABS
• Asynchronous Barrier Snapshotting
• Flow Language
• ~1/6th the comments and meetups
• But Slim loves it 

21. 19© Cloudera, Inc. All rights reserved.
Flink - ABS
Operator
Buffer

22. 20© Cloudera, Inc. All rights reserved.
Operator
Buffer
Operator
Buffer
Flink - ABS
Barrier 1A
Hit
Barrier 1B
Still Behind

23. 21© Cloudera, Inc. All rights reserved.
Operator
Buffer
Flink - ABS
Both
Barriers Hit
Operator
Buffer
Barrier 1A
Hit
Barrier 1B
Still Behind

24. 22© Cloudera, Inc. All rights reserved.
Operator
Buffer
Flink - ABS
Both
Barriers Hit
Operator
Buffer
Barrier is
combined
and can
move on
Buffer can
be flushed
out

25. 23© Cloudera, Inc. All rights reserved.
Kafka Streams
• The new Kid on the Block
• When you only have Kafka
• Low Latency
• High Throughput
• Not exactly once
• Very Young
• Flow Language
• Very different hardware profile then others
• Not widely supported
• Not widely used
• Worries about separation of concern

26. 24© Cloudera, Inc. All rights reserved.
Summary about Engines
• Ingestion
• Flume and KafkaConnect
• Super Real Time and Special
• Consumer
• Counting, MlLib, SQL
• Spark
• Maybe future and cool
• Flink and KafkaStreams
• Odd man out
• Storm

27. 25© Cloudera, Inc. All rights reserved.
Abstractions
Code Abstractions
Beam
SQL Abstraction
SQL
UI Abstraction
StreamSets
Streaming Engines

28. 26© Cloudera, Inc. All rights reserved.
StreamSets Data Collector
Building a Higher Level, Open Source Tool

29. 27© Cloudera, Inc. All rights reserved.
Traditional and Big Data
Founders
StreamSets Company Background
Top tier Investors
Momentum to Date
Strategic Partners
• Founded 2014; exited stealth 9/15
• ~30 employees
• Double-digit enterprise customers
• 10,000 downloads

30. 28© Cloudera, Inc. All rights reserved.
Thank you!

31. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/ingest-
stream-processing