Google slide version of this slide can be accessed from: https://docs.google.com/presentation/d/1Ws73JxlVH39HiKiYuF3vW903j8wFzxPQihXz4CQ_HZM/edit?usp=sharing

1. Imre Nagi
Traveloka Data
@imrenagi
Jakarta
Google Cloud Dataflow
Unified Model for Stream and Batch
processing

2. Imre Nagi
Ping me @imrenagi
Previously:
Software Engineer @CERN @eBay Inc
Currently:
Software Engineer @Traveloka Data
Docker Community Leader, Indonesia

3. Agenda
What is Dataflow?
Dataflow Abstraction
Dataflow Common Pipeline
Stream Analytics

4. What is
Dataflow?
Jakarta

5. Apache Beam ...
A set of SDK that define
programming model that
you use to build your
stream and batch
processing pipeline
Cloud Dataflow
Fully managed distributed
service that runs and optimizes
your beam pipeline

6. Jakarta
Dataflow for ..
● Move
● Filter
● Enrich
ETL
● Connecting to Cloud Pub/Sub
● Read and Write to BigQuery,
Bigtable, etc.
I/O Operation
● Streaming Computing
● Batch Computing
● Machine Learning
Analytics

7. Jakarta
Unified Programming Model
Unified: Stream & Batch Pipeline
Open Source:
Java SDK
Python SDK
Go SDK (New)

8. Jakarta
Cloud Pub/Sub Cloud Dataflow
(Streaming)
Cloud Bigquery
Source Processing Data Store
Unified Model (Streaming)

9. Jakarta
Cloud Pub/Sub
Cloud Storage
Cloud Dataflow
(Streaming)
Cloud Bigquery
Source Processing Data Store
Unified Model (Streaming & Backup)

10. Jakarta
Cloud Storage Cloud Dataflow
(Batch)
Cloud Bigquery
Source Processing Data Store
Unified Model (Batch)

11. Dataflow
Abstraction
Jakarta

12. Jakarta
Represents graph of data
processing transformation
PCollection flows through
the pipeline
Can have multiple I/O in the
beginning and end of
pipeline
Beam Pipeline

13. Jakarta

14. // Define the pipeline option
PipelineOptions options = PipelineOptionsFactory.create();
// Create the pipeline
Pipeline p = Pipeline.create(options);

15. Jakarta
Data Model
PCollection<T> is a
collection of data type T
May be bounded or
unbounded in size
Element might has implicit
or explicit timestamp

16. // Create the PCollection 'lines' by applying a 'Read' transform.
PCollection<String> lines = p.apply(TextIO.read().from("/path/to/some/inputData.txt"));
PCollection<String> linesGCS = p.apply(TextIO.read().from("gs://deeptech/*"));
static final List<String> LINES = Arrays.asList(
"This is the first line",
"You will say this one is the second",
"But it's not. ");
// Generating PCollection from in memory data
PCollection<String> lines = p.apply(Create.of(LINES)).setCoder(StringUtf8Coder.of())
// Generate bounded pcollection
PCollection<Long> bounded = p.apply(GenerateSequence.from(0).to(1000));
// Generate unbounded pcollection
PCollection<Long> unbounded = p.apply(GenerateSequence.from(0));

17. Jakarta
PTransform: Transforming the Data

18. public class HelloDoFn extends DoFn<String, String> {
@ProcessElement
public void processElement(ProcessContext context) {
String name = context.element();
context.output("Hello, " + name + " ! ");
}
}

19. public class StringToLongDoFn extends DoFn<String, Long> {
@ProcessElement
public void processElement(ProcessContext context) {
String name = context.element();
context.output(name.length());
}
}

20. PCollection<KV<String, Integer>> scores = input
.apply(Sum.integersPerKey());

21. Jakarta

22. Jakarta
I/O Transform

23. Dataflow Common
Pipeline
Jakarta

24. Jakarta
Linear Pipeline

25. Jakarta
Combining Multiple
PCollection

26. Jakarta
Producing Multiple
PCollections

27. Jakarta
Multiple Transformation for a PCollection

28. Jakarta
Joining PCollection

29. Stream
Analytics
Jakarta

30. Jakarta
data..

31. Jakarta
Can be big..

32. Jakarta
Tuesday
Wednesday
Thursday
Bigger and bigger..

33. ...maybe infinitely big...
9:008:00 14:0013:0012:0011:0010:00

34. Jakarta
9:008:00 14:0013:0012:0011:0010:00
8:00
8:008:00
Oops.. unknown delays

35. Jakarta
Lambda Architecture ever says that Stream Processing only CAN’T produce
accurate analytics result. Thus, Batch Processing is necessary to fix the
inaccuracy of the stream processing.

36. Jakarta

37. Jakarta
13:00 14:008:00 9:00 10:00 11:00 12:00
Processing
Time
∑ ∑ ∑ ∑ ∑ ∑ ∑8:00 8:00
Grouping via Processing-Time Windows

38. Jakarta
Processing
Time
11:0010:00 15:0014:0013:0012:00
Grouping via event-time windows
Event Time 11:0010:00 15:0014:0013:0012:00
Input
Output
∑ ∑ ∑ ∑ ∑ ∑

39. Jakarta
What is windowing?
Windowing divides data into event-time-based finite chunks.
Often required when doing aggregations over unbounded
data.
Fixed Sliding
1 2 3
54
Key
2
Key
1
Key
3
Time
1 2 3 4 A windowing function
computes which
window(s) an element
belongs to. Temporal
functions can be
parameterized with
duration and
frequency.

40. Jakarta
What about data-dependent windowing?
Sessions
2
431
Time
Unique per key - you
can't know a priori
when a session ends,
so the windowing
function is now also
parameterized by
state.

41. PCollection<KV<String, Integer>> scores = input.apply(
Window.into(FixedWindows.of(Duration.standardMinutes(2))))
.apply(Sum.integersPerKey());

42. Jakarta

43. Jakarta
Windowing specifies where events are aggregated in event time,
but when are events emitted in processing time?

44. Jakarta
Trigger
Triggers: A trigger is a mechanism for declaring
when the output for a window should be
materialized relative to some external signal.
Triggers provide flexibility in choosing when
outputs should be emitted.
They also make it possible to observe the output for a window
multiple times as it evolves

45. Jakarta
Windowed summation on a streaming engine with perfect (left) and heuristic
(right) watermarks.

46. PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)))
.triggering(AtWatermark()
.withEarlyFirings(
AtPeriod(Duration.standardMinutes(1)))
.withLateFirings(AtCount(1))))
.apply(Sum.integersPerKey());

47. Jakarta
Windowed summation on a streaming engine with early and late
firings.

48. PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)))
.triggering(AtWatermark()
.withEarlyFirings(
AtPeriod(Duration.standardMinutes(1)))
.withLateFirings(AtCount(1))))
.withAllowedLateness(Duration.standardMinutes(1)))
.apply(Sum.integersPerKey());

49. Jakarta
Windowed summation on a streaming engine with early and late firings
and allowed lateness

50. First trigger firing: [5, 8, 3]
Second trigger firing: [5, 8, 3, 15, 19, 23]
Third trigger firing: [5, 8, 3, 15, 19, 23, 9, 13, 10]
Jakarta
Accumulation Modes

51. First trigger firing: [5, 8, 3]
Second trigger firing: [15, 19, 23]
Third trigger firing: [9, 13, 10]
Jakarta
Discarding Modes

52. PCollection<KV<String, Integer>> scores = input
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)))
.triggering(AtWatermark()
.withEarlyFirings(
AtPeriod(Duration.standardMinutes(1)))
.withLateFirings(AtCount(1))))
.discardingFiredPanes())
.apply(Sum.integersPerKey());

53. Jakarta
Discarding mode version of early/late firings on a streaming engine

54. Jakarta
1. http://streamingsystems.org/Presentations/Jelena%20Pjesivac-grbo
vic.pdf
2. Stream Analytics with Google Cloud Dataflow: Use Cases &
Patterns, Gaurav Anand
3. Streaming 101 & 102, Tyler Akidau
4. https://streamingbook.net
5. Apache Beam Documentation
Google Slide version from this slide can be accessed from:
https://docs.google.com/presentation/d/1Ws73JxlVH39HiKiYuF3vW
903j8wFzxPQihXz4CQ_HZM/edit?usp=sharing
Credits to: