Talk of Samza new API in Cloud + Data Next Conference @Santa Clara '17

1. Yi Pan
Streams Team @LinkedIn
Committer and PMC Chair, Apache Samza
1

2. class PageKeyViewsCounterTask implements StreamTask, InitableTask {
public void process(IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator) {
GenericRecord record = ((GenericRecord) envelope.getMsg());
String pageKey = record.get("page-key").toString();
int newCount = pageKeyViews.get(pageKey).incrementAndGet();
collector.send(countStream, pageKey, newCount);
}
public void init(Config config, TaskContext context) {
pageKeyViews = (KeyValueStore<String, Counter>) context.getStore(“myPageKeyViews);
}
}
Task-0
Task-1
Task-2
Deployed via YARN

3.  Pros
◦ Simple API
◦ Built-in support for states
◦ Leverage YARN for fault-tolerance
◦ High performance (1.2 Mqps / host)
 Cons
◦ Not easy to write end-to-end processing pipeline in a single program
◦ Deployment is tightly coupled with YARN
◦ No support to run as batch job

4. • High-level API
• Flexible Deployment Model
• Convergence between Batch and Stream Processing
4

5. Application logic: Count PageViewEvent for each member in a 5 minute window
and send the counts to PageViewEventPerMemberStream
Re-partition by
memberId
window map sendTo
PageViewEvent
PageViewEventPerMembe
rStream
5

6. Re-partition window map sendTo
PageViewEvent
PageViewEventByMe
mberId
PageViewEventPerMembe
rStream
Job-1: PageViewRepartitionTask Job-2: PageViewByMemberIdCounterTask
Application in low-level API
6

7. • Job-1: Repartition job
public class PageViewRepartitionTask implements StreamTask {
private final SystemStream pageViewByMIDStream = new SystemStream("kafka", "PaveViewEventByMemberId");
@Override
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception {
PageViewEvent pve = (PageViewEvent) envelope.getMessage();
collector.send(new OutgoingMessageEnvelope(pageViewByMIDStream, pve.memberId, pve));
}
}
7

8. • Job-2: Window-based counter
public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask {
private final SystemStream pageViewCounterStream = new SystemStream("kafka", "PageViewEventPerMemberStream");
private KeyValueStore<String, PageViewPerMemberIdCounterEvent> windowedCounters;
private Long windowSize;
@Override
public void init(Config config, TaskContext context) throws Exception {
this.windowedCounters = (KeyValueStore<String, PageViewPerMemberIdCounterEvent>)
context.getStore("windowed-counter-store");
this.windowSize = config.getLong("task.window.ms");
}
@Override
public void window(MessageCollector collector, TaskCoordinator coordinator) throws Exception {
getWindowCounterEvent().forEach(counter ->
collector.send(new OutgoingMessageEnvelope(pageViewCounterStream, counter.memberId, counter)));
}
@Override
public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) throws Exception {
PageViewEvent pve = (PageViewEvent) envelope.getMessage();
countPageViewEvent(pve);
}
}
8

9. • Job-2: Window-based counter
public class PageViewByMemberIdCounterTask implements InitableTask, StreamTask, WindowableTask {
...
List<PageViewPerMemberIdCounterEvent> getWindowCounterEvent() {
List<PageViewPerMemberIdCounterEvent> retList = new ArrayList<>();
Long currentTimestamp = System.currentTimeMillis();
Long cutoffTimestamp = currentTimestamp - this.windowSize;
String lowerBound = String.format("%08d-", cutoffTimestamp);
String upperBound = String.format("%08d-", currentTimestamp + 1);
this.windowedCounters.range(lowerBound, upperBound).forEachRemaining(entry ->
retList.add(entry.getValue()));
return retList;
}
void countPageViewEvent(PageViewEvent pve) {
String key = String.format("%08d-%s", (pve.timestamp - pve.timestamp % this.windowSize), pve.memberId);
PageViewPerMemberIdCounterEvent counter = this.windowedCounters.get(key);
if (counter == null) {
counter = new PageViewPerMemberIdCounterEvent(pve.memberId, (pve.timestamp - pve.timestamp % this.windowSize), 0);
}
counter.count ++;
this.windowedCounters.put(key, counter);
}
}
9

10. • Samza High Level API (NEW)
– Ability to express a multi-stage processing pipeline in a single user
program
– Built-in library to provide high-level stream transformation functions
10

11. public class RepartitionAndCounterExample implements StreamApplication {
@Override public void init(StreamGraph graph, Config config) {
Supplier<Integer> initialValue = () -> 0;
MessageStream<PageViewEvent> pageViewEvents =
graph.getInputStream("pageViewEventStream", (k, m) -> (PageViewEvent) m);
OutputStream<String, MyStreamOutput, MyStreamOutput> pageViewEventPerMemberStream = graph
.getOutputStream("pageViewEventPerMemberStream", m -> m.memberId, m -> m);
pageViewEvents
.partitionBy(m -> m.memberId)
.window(Windows.keyedTumblingWindow(m -> m.memberId, Duration.ofMinutes(5), initialValue,
(m, c) -> c + 1))
.map(MyStreamOutput::new)
.sendTo(pageViewEventPerMemberStream);
}
}
Built-in transform functions
11

12. • Visualized execution plan
Visualization:
12

13. • Built-in transformation functions in high-level API
filter select a subset of messages from the stream
map map one input message to an output message
flatMap map one input message to 0 or more output messages
merge union all inputs into a single output stream
partitionBy re-partition the input messages based on a specific field
sendTo send the result to an output stream
sink send the result to an external system (e.g. external DB)
window window aggregation on the input stream
join join messages from two input streams
stateless
functions
I/O
functions
stateful
functions
13

14. • High-level API
• Flexible Deployment Model
• Convergence between Batch and Stream Processing
14

15.  Tight dependency on YARN
 Can’t easily port over to non-YARN clusters (e.g. Mesos, Kubernetes, AWS)
 Can’t directly embed stream processing in other application (eg. a web frontend)
15

16. • Flexible deployment of Samza applications
– Samza-as-a-library (NEW)
• Run embedded stream processing in a user program
• Zookeeper based coordination between multiple instances of user program
– Samza in a cluster
• Run stream processing as a managed program in a cluster (e.g.
SamzaContainer in YARN)
• Use the cluster manager (e.g. YARN) to provide deployment, coordination,
and resource management
16

17. Samza Job is composed of a collection of standalone processes
● Full control on
● Application’s life cycle
● Physical resource allocated to Samza processors
● Configuration and initialization
StreamProcessor
Samza
Container
Job
Coordinator
StreamProcessor
Samza
Container
Job
Coordinator
StreamProcessor
Samza
Container
Job
Coordinator...
Leader
17

18. ● ZooKeeper-based JobCoordinator (stateful use case)
● JobCoordinator uses ZooKeeper for leader election
● Leader will perform partition assignments among all active
StreamProcessors
ZooKeeper
StreamProcessor
Samza
Container
Job
Coordinator
StreamProcessor
Samza
Container
Job
Coordinator
StreamProcessor
Samza
Container
Job
Coordinator...
18

19. ● Embedded application code example
public class WikipediaZkLocalApplication {
/**
* Executes the application using the local application runner.
* It takes two required command line arguments
* config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name
* config-path: path to application properties
*
* @param args command line arguments
*/
public static void main(String[] args) {
CommandLine cmdLine = new CommandLine();
OptionSet options = cmdLine.parser().parse(args);
Config config = cmdLine.loadConfig(options);
LocalApplicationRunner runner = new LocalApplicationRunner(config);
WikipediaApplication app = new WikipediaApplication();
runner.run(app);
runner.waitForFinish();
}
}
19

20. ● Embedded application code example
public class WikipediaZkLocalApplication {
/**
* Executes the application using the local application runner.
* It takes two required command line arguments
* config-factory: a fully {@link org.apache.samza.config.factories.PropertiesConfigFactory} class name
* config-path: path to application properties
*
* @param args command line arguments
*/
public static void main(String[] args) {
CommandLine cmdLine = new CommandLine();
OptionSet options = cmdLine.parser().parse(args);
Config config = cmdLine.loadConfig(options);
LocalApplicationRunner runner = new LocalApplicationRunner(config);
WikipediaApplication app = new WikipediaApplication();
runner.run(app);
runner.waitForFinish();
}
}
20
job.coordinator.factory=org.apache.samza.zk.ZkJobCoordinatorFactory
job.coordinator.zk.connect=my-zk.server:2191

21. • Embedded application launch sequence
myApp.main()
Stream
Application
Local
Application
Runner
Stream
Processor
runner.run() streamProcessor.start()
n
21

22. • Cluster-based application launch sequence
run-app.sh
Remote
Application
Runner
JobRunnerjobRunner.run()
n
main()
app.class=my.app.MyStreamApplication
Yarn
RM
run-jc.sh
task.execute=run-local-app.sh
run-local-app.sh
Stream
Application
myApp.main()
Local
Application
Runner
Stream
Processor
runner.run() streamProcessor.start()
n
Job
Coordinator
22

23. 23

24. • High-level API
• Flexible Deployment Model
• Convergence between Batch and Stream Processing
24

25. Application logic: Count PageViewEvent for each member in a 5 minute window
and send the counts to PageViewEventPerMemberStream
Re-partition by
memberId
window map sendTo
PageViewEvent
PageViewEventPerMemb
erStream
HDFS
PageViewEvent: hdfs://mydbsnapshot/PageViewEvent/
PageViewEventPerMemberStream: hdfs://myoutputdb/PageViewEventPerMemberFiles
25

26. • No code change in application
streams.pageViewEventStream.system=kafka
streams.pageViewEventPerMemberStream.system=kafka
streams.pageViewEventStream.system=hdfs
streams.pageViewEventStream.physical.name=hdfs://mydbsnapshot/PageViewEvent/
streams.pageViewEventPerMemberStream.system=hdfs
streams.pageViewEventPerMemberStream.physical.name=hdfs://myoutputdb/PageViewEventPerMemberFiles
old config
new config
26

27. 27
High-level API
Unified Stream & Batch Processing
Remote Runner
Run in Remote Cluster
Cluster-based
Yarn, (Mesos)
Local Runner
Run Locally
Embedded
ZooKeeper, Standalone
APIRUNNERDEPLO
YMENT
PROCESSO
R
StreamProcessor
Streams
Kafka, Kinesis, HDFS ...
Local State
RocksDb, In-Memory
Remote Data
Multithreading
27

28.  Samza runner for Apache Beam
 Event-time processing
 Support for Exactly-once processing
 Support partition expansion for stateful application
 Easy access to Adjunct datasets
 SQL over Streams
28

29. Q&A
29