O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Ben Christensen
Developer – Edge Engineering at Netflix
@benjchristensen
http://techblog.netflix.com/
QConSF - November 2014...
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese an...
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practi...
RxJava
http://github.com/ReactiveX/RxJava
http://reactivex.io
Maven Central: 'io.reactivex:rxjava:1.0.+'
Iterable<T>
pull
Observable<T>
push
T next()
throws Exception
returns;
onNext(T)
onError(Exception)
onCompleted()
Iterable<T>
pull
Observable<T>
push
T next()
throws Exception
returns;
onNext(T)
onError(Exception)
onCompleted()
	
  //	
...
Iterable<T>
pull
Observable<T>
push
T next()
throws Exception
returns;
onNext(T)
onError(Exception)
onCompleted()
	
  //	
...
Single Multiple
Sync T getData()
Iterable<T> getData()
Stream<T> getData()
Async Future<T> getData() Observable<T> getData...
Observable.create(subscriber -> {
subscriber.onNext("Hello World!");
subscriber.onCompleted();
}).subscribe(System.out::pr...
Observable.create(subscriber -> {
subscriber.onNext("Hello");
subscriber.onNext("World!");
subscriber.onCompleted();
}).su...
// shorten by using helper method
Observable.just(“Hello”, “World!”)
.subscribe(System.out::println);
// add onError and onComplete listeners
Observable.just(“Hello”, “World!”)
.subscribe(System.out::println,
Throwable::prin...
// expand to show full classes
Observable.create(new OnSubscribe<String>() {
@Override
public void call(Subscriber<? super...
// add error propagation
Observable.create(subscriber -> {
try {
subscriber.onNext("Hello World!");
subscriber.onCompleted...
// add error propagation
Observable.create(subscriber -> {
try {
subscriber.onNext(throwException());
subscriber.onComplet...
// add error propagation
Observable.create(subscriber -> {
try {
subscriber.onNext("Hello World!");
subscriber.onNext(thro...
// add concurrency (manually)
Observable.create(subscriber -> {
new Thread(() -> {
try {
subscriber.onNext(getData());
sub...
// add concurrency (using a Scheduler)
Observable.create(subscriber -> {
try {
subscriber.onNext(getData());
subscriber.on...
// add operator
Observable.create(subscriber -> {
try {
subscriber.onNext(getData());
subscriber.onCompleted();
} catch (E...
// add error handling
Observable.create(subscriber -> {
try {
subscriber.onNext(getData());
subscriber.onCompleted();
} ca...
// infinite
Observable.create(subscriber -> {
int i=0;
while(!subscriber.isUnsubscribed()) {
subscriber.onNext(i++);
}
})....
Hot Cold
emits whether you’re ready or not
examples
mouse and keyboard events
system events
stock prices
emits when reques...
Hot Cold
emits whether you’re ready or not
examples
mouse and keyboard events
system events
stock prices
emits when reques...
Abstract Concurrency
Cold Finite Streams
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
 Observable<R>	
  b	
  =	
  Observable<T>.flatMap({	
  T	
  t	
  -­‐>	
  	
  
	
  	
  	
  	
  Observable<R>	
  r	
  =	
  ....
 Observable<R>	
  b	
  =	
  Observable<T>.flatMap({	
  T	
  t	
  -­‐>	
  	
  
	
  	
  	
  	
  Observable<R>	
  r	
  =	
  ....
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
 	
  	
  	
  Observable.zip(a,	
  b,	
  (a,	
  b)	
  -­‐>	
  {	
  	
  
	
  	
  	
  	
  	
  	
  ...	
  operate	
  on	
  val...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
class	
  VideoService	
  {	
  
	
  	
  	
  def	
  VideoList	
  getPersonalizedListOfMovies(userId);	
  
	
  	
  	
  def	
 ...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
Non-Opinionated Concurrency
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) {
// first reques...
Decouples Consumption from Production
// first request User object
return getUser(request.getQueryParameters().get("userId...
// first request User object
return getUser(request.getQueryParameters().get("userId")).flatMap(user -> {
// then fetch pe...
// first request User object
return getUser(request.getQueryParameters().get("userId")).flatMap(user -> {
// then fetch pe...
Decouples Consumption from Production
Event Loop
API Request
API Request
API Request
API Request
API Request
Dependency
RE...
// first request User object
return getUser(request.getQueryParameters().get("userId")).flatMap(user -> {
// then fetch pe...
Clear API Communicates Potential Cost
class	
  VideoService	
  {	
  
	
  	
  	
  def	
  Observable<VideoList>	
  getPerson...
class	
  VideoService	
  {	
  
	
  	
  	
  def	
  Observable<VideoList>	
  getPersonalizedListOfMovies(userId);	
  
	
  	
...
class	
  VideoService	
  {	
  
	
  	
  	
  def	
  Observable<VideoList>	
  getPersonalizedListOfMovies(userId);	
  
	
  	
...
Retrieval, Transformation, Combination
all done in same declarative manner
What about … ?
Error Handling
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).onErrorResumeNext(throwable -> {
return Observ...
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).onErrorReturn(throwable -> {
return "fallback ...
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).retryWhen(attempts -> {
return attempts.zipWit...
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).retryWhen(attempts -> {
return attempts.zipWit...
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).retryWhen(attempts -> {
return attempts.zipWit...
Observable.create(subscriber -> {
throw new RuntimeException("failed!");
}).retryWhen(attempts -> {
return attempts.zipWit...
Concurrency
Concurrency
an Observable is sequential
(no concurrent emissions)
scheduling and combining Observables
enables concurrency...
// merging async Observables allows each
// to execute concurrently
Observable.merge(getDataAsync(1), getDataAsync(2))
mer...
// concurrently fetch data for 5 items
Observable.range(0, 5).flatMap(i -> {
return getDataAsync(i);
})
Observable.range(0, 5000).window(500).flatMap(work -> {
return work.observeOn(Schedulers.computation())
.map(item -> {
// ...
Observable.range(0, 5000).buffer(500).flatMap(is -> {
return Observable.from(is).subscribeOn(Schedulers.computation())
.ma...
Flow Control
Flow Control
(backpressure)
no backpressure needed
Observable.from(iterable).take(1000).map(i -> "value_" + i).subscribe(System.out::println);
Observable.from(iterable).take(1000).map(i -> "value_" + i).subscribe(System.out::println);
no backpressure needed
synchro...
Observable.from(iterable).take(1000).map(i -> "value_" + i)
.observeOn(Schedulers.computation()).subscribe(System.out::pri...
Observable.from(iterable).take(1000).map(i -> "value_" + i)
.observeOn(Schedulers.computation()).subscribe(System.out::pri...
Flow Control Options
Hot Cold
emits whether you’re ready or not
examples
mouse and keyboard events
system events
stock prices
emits when reques...
Block
(callstack blocking and/or park the thread)
Hot or Cold Streams
Temporal Operators
(batch or drop data using time)
Hot Streams
Observable.range(1, 1000000).sample(10, TimeUnit.MILLISECONDS).forEach(System.out::println);
110584
242165
544453
942880
Observable.range(1, 1000000).throttleFirst(10, TimeUnit.MILLISECONDS).forEach(System.out::println);
1
55463
163962
308545
...
Observable.range(1, 1000000).debounce(10, TimeUnit.MILLISECONDS).forEach(System.out::println);
1000000
Observable.range(1, 1000000).buffer(10, TimeUnit.MILLISECONDS)
.toBlocking().forEach(list -> System.out.println("batch: " ...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
/* The following will emit a buffered list as it is debounced */
// first we multicast the stream ... using refCount so it...
Observable.range(1, 1000000).window(50, TimeUnit.MILLISECONDS)
.flatMap(window -> window.count())
.toBlocking().forEach(co...
Observable.range(1, 1000000).window(500000)
.flatMap(window -> window.count())
.toBlocking().forEach(count -> System.out.p...
Reactive Pull
(dynamic push-pull)
Push (reactive) when consumer
keeps up with producer.
Switch to Pull (interactive)
when consumer is slow.
Bound all* queue...
*vertically, not horizontally
Push (reactive) when consumer
keeps up with producer.
Switch to Pull (interactive)
when cons...
Reactive Pull
hot vs cold
Reactive Pull
cold supports pull
Observable.from(iterable)
Observable.from(0, 100000)
Cold Streams
emits when requested
(generally at controlled rate)
exam...
Observable.from(iterable)
Observable.from(0, 100000)
Cold Streams
emits when requested
(generally at controlled rate)
exam...
Reactive Pull
hot receives signal
Reactive Pull
hot receives signal
*including Observables that don’t
implement reactive pull support
hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
hotSourceStream.onBackpressureDrop().observeOn(aScheduler);
stream.onBackpressure(strategy).subscribe
Hot Infinite Streams
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
(movieId)
movieId=12345 movieId=34567
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
Stage 1 Stage 2
groupBy
Event Streams
sink
Stage 1 Stage 2
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
12345
56789
34567
Stage 1 Stage 2
12345
56789
34567
12345
56789
34567
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
10mins || 1000
4:40-4:50pm
4:50-5:00pm
5:00-5:07pm
(burst to 1000)
5:07-5:17pm
10mins || 1000
4:40-4:50pm
4:50-5:00pm
5:00-5:07pm
(burst to 1000)
5:07-5:17pm
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
MantisJob
.source(NetflixSources.moviePlayAttempts())
.stage(playAttempts -> {
return playAttempts.groupBy(playAttempt -> ...
stream.onBackpressure(strategy?).subscribe
stream.onBackpressure(buffer).subscribe
stream.onBackpressure(drop).subscribe
stream.onBackpressure(sample).subscribe
stream.onBackpressure(scaleHorizontally).subscribe
Reactive-Streams
https://github.com/reactive-streams/reactive-streams
Reactive-Streams
https://github.com/reactive-streams/reactive-streams
https://github.com/ReactiveX/RxJavaReactiveStreams
final ActorSystem system = ActorSystem.create("InteropTest");
final FlowMaterializer mat = FlowMaterializer.create(system)...
final ActorSystem system = ActorSystem.create("InteropTest");
final FlowMaterializer mat = FlowMaterializer.create(system)...
// RxJava Observable
Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000)
.grou...
// RxJava Observable
Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000)
.grou...
compile 'io.reactivex:rxjava:1.0.+'
compile 'io.reactivex:rxjava-reactive-streams:0.3.0'
compile 'io.ratpack:ratpack-rx:0....
compile 'io.reactivex:rxjava:1.0.+'
compile 'io.reactivex:rxjava-reactive-streams:0.3.0'
compile 'io.ratpack:ratpack-rx:0....
RxJava 1.0 Final
November 18th
Mental Shift
imperative → functional
sync → async
pull → push
Concurrency and async are non-trivial.
Rx doesn’t trivialize it.
Rx is powerful and rewards those
who go through the learn...
Single Multiple
Sync T getData()
Iterable<T> getData()
Stream<T> getData()
Async Future<T> getData() Observable<T> getData...
Abstract Concurrency
Non-Opinionated Concurrency
Decouple Production from Consumption
Powerful Composition
of Nested, Conditional Flows
First-class Support of
Error Handling, Scheduling
& Flow Control
RxJava
http://github.com/ReactiveX/RxJava
http://reactivex.io
Reactive Programming in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html
Optimizing th...
Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/rx-
service-architecture
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
 Reactive Programming with Rx
Próximos SlideShares
Carregando em…5
×

Reactive Programming with Rx

2.596 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1DXFg0h.

Ben Christensen summarizes why the Rx programming model was chosen and demonstrates how it is applied to a variety of use cases. Filmed at qconsf.com.

Ben Christensen is a software engineer on the Netflix Edge Services Platform team responsible for fault tolerance, performance, architecture and scale while enabling millions of customers to access the Netflix experience across more than 1,000 different device types.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Reactive Programming with Rx

  1. 1. Ben Christensen Developer – Edge Engineering at Netflix @benjchristensen http://techblog.netflix.com/ QConSF - November 2014 ReactiveProgrammingwithRx
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /rx-service-architecture
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  4. 4. RxJava http://github.com/ReactiveX/RxJava http://reactivex.io Maven Central: 'io.reactivex:rxjava:1.0.+'
  5. 5. Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted()
  6. 6. Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted()  //  Iterable<String>  or  Stream<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .limit(5)      .map(s  -­‐>  s  +  "_transformed")      .forEach(t  -­‐>  System.out.println("onNext  =>  "  +  t))    //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map(s  -­‐>  s  +  "_transformed")      .forEach(t  -­‐>  System.out.println("onNext  =>  "  +  t))  
  7. 7. Iterable<T> pull Observable<T> push T next() throws Exception returns; onNext(T) onError(Exception) onCompleted()  //  Observable<String>      //  that  emits  75  Strings    getDataFromNetwork()      .skip(10)      .take(5)      .map(s  -­‐>  s  +  "_transformed")      .forEach(t  -­‐>  System.out.println("onNext  =>  "  +  t))    //  Iterable<String>  or  Stream<String>      //  that  contains  75  Strings    getDataFromLocalMemory()      .skip(10)      .limit(5)      .map(s  -­‐>  s  +  "_transformed")      .forEach(t  -­‐>  System.out.println("onNext  =>  "  +  t))  
  8. 8. Single Multiple Sync T getData() Iterable<T> getData() Stream<T> getData() Async Future<T> getData() Observable<T> getData()
  9. 9. Observable.create(subscriber -> { subscriber.onNext("Hello World!"); subscriber.onCompleted(); }).subscribe(System.out::println);
  10. 10. Observable.create(subscriber -> { subscriber.onNext("Hello"); subscriber.onNext("World!"); subscriber.onCompleted(); }).subscribe(System.out::println);
  11. 11. // shorten by using helper method Observable.just(“Hello”, “World!”) .subscribe(System.out::println);
  12. 12. // add onError and onComplete listeners Observable.just(“Hello”, “World!”) .subscribe(System.out::println, Throwable::printStackTrace, () -> System.out.println("Done"));
  13. 13. // expand to show full classes Observable.create(new OnSubscribe<String>() { @Override public void call(Subscriber<? super String> subscriber) { subscriber.onNext("Hello World!"); subscriber.onCompleted(); } }).subscribe(new Subscriber<String>() { @Override public void onCompleted() { System.out.println("Done"); } @Override public void onError(Throwable e) { e.printStackTrace(); } @Override public void onNext(String t) { System.out.println(t); } });
  14. 14. // add error propagation Observable.create(subscriber -> { try { subscriber.onNext("Hello World!"); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println);
  15. 15. // add error propagation Observable.create(subscriber -> { try { subscriber.onNext(throwException()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println);
  16. 16. // add error propagation Observable.create(subscriber -> { try { subscriber.onNext("Hello World!"); subscriber.onNext(throwException()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribe(System.out::println);
  17. 17. // add concurrency (manually) Observable.create(subscriber -> { new Thread(() -> { try { subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).start(); }).subscribe(System.out::println); ?
  18. 18. // add concurrency (using a Scheduler) Observable.create(subscriber -> { try { subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .subscribe(System.out::println); ? Learn more about parameterized concurrency and virtual time with Rx Schedulers at https://github.com/ReactiveX/RxJava/wiki/Scheduler
  19. 19. // add operator Observable.create(subscriber -> { try { subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .map(data -> data + " --> at " + System.currentTimeMillis()) .subscribe(System.out::println); ?
  20. 20. // add error handling Observable.create(subscriber -> { try { subscriber.onNext(getData()); subscriber.onCompleted(); } catch (Exception e) { subscriber.onError(e); } }).subscribeOn(Schedulers.io()) .map(data -> data + " --> at " + System.currentTimeMillis()) .onErrorResumeNext(e -> Observable.just("Fallback Data")) .subscribe(System.out::println);
  21. 21. // infinite Observable.create(subscriber -> { int i=0; while(!subscriber.isUnsubscribed()) { subscriber.onNext(i++); } }).subscribe(System.out::println); Note: No backpressure support here. See Observable.from(Iterable) or Observable.range() for actual implementations
  22. 22. Hot Cold emits whether you’re ready or not examples mouse and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data })
  23. 23. Hot Cold emits whether you’re ready or not examples mouse and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data }) flow control flow control & backpressure
  24. 24. Abstract Concurrency
  25. 25. Cold Finite Streams
  26. 26. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  27. 27. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  28. 28. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  29. 29.  Observable<R>  b  =  Observable<T>.flatMap({  T  t  -­‐>            Observable<R>  r  =  ...  transform  t  ...          return  r;    }) flatMap
  30. 30.  Observable<R>  b  =  Observable<T>.flatMap({  T  t  -­‐>            Observable<R>  r  =  ...  transform  t  ...          return  r;    }) flatMap
  31. 31. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  32. 32. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  33. 33. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  34. 34. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  35. 35. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  36. 36. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  37. 37. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  38. 38. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  39. 39. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  40. 40. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  41. 41.        Observable.zip(a,  b,  (a,  b)  -­‐>  {                ...  operate  on  values  from  both  a  &  b  ...              return  Arrays.asList(a,  b);          }) zip { ( , ) }
  42. 42. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  43. 43. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  44. 44. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  45. 45. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  46. 46. class  VideoService  {        def  VideoList  getPersonalizedListOfMovies(userId);        def  VideoBookmark  getBookmark(userId,  videoId);        def  VideoRating  getRating(userId,  videoId);        def  VideoMetadata  getMetadata(videoId);   } class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   } ...createanobservableapi: insteadofablockingapi...
  47. 47. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } 1 2 3 4 5 6
  48. 48. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); } 1 2 3 4 5 6
  49. 49. Non-Opinionated Concurrency
  50. 50. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  51. 51. public Observable<Void> handle(HttpServerRequest<ByteBuf> request, HttpServerResponse<ByteBuf> response) { // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }).flatMap(data -> { // output as SSE as we get back the data (no waiting until all is done) return response.writeAndFlush(new ServerSentEvent(SimpleJson.mapToJson(data))); }); }
  52. 52. Decouples Consumption from Production // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); })
  53. 53. // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) Decouples Consumption from Production
  54. 54. // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) Decouples Consumption from Production
  55. 55. Decouples Consumption from Production Event Loop API Request API Request API Request API Request API Request Dependency REST API new Command().observe() new Command(). observe() new Command(). observe() new Command(). observe() new Command(). observe() Collapser
  56. 56. // first request User object return getUser(request.getQueryParameters().get("userId")).flatMap(user -> { // then fetch personal catalog Observable<Map<String, Object>> catalog = getPersonalizedCatalog(user) .flatMap(catalogList -> { return catalogList.videos().<Map<String, Object>> flatMap(video -> { Observable<Bookmark> bookmark = getBookmark(video); Observable<Rating> rating = getRating(video); Observable<VideoMetadata> metadata = getMetadata(video); return Observable.zip(bookmark, rating, metadata, (b, r, m) -> { return combineVideoData(video, b, r, m); }); }); }); // and fetch social data in parallel Observable<Map<String, Object>> social = getSocialData(user).map(s -> { return s.getDataAsMap(); }); // merge the results return Observable.merge(catalog, social); }) ~5 network calls (#3 and #4 may result in more due to windowing) 1 2 3 4 5
  57. 57. Clear API Communicates Potential Cost class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   }
  58. 58. class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   } Implementation Can Differ BIO Network Call Local Cache Collapsed Network Call
  59. 59. class  VideoService  {        def  Observable<VideoList>  getPersonalizedListOfMovies(userId);        def  Observable<VideoBookmark>  getBookmark(userId,  videoId);        def  Observable<VideoRating>  getRating(userId,  videoId);        def  Observable<VideoMetadata>  getMetadata(videoId);   } Implementation Can Differ and Change Local Cache Collapsed Network CallCollapsed Network Call BIO NIO Network Call
  60. 60. Retrieval, Transformation, Combination all done in same declarative manner
  61. 61. What about … ?
  62. 62. Error Handling
  63. 63. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).onErrorResumeNext(throwable -> { return Observable.just("fallback value"); }).subscribe(System.out::println);
  64. 64. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).onErrorReturn(throwable -> { return "fallback value"; }).subscribe(System.out::println);
  65. 65. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace());
  66. 66. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace());
  67. 67. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace());
  68. 68. Observable.create(subscriber -> { throw new RuntimeException("failed!"); }).retryWhen(attempts -> { return attempts.zipWith(Observable.range(1, 3), (throwable, i) -> i) .flatMap(i -> { System.out.println("delay retry by " + i + " second(s)"); return Observable.timer(i, TimeUnit.SECONDS); }).concatWith(Observable.error(new RuntimeException("Exceeded 3 retries"))); }) .subscribe(System.out::println, t -> t.printStackTrace());
  69. 69. Concurrency
  70. 70. Concurrency an Observable is sequential (no concurrent emissions) scheduling and combining Observables enables concurrency while retaining sequential emission
  71. 71. // merging async Observables allows each // to execute concurrently Observable.merge(getDataAsync(1), getDataAsync(2)) merge
  72. 72. // concurrently fetch data for 5 items Observable.range(0, 5).flatMap(i -> { return getDataAsync(i); })
  73. 73. Observable.range(0, 5000).window(500).flatMap(work -> { return work.observeOn(Schedulers.computation()) .map(item -> { // simulate computational work try { Thread.sleep(1); } catch (Exception e) {} return item + " processed " + Thread.currentThread(); }); })
  74. 74. Observable.range(0, 5000).buffer(500).flatMap(is -> { return Observable.from(is).subscribeOn(Schedulers.computation()) .map(item -> { // simulate computational work try { Thread.sleep(1); } catch (Exception e) {} return item + " processed " + Thread.currentThread(); }); })
  75. 75. Flow Control
  76. 76. Flow Control (backpressure)
  77. 77. no backpressure needed Observable.from(iterable).take(1000).map(i -> "value_" + i).subscribe(System.out::println);
  78. 78. Observable.from(iterable).take(1000).map(i -> "value_" + i).subscribe(System.out::println); no backpressure needed synchronous on same thread (no queueing)
  79. 79. Observable.from(iterable).take(1000).map(i -> "value_" + i) .observeOn(Schedulers.computation()).subscribe(System.out::println); backpressure needed
  80. 80. Observable.from(iterable).take(1000).map(i -> "value_" + i) .observeOn(Schedulers.computation()).subscribe(System.out::println); backpressure needed asynchronous (queueing)
  81. 81. Flow Control Options
  82. 82. Hot Cold emits whether you’re ready or not examples mouse and keyboard events system events stock prices emits when requested (generally at controlled rate) examples database query web service request reading file Observable.create(subscriber -> { // register with data source }) Observable.create(subscriber -> { // fetch data }) flow control flow control & backpressure
  83. 83. Block (callstack blocking and/or park the thread) Hot or Cold Streams
  84. 84. Temporal Operators (batch or drop data using time) Hot Streams
  85. 85. Observable.range(1, 1000000).sample(10, TimeUnit.MILLISECONDS).forEach(System.out::println); 110584 242165 544453 942880
  86. 86. Observable.range(1, 1000000).throttleFirst(10, TimeUnit.MILLISECONDS).forEach(System.out::println); 1 55463 163962 308545 457445 592638 751789 897159
  87. 87. Observable.range(1, 1000000).debounce(10, TimeUnit.MILLISECONDS).forEach(System.out::println); 1000000
  88. 88. Observable.range(1, 1000000).buffer(10, TimeUnit.MILLISECONDS) .toBlocking().forEach(list -> System.out.println("batch: " + list.size())); batch: 71141 batch: 49488 batch: 141147 batch: 141432 batch: 195920 batch: 240462 batch: 160410
  89. 89. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] []
  90. 90. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] []
  91. 91. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] []
  92. 92. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] []
  93. 93. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] []
  94. 94. /* The following will emit a buffered list as it is debounced */ // first we multicast the stream ... using refCount so it handles the subscribe/unsubscribe Observable<Integer> burstStream = intermittentBursts().take(20).publish().refCount(); // then we get the debounced version Observable<Integer> debounced = burstStream.debounce(10, TimeUnit.MILLISECONDS); // then the buffered one that uses the debounced stream to demark window start/stop Observable<List<Integer>> buffered = burstStream.buffer(debounced); // then we subscribe to the buffered stream so it does what we want buffered.toBlocking().forEach(System.out::println); [0, 1, 2] [0, 1, 2] [0, 1, 2, 3, 4, 5, 6] [0, 1, 2, 3, 4] [0, 1] [] https://gist.github.com/benjchristensen/e4524a308456f3c21c0b
  95. 95. Observable.range(1, 1000000).window(50, TimeUnit.MILLISECONDS) .flatMap(window -> window.count()) .toBlocking().forEach(count -> System.out.println("num items: " + count)); num items: 477769 num items: 155463 num items: 366768
  96. 96. Observable.range(1, 1000000).window(500000) .flatMap(window -> window.count()) .toBlocking().forEach(count -> System.out.println("num items: " + count)); num items: 500000 num items: 500000
  97. 97. Reactive Pull (dynamic push-pull)
  98. 98. Push (reactive) when consumer keeps up with producer. Switch to Pull (interactive) when consumer is slow. Bound all* queues.
  99. 99. *vertically, not horizontally Push (reactive) when consumer keeps up with producer. Switch to Pull (interactive) when consumer is slow. Bound all* queues.
  100. 100. Reactive Pull hot vs cold
  101. 101. Reactive Pull cold supports pull
  102. 102. Observable.from(iterable) Observable.from(0, 100000) Cold Streams emits when requested (generally at controlled rate) examples database query web service request reading file
  103. 103. Observable.from(iterable) Observable.from(0, 100000) Cold Streams emits when requested (generally at controlled rate) examples database query web service request reading file Pull
  104. 104. Reactive Pull hot receives signal
  105. 105. Reactive Pull hot receives signal *including Observables that don’t implement reactive pull support
  106. 106. hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
  107. 107. hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
  108. 108. hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
  109. 109. hotSourceStream.onBackpressureBuffer().observeOn(aScheduler);
  110. 110. hotSourceStream.onBackpressureDrop().observeOn(aScheduler);
  111. 111. stream.onBackpressure(strategy).subscribe
  112. 112. Hot Infinite Streams
  113. 113. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  114. 114. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  115. 115. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here) Hot Infinite Stream
  116. 116. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  117. 117. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  118. 118. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  119. 119. (movieId) movieId=12345 movieId=34567
  120. 120. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  121. 121. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  122. 122. Stage 1 Stage 2 groupBy Event Streams sink
  123. 123. Stage 1 Stage 2 12345 56789 34567
  124. 124. Stage 1 Stage 2 12345 56789 34567
  125. 125. Stage 1 Stage 2 12345 56789 34567
  126. 126. Stage 1 Stage 2 12345 56789 34567
  127. 127. Stage 1 Stage 2 12345 56789 34567 12345 56789 34567
  128. 128. Stage 1 Stage 2 12345 56789 34567 12345 56789 34567
  129. 129. Stage 1 Stage 2 12345 56789 34567 12345 56789 34567
  130. 130. Stage 1 Stage 2 12345 56789 34567 12345 56789 34567
  131. 131. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  132. 132. 10mins || 1000 4:40-4:50pm 4:50-5:00pm 5:00-5:07pm (burst to 1000) 5:07-5:17pm
  133. 133. 10mins || 1000 4:40-4:50pm 4:50-5:00pm 5:00-5:07pm (burst to 1000) 5:07-5:17pm
  134. 134. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  135. 135. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  136. 136. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  137. 137. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  138. 138. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  139. 139. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  140. 140. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  141. 141. MantisJob .source(NetflixSources.moviePlayAttempts()) .stage(playAttempts -> { return playAttempts.groupBy(playAttempt -> { return playAttempt.getMovieId(); }) }) .stage(playAttemptsByMovieId -> { playAttemptsByMovieId .window(10,TimeUnit.MINUTES, 1000) // buffer for 10 minutes, or 1000 play attempts .flatMap(windowOfPlayAttempts -> { return windowOfPlayAttempts .reduce(new FailRatioExperiment(playAttemptsByMovieId.getKey()), (experiment, playAttempt) -> { experiment.updateFailRatio(playAttempt); experiment.updateExamples(playAttempt); return experiment; }).doOnNext(experiment -> { logToHistorical("Play attempt experiment", experiment.getId(),experiment); // log for offline analysis }).filter(experiment -> { return experiment.failRatio() >= DYNAMIC_PROP("fail_threshold").get(); }).map(experiment -> { return new FailReport(experiment, runCorrelations(experiment.getExamples())); }).doOnNext(report -> { logToHistorical("Failure report", report.getId(), report); // log for offline analysis }) }) }) .sink(Sinks.emailAlert(report -> { return toEmail(report)})) // anomalies trigger events (simple email here)
  142. 142. stream.onBackpressure(strategy?).subscribe
  143. 143. stream.onBackpressure(buffer).subscribe
  144. 144. stream.onBackpressure(drop).subscribe
  145. 145. stream.onBackpressure(sample).subscribe
  146. 146. stream.onBackpressure(scaleHorizontally).subscribe
  147. 147. Reactive-Streams https://github.com/reactive-streams/reactive-streams
  148. 148. Reactive-Streams https://github.com/reactive-streams/reactive-streams https://github.com/ReactiveX/RxJavaReactiveStreams
  149. 149. final ActorSystem system = ActorSystem.create("InteropTest"); final FlowMaterializer mat = FlowMaterializer.create(system); // RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // convert to Akka Streams Source and transform using Akka Streams ‘map’ and ‘take’ operators Source<String> stringSource = Source.from(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Akka to Rx Observable return RxReactiveStreams.toObservable(stringSource.runWith(Sink.<String> fanoutPublisher(1, 1), mat)); }); strings.toBlocking().forEach(System.out::println); system.shutdown(); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'com.typesafe.akka:akka-stream-experimental_2.11:0.10-M1'
  150. 150. final ActorSystem system = ActorSystem.create("InteropTest"); final FlowMaterializer mat = FlowMaterializer.create(system); // RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // convert to Akka Streams Source and transform using Akka Streams ‘map’ and ‘take’ operators Source<String> stringSource = Source.from(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Akka to Rx Observable return RxReactiveStreams.toObservable(stringSource.runWith(Sink.<String> fanoutPublisher(1, 1), mat)); }); strings.toBlocking().forEach(System.out::println); system.shutdown(); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'com.typesafe.akka:akka-stream-experimental_2.11:0.10-M1'
  151. 151. // RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // Convert to Reactor Stream and transform using Reactor Stream ‘map’ and ‘take’ operators Stream<String> linesStream = Streams.create(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Reactor Stream to Rx Observable return RxReactiveStreams.toObservable(linesStream); }); strings.toBlocking().forEach(System.out::println); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'org.projectreactor:reactor-core:2.0.0.M1'
  152. 152. // RxJava Observable Observable<GroupedObservable<Boolean, Integer>> oddAndEvenGroups = Observable.range(1, 1000000) .groupBy(i -> i % 2 == 0) .take(2); Observable<String> strings = oddAndEvenGroups.<String> flatMap(group -> { // schedule odd and even on different event loops Observable<Integer> asyncGroup = group.observeOn(Schedulers.computation()); // convert to Reactive Streams Publisher Publisher<Integer> groupPublisher = RxReactiveStreams.toPublisher(asyncGroup); // Convert to Reactor Stream and transform using Reactor Stream ‘map’ and ‘take’ operators Stream<String> linesStream = Streams.create(groupPublisher).map(i -> i + " " + group.getKey()).take(2000); // convert back from Reactor Stream to Rx Observable return RxReactiveStreams.toObservable(linesStream); }); strings.toBlocking().forEach(System.out::println); compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'org.projectreactor:reactor-core:2.0.0.M1'
  153. 153. compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'io.ratpack:ratpack-rx:0.9.10' try { RatpackServer server = EmbeddedApp.fromHandler(ctx -> { Observable<String> o1 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "A " + i; }); Observable<String> o2 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "B " + i; }); Observable<String> o = Observable.merge(o1, o2); ctx.render( ServerSentEvents.serverSentEvents(RxReactiveStreams.toPublisher(o), e -> e.event("counter").data("event " + e.getItem())) ); }).getServer(); server.start(); System.out.println("Port: " + server.getBindPort()); } catch (Exception e) { e.printStackTrace(); }
  154. 154. compile 'io.reactivex:rxjava:1.0.+' compile 'io.reactivex:rxjava-reactive-streams:0.3.0' compile 'io.ratpack:ratpack-rx:0.9.10' try { RatpackServer server = EmbeddedApp.fromHandler(ctx -> { Observable<String> o1 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "A " + i; }); Observable<String> o2 = Observable.range(0, 2000) .observeOn(Schedulers.computation()).map(i -> { return "B " + i; }); Observable<String> o = Observable.merge(o1, o2); ctx.render( ServerSentEvents.serverSentEvents(RxReactiveStreams.toPublisher(o), e -> e.event("counter").data("event " + e.getItem())) ); }).getServer(); server.start(); System.out.println("Port: " + server.getBindPort()); } catch (Exception e) { e.printStackTrace(); }
  155. 155. RxJava 1.0 Final November 18th
  156. 156. Mental Shift imperative → functional sync → async pull → push
  157. 157. Concurrency and async are non-trivial. Rx doesn’t trivialize it. Rx is powerful and rewards those who go through the learning curve.
  158. 158. Single Multiple Sync T getData() Iterable<T> getData() Stream<T> getData() Async Future<T> getData() Observable<T> getData()
  159. 159. Abstract Concurrency
  160. 160. Non-Opinionated Concurrency
  161. 161. Decouple Production from Consumption
  162. 162. Powerful Composition of Nested, Conditional Flows
  163. 163. First-class Support of Error Handling, Scheduling & Flow Control
  164. 164. RxJava http://github.com/ReactiveX/RxJava http://reactivex.io
  165. 165. Reactive Programming in the Netflix API with RxJava http://techblog.netflix.com/2013/02/rxjava-netflix-api.html Optimizing the Netflix API http://techblog.netflix.com/2013/01/optimizing-netflix-api.html Reactive Extensions (Rx) http://www.reactivex.io Reactive Streams https://github.com/reactive-streams/reactive-streams Ben Christensen @benjchristensen RxJava https://github.com/ReactiveX/RxJava @RxJava RxJS http://reactive-extensions.github.io/RxJS/ @ReactiveX jobs.netflix.com
  166. 166. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/rx- service-architecture

×