O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Reactive Stream Processing with Akka Streams

13.053 visualizações

Publicada em

Reactive Stream Processing with Akka Streams

  1. 1. Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Konrad `@ktosopl` Malawski streams reactive stream processing with
  2. 2. Konrad `@ktosopl` Malawski Akka Team Reactive Streams TCK sbt-jmh … hAkker @
  3. 3. Konrad `@ktosopl` Malawski typesafe.com geecon.org Java.pl / KrakowScala.pl sckrk.com / meetup.com/Paper-Cup @ London GDGKrakow.pl meetup.com/Lambda-Lounge-Krakow hAkker @
  4. 4. Agenda
  5. 5. Agenda • Reactive Streams • Background and Specification • Protocol details • Akka Streams • Concepts and goals • Building Blocks • Akka Streams in Action • Q & A
  6. 6. Streams
  7. 7. Streams
  8. 8. Streams “You cannot enter the same river twice” ~ Heraclitus http://en.wikiquote.org/wiki/Heraclitus
  9. 9. Streams Real Time Stream Processing When you attach “late” to a Publisher, you may miss initial elements – it’s a river of data. http://en.wikiquote.org/wiki/Heraclitus
  10. 10. Reactive Streams
  11. 11. Reactive Streams Stream processing
  12. 12. Reactive Streams Back-pressured Stream processing
  13. 13. Reactive Streams Back-pressured Asynchronous Stream processing
  14. 14. Reactive Streams Back-pressured Asynchronous Stream processing Standardised (!)
  15. 15. Reactive Streams: Goals 1. Back-pressured Asynchronous Stream processing 2. Standard implemented by many libraries
  16. 16. Reactive Streams - Specification & TCK http://reactive-streams.org
  17. 17. Reactive Streams - Who? http://reactive-streams.org Kaazing Corp. rxJava @ Netflix, reactor @ Pivotal (SpringSource), vert.x @ Red Hat, Twitter, akka-streams @ Typesafe, spray @ Spray.io, Oracle, java (?) – Doug Lea - SUNY Oswego …
  18. 18. Reactive Streams - Inter-op http://reactive-streams.org We want to make different implementations co-operate with each other.
  19. 19. Reactive Streams - Inter-op http://reactive-streams.org The different implementations “talk to each other” using the Reactive Streams protocol.
  20. 20. Reactive Streams - Inter-op http://reactive-streams.org The Reactive Streams SPI is NOT meant to be user-api. You should use one of the implementing libraries.
  21. 21. package com.rolandkuhn.rsinterop import ratpack.rx.RxRatpack import ratpack.test.embed.EmbeddedApp import ratpack.handling.Handler import ratpack.handling.Context import rx.Observable import scala.collection.JavaConverters._ import akka.stream.scaladsl.Flow import akka.stream.scaladsl.Source import rx.RxReactiveStreams import akka.stream.scaladsl.Sink import akka.actor.ActorSystem import akka.stream.FlowMaterializer import ratpack.http.ResponseChunks import java.util.function.Consumer import ratpack.test.http.TestHttpClient import reactor.rx.Streams object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) Reactive Streams - Inter-op
  22. 22. EmbeddedApp.fromHandler(new Handler { override def handle(ctx: Context): Unit = { // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs) // Akka Streams Source val stringSource = Source(intPub).map(_.toString) // Reactive Streams Publisher val stringPub = stringSource.runWith(Sink.fanoutPublisher(1, 1)) // Reactor Stream val linesStream = Streams.create(stringPub).map[String](new reactor.function.Function[String, String] { override def apply(in: String) = in + "n" }) // and now render the HTTP response (RatPack) ctx.render(ResponseChunks.stringChunks(linesStream)) } }).test(new Consumer[TestHttpClient] { override def accept(client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } }) } Reactive Streams - Inter-op
  23. 23. What is back-pressure?
  24. 24. Back-pressure? Example Without Publisher[T] Subscriber[T]
  25. 25. Back-pressure? Example Without Fast Publisher Slow Subscriber
  26. 26. Back-pressure? “Why would I need that!?”
  27. 27. Back-pressure? Push + NACK model
  28. 28. Back-pressure? Push + NACK model Subscriber usually has some kind of buffer.
  29. 29. Back-pressure? Push + NACK model
  30. 30. Back-pressure? Push + NACK model
  31. 31. Back-pressure? Push + NACK model What if the buffer overflows?
  32. 32. Back-pressure? Push + NACK model (a) Use bounded buffer, drop messages + require re-sending
  33. 33. Back-pressure? Push + NACK model (a) Kernel does this! Routers do this! (TCP) Use bounded buffer, drop messages + require re-sending
  34. 34. Back-pressure? Push + NACK model (b) Increase buffer size… Well, while you have memory available!
  35. 35. Back-pressure? Push + NACK model (b)
  36. 36. Back-pressure? NACKing is NOT enough.
  37. 37. Negative ACKnowledgement
  38. 38. Back-pressure? Example NACKing Buffer overflow is imminent!
  39. 39. Back-pressure? Example NACKing Telling the Publisher to slow down / stop sending…
  40. 40. Back-pressure? Example NACKing NACK did not make it in time, because M was in-flight!
  41. 41. Back-pressure? speed(publisher) < speed(subscriber)
  42. 42. Back-pressure? Fast Subscriber, No Problem No problem!
  43. 43. Back-pressure? Reactive-Streams = “Dynamic Push/Pull”
  44. 44. Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber Back-pressure? RS: Dynamic Push/Pull
  45. 45. Solution: Dynamic adjustment Back-pressure? RS: Dynamic Push/Pull Just push – not safe when Slow Subscriber Just pull – too slow when Fast Subscriber
  46. 46. Back-pressure? RS: Dynamic Push/Pull Slow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.
  47. 47. Back-pressure? RS: Dynamic Push/Pull Fast Publisher will send at-most 3 elements. This is pull-based-backpressure.
  48. 48. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives!
  49. 49. Back-pressure? RS: Dynamic Push/Pull Fast Subscriber can issue more Request(n), before more data arrives. Publisher can accumulate demand.
  50. 50. Back-pressure? RS: Accumulate demand Publisher accumulates total demand per subscriber.
  51. 51. Back-pressure? RS: Accumulate demand Total demand of elements is safe to publish. Subscriber’s buffer will not overflow.
  52. 52. Back-pressure? RS: Requesting “a lot” Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)
  53. 53. Back-pressure? RS: Dynamic Push/Pull MAX speed
  54. 54. Back-pressure? RS: Dynamic Push/Pull Easy MAX speed
  55. 55. Back-pressure? RS: Dynamic Push/Pull Easy MAX speed
  56. 56. Pipelining in Reactive Streams
  57. 57. Pipelining in Reactive Streams A Publisher may be able to pre-generate some data…
  58. 58. Pipelining in Reactive Streams
  59. 59. Pipelining in Reactive Streams Request(5)
  60. 60. Pipelining in Reactive Streams
  61. 61. Pipelining in Reactive Streams
  62. 62. Pipelining in Reactive Streams
  63. 63. Pipelining in Reactive Streams
  64. 64. Pipelining in Reactive Streams
  65. 65. Pipelining in Reactive Streams …and since signalling happens asynchronously…
  66. 66. Pipelining in Reactive Streams pending demand pending demand reserve buffer space
  67. 67. Pipelining in Reactive Streams signal demand buffer space for incoming elements
  68. 68. Pipelining in Reactive Streams The goal here is to never wait, unless back-pressured.
  69. 69. Reactive Streams SPI
  70. 70. Reactive Streams SPI public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); }
  71. 71. Reactive Streams SPI public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); } gives a public interface Subscription { public void request(long n); public void cancel(); } A
  72. 72. Reactive Streams SPI public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete(); } public interface Publisher<T> { public void subscribe(Subscriber<? super T> s); } gives a to a public interface Subscription { public void request(long n); public void cancel(); }
  73. 73. How does fit in here?
  74. 74. Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-camel: integration akka-streams: stream processing …
  75. 75. Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
  76. 76. An Actor can only: • Send and receive messages • Create Actors • Change it’s behaviour Akka Akka is a high-performance concurrency library for Scala and Java. At it’s core it focuses on the Actor Model:
  77. 77. class Player extends Actor { def receive = { case NextTurn => sender() ! decideOnMove() } def decideOnMove(): Move = ??? } Akka
  78. 78. Akka Akka has multiple modules: akka-actor: actors (concurrency abstraction) akka-camel: integration akka-remote: remote actors akka-cluster: clustering akka-persistence: CQRS / Event Sourcing akka-streams: stream processing …
  79. 79. streams
  80. 80. Akka Streams – design decisions Superior: • reusability • expansibility • performance • bound buffer space • Java and Scala APIs
  81. 81. Akka Streams – bounded buffer space Why reasoning about buffer space is a big deal:
  82. 82. Akka Streams – Linear Flow
  83. 83. Akka Streams – Linear Flow
  84. 84. Akka Streams – Linear Flow
  85. 85. Akka Streams – Linear Flow
  86. 86. Akka Streams – Linear Flow Flow[Double].map(_.toInt). [...] No Source attached yet. “Pipe ready to work with Doubles”.
  87. 87. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") An ActorSystem is the world in which Actors live in. AkkaStreams uses Actors, so it needs ActorSystem.
  88. 88. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() Contains logic on HOW to materialise the stream.
  89. 89. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() A materialiser chooses HOW to materialise a Stream. The Flow’s AST is fully “lifted”. The Materialiser can choose to materialise the Flow in any way it sees fit. Our implementation uses Actors. But you could easily plug in an SparkMaterializer!
  90. 90. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() You can configure it’s buffer sizes etc.
  91. 91. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = Source(1 to 3).runWith(foreachSink)
  92. 92. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() val foreachSink = Sink.foreach[Int](println) val mf = Source(1 to 3).runWith(foreachSink)(mat) Uses the implicit FlowMaterializer
  93. 93. Akka Streams – Linear Flow implicit val sys = ActorSystem("tokyo-sys") implicit val mat = FlowMaterializer() // sugar for runWith Source(1 to 3).foreach(println)
  94. 94. Akka Streams – Linear Flow val mf = Flow[Int]. map(_ * 2). runWith(Sink.foreach(println)) // is missing a Source, // can NOT run == won’t compile!
  95. 95. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  96. 96. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  97. 97. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run!
  98. 98. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run()
  99. 99. Akka Streams – Linear Flow val f = Flow[Int]. map(_ * 2). runWith(Sink.foreach(i => println(s"i = $i”))). // needs Source to run! f.connect(Source(1 to 10)).run() With a Source attached… it can run()
  100. 100. Akka Streams – Linear Flow Flow[Int]. map(_.toString). runWith(Source(1 to 10), Sink.ignore) Connects Source and Sink, then runs
  101. 101. Akka Streams – Flows are reusable f.withSource(IterableSource(1 to 10)).run() f.withSource(IterableSource(1 to 100)).run() f.withSource(IterableSource(1 to 1000)).run()
  102. 102. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
  103. 103. Akka Streams <-> Actors – Advanced Each “group” is a stream too! It’s a “Stream of Streams”. val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber)
  104. 104. Akka Streams <-> Actors – Advanced groupBy(_.last). GroupBy groups “11” to group “1”, “12” to group “2” etc.
  105. 105. Akka Streams <-> Actors – Advanced groupBy(_.last). It offers (groupKey, subStreamSource) to Subscriber Source
  106. 106. Akka Streams <-> Actors – Advanced groupBy(_.last). It can then start children, to handle the sub-flows! Source
  107. 107. Akka Streams <-> Actors – Advanced groupBy(_.last). For example, one child for each group. Source
  108. 108. Akka Streams <-> Actors – Advanced val subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”)) Source(1 to 100). map(_.toString). filter(_.length == 2). drop(2). groupBy(_.last). runWith(subscriber) The Actor, will consume SubStream offers.
  109. 109. Akka Streams – FlowGraph FlowGraph
  110. 110. Akka Streams – FlowGraph Linear Flows or non-akka pipelines Could be another RS implementation!
  111. 111. Akka Streams – GraphFlow Fan-out elements and Fan-in elements
  112. 112. Akka Streams – GraphFlow // first define some pipeline pieces val f1 = Flow[Input].map(_.toIntermediate) val f2 = Flow[Intermediate].map(_.enrich) val f3 = Flow[Enriched].filter(_.isImportant) val f4 = Flow[Intermediate].mapFuture(_.enrichAsync) // then add input and output placeholders val in = SubscriberSource[Input] val out = PublisherSink[Enriched]
  113. 113. Akka Streams – GraphFlow
  114. 114. Akka Streams – GraphFlow FlowGraph { implicit b => import FlowGraphImplicits._ val bcast = Broadcast[Intermediate] val merge = Merge[Enriched] in ~> f1 ~> bcast ~> f2 ~> merge bcast ~> f4 ~> merge ~> f3 ~> out }
  115. 115. Akka Streams – GraphFlow FlowGraph { implicit b => import FlowGraphImplicits._ val bcast = Broadcast[Intermediate] val merge = Merge[Enriched] in ~> f1 ~> bcast ~> f2 ~> merge bcast ~> f4 ~> merge ~> f3 ~> out }
  116. 116. Akka Streams – GraphFlow val g = FlowGraph {} FlowGraph is immutable and safe to share and re-use!
  117. 117. streams in action
  118. 118. There’s more to explore! Topics we did explore today: • asynchronous non-blocking back-pressure • complex graph processing pipelines • streams powered TCP server / client • a sneak peek into custom elements
  119. 119. There’s more to explore! Topics we didn’t explore today: • explicit buffering, and overflow strategies • integrating with Akka Actors • time-based operators (takeWhile, dropWhile, timer transforms) • plenty additional combinators and junctions • implementing custom processing stages and junctions
  120. 120. There’s more to explore! Future plans: • API stabilisation and documentation (1.0 soon) • Improve testability & TestKit • Performance tuning of Streams & HTTP • Provide more Sinks / Sources and operations • Visualising flow graphs • great experiment by Tim Harper
 https://github.com/timcharper/reactive-viz • Distributing computation graphs (?)
  121. 121. Links • http://akka.io • http://reactive-streams.org
 • https://groups.google.com/group/akka-user
 • Tim Harper’s awesome complex pipeline example + visualisation
 https://github.com/timcharper/reactive-viz
 • 1.0-M2 Documentation (not complete)
 http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html • Complete JavaDSL for all operations
 https://github.com/akka/akka/pulls?q=is%3Apr+javadsl
  122. 122. ©Typesafe 2015 – All Rights Reserved Reactive Application Development! SBT in Action! Akka in Action! Play for Scala! Play for Java! Atomic Scala! Check out eBooks at Typesafe!! Visit http://typesafe.com/resources/e-books to view all!! Check out eBooks at Typesafe Visit http://typesafe.com/resources/e-books to view all  
  123. 123. ©Typesafe 2015 – All Rights Reserved

×