Streaming libraries are kind of complex. Why is that so? They have to handle many different concerns: streaming, composition, concurrency, resources management,... Is it possible to isolate those concerns into smaller, composable libraries?

1. Streams, effects
and
Beautiful folds
Eric Torreborre

3. About Zalando

9. Every application ever
written can be viewed
as some sort of
transformation on data.
Daniel Spiewak

14. ???

15. Streams!
or iteratees...
or observables...

17. data ConduitM i o m r
This type represents a general component which can
• consume a stream of input values i
• produce a stream of output values o
• perform actions in the m monad
• and produce a final result r

18. type Pipe a b = Proxy () a () b
A Proxy is a monad transformer that receives and
sends information on both an upstream and
downstream interface
data Proxy a' a b' b m r

19. Machines are demand-driven coroutines ... they are entirely pure.
A Machine can be driven by a Driver which can have effects when
feeding the Machine input or when reading its output
• you can construct a `Plan` and then `compile` it to a `Machine`
• a `Plan[K, O, A]` is a specification for a pure `Machine` that reads inputs
selected by `K` and writes values of type `O`
• the `Plan` has intermediate results of type `A` which are placeholders for further
plans
trait Plan[+K, +O, +A]
type Machine[+K, +O] =
Plan[K, O, Nothing]

22. Motivation
• data too large
• actors offer no static guarantee
• back-pressure

23. class Flow[-In, +Out, +Mat](m: Module)
extends Graph[FlowShape[In, Out], Mat]
A Flow is a set of stream processing steps that has one
open input and one open output.
Materialized: 'a value that can be obtained from the
materialized processing pipeline'

25. The design goals are
• compositionality
• expressiveness
• resource safety
• speed

26. StreamCore / Stack
Segment (Fail, Emit, Handler, Append)
Scope
Chunk / Pull
Stream[+F[_],+O](core: CoreRef[F,O])
A stream producing output of type `O` and which may
evaluate `F` effects

28. Motivation
• high-performance
• lightweight
• conceptually simple
• debuggable
• type-safe and concise

29. class Pipe[-A, +B] extends StreamOps[B]
Simple transformations are stream graph components with
one input port and one output port
trait StreamOps[A] {
type Repr[T] <:
StreamOps[T] {
type Repr[X] <: self.Repr[X]
}
}
}
FanIn, Spout, shapeless

31. Motivation
• resource management
• streaming
• fusion
• error recovery
• not for concurrency

34. Motivation
• asynchronous
• reactive
• streaming
• back-pressure

35. class Observable[A]
class Consumer[A, B] extends
(Observable[A] => Task[B])
• Observable implements the "Reactive Pattern" = subscribe
Subscribers
• Consumer is a specification of how to consume an
Observable to return a result

36. Differences?

37. Streaming
Composition
Safety
Concurrency
Backpressure
Topology

39. Producer
EffectsFolds

40. Streaming
Composition
Backpressure

41. ...

44. Create

46. Transform

49. Monad

51. Transform

52. Nice way to
the room?

53. Run

55. Streaming
Composition
Backpressure

56. Streaming
Composition
Topology

60. def count():
current = 0
while True:
(yield current)
current = current + 1

61. c = count()
next(c)
print(c.send(1)) // 1
print(c.send(1)) // 2
print(c.send(1)) // 3
c.close()

62. Run

63. Folds
count
minimum
maximum
all any
headlast
firstN
lastN plus
times
varianceaverage
quantiles countMinSketch

65. Applicative

68. Topology

69. Sinks

72. Streaming
Composition
Topology

74. Safety
Concurrency
Backpressure

79. TimedFutureEffect
TaskEffect
TaskEffect
TwitterFutureEffect
TaskEffect

80. Short-circuit

86. Safety

94. Safety
Concurrency
Backpressure

95. Backpressure

96. Merging queues

97. Producer
EffectsFolds

105. tricky

106. Producer
Fold
Safe?
Safe Producer

108. tricky II