This document provides an overview of Lamina, a Clojure library for working with streams of event data: - Lamina allows transforming, aggregating, analyzing, and reacting to streams of event data. It is useful for creating narratives from sequences of events over time. - Key concepts include channels for streaming data, functions for processing streams like map and reduce, and probe channels for instrumenting code and generating event streams. - Lamina includes tools for visualizing event streams, calculating statistics like rates and quantiles, and distributing streams by properties before aggregation. This allows flexible analysis of distributed event-based applications.

1. Generating and Analyzing Events
Zach Tellman
Runa
@ztellman

2. linear threads of
execution

3. linear threads of
execution

4. linear threads of
execution

6. logging

7. logging
“but we have only the silent evidence of
scattered cups and dice, ... brooches and sandals.
How can we make these objects live?”
- Peter Ackroyd, London: A Biography

8. logging

9. logging
• is a lossy stream of data
• is a verbose stream of data
• is a necessary stream of data

10. lamina
• is for transforming streams of data
• is for aggregating streams of data
• is for analyzing streams of data
• is for reacting to streams of data

11. lamina
• is for creating narratives from events

12. what we talk about
when we talk about
events

13. a single event
• is called a future
• is called a promise
• is called an async-result

14. a stream of events
• is called a channel

15. channel
node queue

16. (enqueue ch 1)

17. (map* inc ch)

18. (enqueue ch 2)

19. (map* dec ch)

20. (enqueue ch 3)

21. lamina.viz
(view-graph ch)
(view-propagation ch 1)

22. (join a c)
(join b c)

23. (receive-all c prn)

24. (close a)

25. (close a)

26. (close a)

27. (close a)

28. (close a)

29. other familiar functions
• mapcat*
• reduce*
• reductions*
• take*
• take-while*
• partition*
• partition-all*

30. so why not just use
seqs?

31. data
+ time
event

32. less familiar functions
• sample-every
• partition-every
• rate
• mean
• periodically
• combine-latest

33. 1 lazy-seq
+ 1 thread
1 eager-seq

34. one thread per edge

35. back-propagation

36. back-propagation

37. lamina
• is for temporal streams of data
• is for granular streams of data
• is for reasoning about streams of data

38. probe-channels
• are grounded
• are permanent
• are named

39. generating probe data
(trace :foo:bar
{:value 123
:description "a number!"})

40. consuming probe data
(probe-channel :foo:bar)
(probe-channel ["foo" :bar])

41. consuming probe data
(select-probes "fo*" #"ba[r|z]")

42. consuming probe data

43. instrumenting functions
(instrument + {:name "plus"})

44. (probe-channel :plus:enter)
{:name "plus"
:timestamp 1234567890
:args [1 2 3]}

45. (probe-channel :plus:return)
{:name "plus"
:timestamp 1234567890
:offset 0
:duration 10000
:sub-tasks nil
:args [1 2 3]
:result 6}

46. (probe-channel :plus:error)
{:name "plus"
:timestamp 1234567890
:offset 0
:duration 10000
:sub-tasks nil
:args [nil 1]
:error ...}

47. (defn-instrumented bar
[x y]
(+ x y))
(defn-instrumented foo
{:name "foo"}
[x y]
(bar x y))

48. {:name "foo"
:timestamp 1234567890
:offset 0
:duration 10000
:sub-tasks ({:name "user:bar"
:timestamp 1234567890
:offset 1000
:duration 8000
:sub-tasks nil
:args [1 2]
:result 3})
:args [1 2]
:result 3}

49. (time* (foo 1 2))
time - 13.0us
foo - 10.0us
user:bar - 8.0us

50. the cost of instrumentation
(+ 1 1)
vs
(apply + [1 1])

51. executors
• are instrumented thread-pools

52. executors
(executor
{:name :some-threads
:min-thread-count 2
:max-thread-count 8})

53. (defexecutor ex
{:name "ex"
:max-thread-count 4})
(defn-instrumented foo
{:name "foo"
:executor ex}
[x y]
(+ x y))

54. (probe-channel :foo:return)
(probe-channel :ex)
{:name "foo"
:offset 0
:timestamp 1234567890
:duration 10000
:enqueued-duration 500000
:sub-tasks nil
:args [1 2]
:result 3}

55. lamina.stats
• rate
• sum
• mean
• quantiles
• variance
• outliers

56. mean
an exponentially weighted moving average
with a configurable window

57. mean
(->> (probe-channel :ex)
(map* :enqueued-duration)
mean)

58. quantiles
a statistical distribution over the
last five minutes using reservoir sampling

59. quantiles
(->> (probe-channel :foo:enter)
rate
quantiles)

60. quantiles
{50 5
75 7.5
95 9.5
99 9.9
99.9 9.99}

61. outliers
a filtered view of the data stream that
only emits unusual events

62. outliers
(->> (probe-channel :foo:return)
(outliers :duration))

63. it’s easy to merge streams of data,
but hard to split them apart

64. distributor
creates sub-streams grouped by facet,
and applies identical operations to each stream

65. distributor
(distributor :uri
(fn [facet ch]
(->> ch
(close-on-idle 10000)
rate
(map* #(vector facet %)))))

66. distributor

67. aggregate
merges multiple periodic streams

68. aggregate
(aggregate first ch)

69. aggregate
["/" 1]
["/abc" 1]
["/def" 1]

70. aggregate
["/" 1]
["/abc" 1]
["/def" 1]
["/" 1]

71. aggregate
{"/" ["/" 1]
"/abc" ["/abc" 1]
"/def" ["/def" 1]}
["/" 1]

72. distribute-aggregate
(distribute-aggregate
:uri
(fn [facet ch] (rate ch))
ch)

73. distribute-aggregate
{"/" 1
"/abc" 1
"/def" 1}

74. (distribute-aggregate
:name
(fn [_ ch]
(->> ch
(map* :duration)
sum))
(probe-channel :ex))

75. aleph
“the only place on earth where all places are --
seen from every angle, each standing clear,
without any confusion or blending.”
- Jorge Luis Borges, The Aleph

76. traffic
{:name "http-server"
:remote-address "127.0.0.1"
:bytes 42}

77. (distribute-aggregate
:name
(fn [_ ch]
(->> ch
(map* :bytes)
sum))
(select-probes "*traffic:out"))

78. (distribute-aggregate
:remote-address
(fn [_ ch]
(->> ch
(close-on-idle 10000)
(map* :bytes)
sum))
(select-probes "*traffic:out"))

79. what’s missing
• further examples
• inter-process aggregation
• data endpoints/tooling

80. questions?