Ludwik Bukowski from Erlang Solutions presented Elixir with Datadog at EMPEX LA 2019.

1. www.erlang-solutions.com
Elixir with
Datadog
Ludwik Bukowski
ludwik.bukowski@erlang-solutions.com
EMPEX
Los Angeles 2019

2. www.erlang-solutions.com

3. www.erlang-solutions.com

4. www.erlang-solutions.com
graphQL

5. www.erlang-solutions.com
Absinthe

6. www.erlang-solutions.com

7. www.erlang-solutions.com
Need of Observability

8. www.erlang-solutions.com
Need of Observability
Metrics!

9. www.erlang-solutions.com
Need of Observability

10. www.erlang-solutions.com
Datadog
- Metrics
- Tracing
- Logs
- Analysis

11. www.erlang-solutions.com
Metrics - method
Agent

12. www.erlang-solutions.com
Metrics - method
Agent
- Statix library

13. www.erlang-solutions.com
Statix library
Statix.increment("hits", 1, [])
-----------> counter
Statix.gauge("cpu_usage", 0.83, [])
-----------> gauge
Statix.histogram("online_users", 123, [])
-----------> histogram
https://github.com/lexmag/statix

14. www.erlang-solutions.com

15. www.erlang-solutions.com

16. www.erlang-solutions.com

17. www.erlang-solutions.com
Metrics - instrumentation

18. www.erlang-solutions.com
Metrics - instrumentation
Measure two times; in the beginning of
plugs pipeline and just before sending
response back to the user
Use:
Plug.Conn.register_before_send/2
Phoenix

19. www.erlang-solutions.com
Metrics - instrumentation
request
response
Plug A
Plug B
Plug C
Plug D
...
Phoenix

20. www.erlang-solutions.com
Metrics - instrumentation
request
response
Plug A
Plug B
Plug C
Plug D
...
Start Time←
← Calculate diff← End time ←

21. www.erlang-solutions.com
Metrics - instrumentation

22. www.erlang-solutions.com
Metrics - instrumentation

23. www.erlang-solutions.com
Metrics - instrumentation

24. www.erlang-solutions.com
Metrics - instrumentation
For Controllers and Views’
instrumentation use built-in Phoenix
instrumentation
Use:
config :realworld, RealWorld.Endpoint,
instrumenters: [RealWorld.Phoenix.Instrumenter]

25. www.erlang-solutions.com
Metrics - instrumentation
Register callback for LogEntry structure. The
structure has fields: query_time, queue_time,
queue_time
Use:
config :real_world, RealWorld.Repo,
loggers: [
{RealWorld.Metrics, :record_metric,[]} ]
Ecto

26. www.erlang-solutions.com
Metrics - instrumentation
config :real_world, RealWorld.Repo,
loggers: [
{RealWorld.Metrics, :record_metric,[]} ]

27. www.erlang-solutions.com
Need of Observability

28. www.erlang-solutions.com
time

29. www.erlang-solutions.com
Flame
graph!
time

30. www.erlang-solutions.com
Spandex library
- Each query is a new Trace
- Trace have spans. Spans have metadata
- Spans storage in process dictionary or in ETS
- start_span/1, finish_span/1
https://github.com/spandex-project/spandex
Tracing - method

31. www.erlang-solutions.com
Tracing - instrumentation
request
response
Spandex.Plug.StartTrace
Plug B
Plug C
Spandex.Plug.EndTrace
...
Phoenix/Plug

32. www.erlang-solutions.com
Tracing - instrumentation
request
response
Spandex.Plug.StartTrace
Plug B
Plug C
Spandex.Plug.EndTrace
...
Use spandex_phoenix library
Phoenix/Plug

33. www.erlang-solutions.com
Tracing - instrumentation
Absinthe

34. www.erlang-solutions.com
Tracing - instrumentation
start span

35. www.erlang-solutions.com
Tracing - instrumentation
end span

36. www.erlang-solutions.com
Tracing - instrumentation
Create span before field is resolved and
finish it after the resolution.
Use:
Implement Middleware behaviour
Absinthe

37. www.erlang-solutions.com
Tracing - instrumentation

38. www.erlang-solutions.com
Tracing - instrumentation

39. www.erlang-solutions.com
Tracing - instrumentation
Register callback for LogEntry structure. The
structure has fields: query_time, queue_time,
queue_time. Artificially create a span, i.e
update its start end finish times.
Use:
Spandex.update_span/1
Ecto

40. www.erlang-solutions.com
Tracing - instrumentation
Ecto

41. www.erlang-solutions.com
Tracing - instrumentation
Ecto

42. www.erlang-solutions.com
Tracing - instrumentation
Ecto

43. www.erlang-solutions.com
Tracing - instrumentation
Ecto

44. www.erlang-solutions.com
Tracing - instrumentation
Ecto
Use spandex_ecto library

45. www.erlang-solutions.com
Tracing - instrumentation
Simply wrap function calls with
start_span/1 and finish_span/1 using
decoration.
Use:
@decorate annotation
https://github.com/arjan/decorator
Custom code

46. www.erlang-solutions.com
...
@decorate tracer(:cache)
def get_from_cache(key) do
val = Cachex.get(:my_cache, key)
{:ok, val}
end
https://github.com/arjan/decorator
Tracing - instrumentation
Custom code

47. www.erlang-solutions.com
...
@decorate tracer(:cache)
def get_from_cache(key) do
val = Cachex.get(:my_cache, key)
{:ok, val}
end
https://github.com/arjan/decorator
Tracing - instrumentation
Custom code

48. www.erlang-solutions.com
Tracing - instrumentation
https://github.com/arjan/decorator
Custom code

49. www.erlang-solutions.com
Tracing - instrumentation
https://github.com/arjan/decorator
Custom code

50. www.erlang-solutions.com
- Number of traces/number of spans sent should be controlled. Configurable in
spandex (batch_size and sync_threshold options)
- Metadata size consumes network between node and agent and agent and
datadog
- How track asynchronously spawned processes
- For query that involves more than one node distributed tracing in spandex can
be used.
- Tracing overhead. Few hundreds spans per trace still consumes few
milliseconds per query, measured with xprof
https://github.com/Appliscale/xprof
- Telemetry integration ?
Tracing - issues

51. www.erlang-solutions.com
others
- Great Datadog support team
- Events, monitors, logs and analysis

52. www.erlang-solutions.com
Thank you

53. www.erlang-solutions.com
DON’T BE JELLO.
ACCELERATE WITH OUR
ELIXIR ARCHITECTURE
SESSIONS
www.erlang-solutions.com/solutions/
elixir-development
Thank you