Observable Node.js Applications - EnterpriseJS
•
3 gostaram•1,805 visualizações
This document discusses making Node.js applications more observable at scale. It describes using tools like Bunyan for structured logging, Restify for building observable REST applications, and Unix tools for querying logs. Observability is important for production systems to detect errors, latency issues, and memory leaks. The presenter advocates building observability into applications from the start through techniques like scoped child loggers and request capture streams.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Observable Node.js Applications - EnterpriseJS
Semelhante a Observable Node.js Applications - EnterpriseJS (20)
Último
Último (20)
Observable Node.js Applications - EnterpriseJS
- 1. Observable Node.js Applications at Scale Yunong Xiao, Senior Node.js Engineer, UI Platform, Netflix @yunongx, yunong@netflix.com July 2015EnterpriseJS
- 10. What’s Wrong? Increased Errors Increased Latency Memory Leak
- 14. DATA “It is a capital mistake to theorize before one has DATA. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” Sherlock Holmes -A Scandal in Bohemia
- 17. Hard Harder Insane Development Production Production w/ Direct Customer Impact
- 18. Dev === Prod
- 21. node-bunyan Streaming JSON logging format
- 22. Streaming JSON
- 23. Unix Philosophy grep(1), cut(1), wc(1), awk(1), perl(1), jq(1), json(1), daggr(1)
- 24. Query Logs in Real Time $ cat http.log | grep audit| bunyan -0 --strict - c 'this.res.statusCode === 200' | json -ga ‘req.timers["fetch-api"]' req.url | awk '{print $1 " " substr($2,0,7)}' | daggr -k 2 -v 1 quantize
- 25. Illegible!
- 26. bunyan(1) CLI
- 28. { "name": "demo", "hostname": "Yunongs-MacBook-Pro.local", "pid": 35919, "level": 30, "authInfo": { "username": "yunong", "password": "xxxxxx" }, "msg": "got request", "time": "2015-07-07T21:46:01.317Z", "v": 0 }
- 31. Features • Lightweight API • Log levels: trace, debug, info, warn, error, fatal • Extensible Streams interface. • Custom object rendering with serializers.
- 32. HTTP Request • Make API requests • Persist state to file system • Query database • Update caches
- 33. Async HTTP Request • Make API requests • Persist state to file system • Query database • Update caches
- 37. node-vasync • Observable Async workflow • https://github.com/davepacheco/node-vasync
- 38. async f() state failed f() successful f() pending f() finished f() # of errors
- 42. How Do I See this in Prod? • Logs • Core Dumps • REPL • HTTP API
- 44. • Observability • Metrics • Bunyan integration
- 46. Audit Logs
- 47. Audit Logsstatus code request ID (UUID) URL request headers response headers individual handler timers req latency
- 49. Too many logs!
- 50. Request Capture Stream • Captures all log statements at trace level in memory. • Dump all logs for a particular request only on error.
- 58. Logging • Native Bunyan integration • Request capture stream • Audit logs • Scoped child loggers
- 59. What about Data Processing?
- 60. Examples Find all logs for a specific request. $ grep $uuid restify.log
- 61. Examples Count the # of non-200 responses. $ grep restify-audit restify.log | bunyan -c ‘this.res.statusCode !== 200` -0 | wc -l
- 62. Examples Show all requests that took longer than 200ms. $ grep restify-audit restify.log | bunyan -c ‘this.latency > 200` -0 | wc -l
- 63. Advanced Example Request latency distribution by URL
- 64. restify + Bunyan • Streaming JSON. • Processing using Unix tools is easy. • Helpful tools: • Unix: cut(1), wc(1), grep(1), awk(1), perl(1), json(1), jq(1) … • JSON: https://github.com/trentm/json (npm install -g json) • daggr: https://github.com/joyent/daggr (npm install -g daggr)
- 65. Increased Errors Increased Latency
- 66. restify Core Contributors at Netflix @stinkydofu, @mjr578, @yunongx
- 67. restify Interested? Contribute to restify! restify.com github.com/restify/node-restify
- 68. distributed processing • elasticsearch • spark • hive bunyanrestify audit logs req id req headers req latency handler latencies URL res codeErrors res headers Processing w/ Unix tools • grep(1) • awk(1) • cut(1) • wc(1) • sort(1) • json(1) • bunyan(1) • … req scoped logs application specific context req capture stream dumps all logs on error DTrace* req latency handler latencies * Where available Production Observability
- 69. Observable Toolkit • vasync: https://github.com/davepacheco/node-vasync • Observable async operations • bunyan: https://github.com/trentm/node-bunyan • Streaming JSON logs • restify: restify.com • Observable REST applications • Unix Tools • Easily process JSON logs
- 70. One more thing…
- 71. DTrace
- 72. Thanks • Questions? We’re hiring! • yunong@netflix.com • @yunongx • http://restify.com July 2015EnterprisJS