Npm has modules for devops, like logging, metrics, service discovery. But when you arrive to production, you may find that these are already handled by old players. Avoid the same mistakes I did, when my first node app was on its way to the world.

1. Redundant Devops
about reinventing the wheel

2. Szabolcs Szabolcsi-Toth
@_Nec

3. Senior Engineer

4. JSConf
Budapest
2017
Curator, Organizer

5. What’s this about?

6. Metrics
Error Logs
Logging
Secret Store
Service Discovery
Process Supervision
Running Programs
Connecting Services

7. Metrics

8. Metrics are
• time bound, historical
• numeric data
• software, network or hardware property

9. Metrics are great!
• see trends
• mark releases
• notice anomalies:
spikes & gaps
• create alerts
!
?

10. Metric delivery
• collect (scrape) or push data?
• collect periodically
• put metric data where it can
be collected

11. Tools for metrics
• prometheus
• graphite

12. Node best practices
• put your metrics on an accessible endpoint
/metrics
/status
• there are node libs to automate this
instrument http
• let the metrics tool do scraping, delivery
• watch those nice graphs ☺
check out grafana

13. Key metrics
• latency
check for slow queries,
create performance tests on them
iterate code, re-test again
do not average, use a histogram
• resource usage
slow memory leaks
disk is getting full
predict resource shortage via trends
latency

14. Sending metrics
is not the job of your app

15. Error logs

16. Catch errors as fast as possible!
• instant alert of production errors
• use while feature testing
• keep an eye on it during releases
• aggregate errors in a single service, see all
• catch before the user

17. Ideal error reports have
• environment of error
build / release / branch / server
• stack trace
exact code location
• custom data
anything that helps identifying the problem

18. Error log delivery
• can happen any time,
hopefully rare
• push data
• expect the unexpected,
handle the unhandled
• never log secrets
• sampling, throttling, timeout
do not let error logging itself
kill your app

19. Tools & services for error reporting
• airbrake
• errbit (airbrake api, open source)
• sentry
• raygun
• rollbar
• …

20. Integrate, get notified!
• pagerduty
• slack / hipchat
chatops - resolve, react within your chat

21. Logging

22. Logging vs Error logging
• logging is anticipated
• error logs are occasional

23. Log levels

25. Log levels, recap
• fatal - needs instant intervention, see error logs
• error - inform the user, see error logs
• warn - escalate if happens again
• info - just a step in a regular flow
• debug - full of lines, and traces

26. Benefits of logging, custom logs
• debug
• custom events
• tracking the usage and behaviour of app
• profile, AB test, product development

27. Logging in node
• console.log
• bragi
• debug
• npmlog
• winston

28. Logging in node - general
• has timestamps
• has loglevels
• can be routed to stdout/stderr
• can be formatted
• create or use Correlation ID

29. Correlation ID quick quide
cID
cID
cID
cID
cID
cID
cID
cID
cID
services
logs

30. Best practices
• just put it to stdout
(docker & kubernetes clearly ecourages this)
• let the log collector handle it
• pipe stdout to a file, or whatever you like
• able to set to debug mode runtime
use signals
• never log secrets

31. Log collectors
• fluentd
• logstash
• syslog-ng
• rsyslog

32. A good log collector should
• read from stdout / file tail
• use your correlation ID
• remove the burden of transferring your logs

33. Remote logging
• Stackdriver (fluentd based)
• Elasticsearch (fluentd based)

34. Sending logs
is not the job of your app

35. Secret Store

36. Secrets
• passwords / usernames
• db names
• API keys
• private keys

37. NOT Secret Storage
× source code
× private VCS repositories
× config files
× simple database fields
× ENV variables

38. Benefits
• ACL, policies
access set of secrets by
revokeable tokens
• centralized key rotation
edit, update all secrets
at one place
• single use access,
n-use access
• time bound keys
• audit log
• runtime access
no secrets stored on
disk
• build-time access

39. How it works

40. build
server
app
server
Secret Store
Build time Run time
Version Control
secret/name secret/name
secrets built in the
deployed code
secrets were requested
on app startup, stored only
in memory
- token
- secret/name
- actual secrets
- token
- secret/name
- actual secrets

41. Secret store server
• powerful encryption
• has to be unlocked on start
• secrets are totally inaccessible without
unlocking

42. Secret store services
• HashiCorp Vault
• Amazon KMS
• Docker Swarm
• Keywhiz

43. Never store your secrets in your
source code

44. Service Discovery

45. Service discovery can help
• Service Registration
and notify other services of the registered one
• Service Discovery
searching for services?
• Monitoring
is a service active and responding?
• Load Balancing
direct traffic to the new service

46. How it works
• can act like a DNS
simple usecase
internal network
• can write / create configs
more complex
more control

47. How it works
APP
SD AGENT
check PORT
check PID
LBStart
scraping
metrics
Loadbalancer
directs
traffic
Service registry

48. Service discovery agent
• separate task, job, process
• can be configured what to check
• independent of your app

49. Service discovery services
• Apache Zookeeper
• Netflix Eureka
• HashiCorp Consul
• Doozer
• Etcd (can be used to build service discovery)

50. Registering services
is not the job of your app

51. Process
Supervision

52. Process supervision
• keeping your app working
• based on some property you define
not just process id, but
port
ping
http response
• can fail after trying

53. Process supervision
in Node-land
• PM2
• forever

54. Process supervision in general
• monit
manage any process
small footprint
simple

55. Pro Con
Using
Monit
Not
using
Monit
monit can instantly
restart your failing
service
you might not know
why it was failing
MTTR* can be
relatively high
you can debug
what actually
happened
*Mean Time To Repair

56. Running Programs

57. Simple role
• start & stop your app
watch the process itself
handle process state
• send signals to the app
signals can be interpreted as tasks

58. Running Programs in general
• runit
• upstart
• systemd
• Supervisord
• God
• Circus

59. A good program runner
• distribution independent
you can migrate your scripts any time
• easy to config

60. monit + runit (or similar)
• avoid using auto restart in both
can create weird race conditions,
they do not know about each other
• use runit to configure app start/stop
• let monit decide when to restart & use runit

61. Connecting
Services

62. Goals & benefits
• decoupling
separate services
loosen up the connection between them
• scaling
scale up easily when needed
scale down after

63. HTTP based APIs
vs
Message Queues

64. HTTP based APIs
LOADBALANCER
Service “1” Service “2”

65. Message Queues
Service “1” Service “2”
MESSAGE QUEUE

66. HTTP based APIs
or
Message Queues?
It depends

67. HTTP APIs
• async / sync
• remote
• open API
Msg Queues
• async (usually)
• grouped, close
• low latency

68. Lessons learned

69. Prototype & learn
• use whatever modules and services
you like
• get ready to go to live & production
environments
• get ready to scale easily

70. Focus your app
• your app should do it’s job!
• not sending logs, metrics, notifying
service registries or keeping itself
running
• keep it simple

71. Talk to your ops
• they are here to run your app
• can help you a lot
• get on a common ground
• ask the right questions

72. With many thanks to
Peter Wilcsinszky / @pepov
Ferenc Kovacs / @Tyr43l

73. Let’s talk! :)
Find me around here,
or come visit us in 2 weeks!
JSConf
Budapest
2017

74. Thank you