The good, the bad, and the ugly technical details of running Node.js services for the world’s largest Spanish learning website, SpanishDict.com. Presented at the 5/8/2013 Seattle Node.js Meetup - http://www.meetup.com/Seattle-Node-js/events/115959992/

1. Node.js in
Production
-
|
Seattle Node.js
May 8th, 2013
Ryan Roemer @ryan_roemer

2. Overview
Production!
Well, what our production looks like.
Five Node.js-related things we've
learned.
Some additional resources.

3. What is
"Production"?

4. Curiosity Media

5. Brought to you
by...
A team of three engineers
Who are full-time developers
Running everything in the cloud
With minimal time available for ops

6. SpanishDict.com

7. Demo

8. A Spanish-English
Dictionary
is the world’s
largest Spanish-English dictionary,
translation, and language learning
website. We develop and provide
reliable, accurate, easy-to-use
resources for learning Spanish.
SpanishDict.com

9. Our visitors
A into our data and usage:
6,000,000 Unique visitors every month
1,000,000 Translations
100,000 Questions and answers
25,000 Flashcards
5,000 Video pronunciations
90 Lessons
quick glance

10. Our Services
Node Other
API server Web site
Auto-suggest server Data mining
Translation server Operations
Text-to-speech server

11. Our Traffic
Very low latency for our db-backed
services.
Service Reqs/min
API server 35K / min
Auto-suggest server 15K / min
Translation server 2.5K / min
Text-to-speech server 400 / min

12. Five Node.js
production tips
1. Know when to Node
2. Keep up with Node
3. Design for failure
4. Isolate services
5. Analyze everything

13. 1. Know when to
Node
Should you use Node.js?

14. Yes
Small apps (think JSON APIs)
"Glue" for services or data
Proxies
Concurrent data
Use the module
Lots of connections
stream

15. Maybe not
Computation
Legacy applications
"Solved" problems (fuzzy search, NLP,
etc.)

16. 2. Keep up with
Node
Bleeding edge, lots of breakage.
Stay up to date with Node.js and libraries.

17. Infrastructure

18. Node.js
deployments
PAAS: Often, the easier way.
, , , etc.
IAAS: Expect some DIY
Build custom Node.js versions
Install modules from scratch
Get ready to roll back
Joyent Heroku Nodejitsu

19. 3. Design for failure
Fail and recover at multiple levels.

20. App-level
Errors
Handle: uncaughtException
Listen: foo.on("error")
Use the module
Workers: die early on errors
Master: monitor and kill workers
cluster

21. Server-level
Use or alternatives
Restart the Node.js master
monit

22. Service-level
Have lots of small apps
Stateless, fungible servers
Hot failover wherever possible

23. 4. Isolate services
Separate resource and failure classes.

24. Resources
CPU/Load: Run out of this and it's over.
Also, memory, I/O, etc.
... and combinations thereof

25. Our pains
Node.js apps aren't necessarily good
neighbors.
Suggest (DB) and translate (http)
Backend (DB) and web site (CPU/load,
memory)
Read and write servers

26. Takeaways
Always preserve CPU
Monitor system stats for cross-pressure

27. 5. Analyze
everything
How well are we addressing lessons 1-4?
Data drives problem discovery and action.

28. Log, Monitor, Mine

29. Things to look for
Some metrics that affect Node.js apps
Type Metrics Uses
System CPU, I/O, memory,
network
Alert
Server Throughput, latency Alert,
Report
Traffic Peaks (weeks,
months)
Report
Errors Quantitative,
qualitative
Alert,
Report

30. Decisions, Goals
Identify
Resource pressure
Bugs
Decide
Scale up, scale down?
Separate?

31. Demo

32. Recap
1. Know when to Node
2. Keep up with Node
3. Design for failure
4. Isolate services
5. Analyze everything

33. Further Reading
by
by
by
Production Node.js Secrets
@dshaw
Node.js in Production @felixge
Zero to Node @williamjohnbert

34. Thanks!
|Ryan Roemer @ryan_roemer
@SeattleNode