Video and slides synchronized, mp3 and slide download available at http://bit.ly/145Y513. Jay Kreps discusses the evolution of LinkedIn's architecture and lessons learned scaling from a monolithic application to a distributed set of services, from one database to distributed data stores.Filmed at qconnewyork.com. Jay Kreps is a Principle Staff Engineer at LinkedIn where he acts as the technical lead for data infrastructure and relevance areas. He is the original engineer for several open source projects developed there including Voldemort (a key-value store), Azkaban (a Hadoop workflow scheduler), and Apache Kafka (a distributed messaging system).

1. Lessons From Building and
Scaling LinkedIn
Jay Kreps

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/linkedin-architecture-stack

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

5. what’s changed?
• data size
• traffic
• # engineers
• code size

6. what did I learn?

7. scaling
the site

8. #1: scalability is
about managing
state

9. and state belongs in
data systems

10. in the beginning…

11. …going distributed…

12. …and now

13. #2:
make
simple
cheap
scalable
primitives

14. #3: ops first

15. #4: do hard
things later

16. LinkedIn data systems
• Voldemort—key/value storage
– 400k peak queries per second
– 400 tables
• Kafka—messaging
– 460k peak unique message writes per second
– 2.3m peak message reads
• GraphDB—graph operations
– 80k peak queries per second
• Hadoop—batch processing and analytics
– 5000 nodes
– 17 PB
– 30,000 jobs per day
– 700 users

17. services

18. cutting up the application

19. in the beginning…

20. …going distributed…

21. ..and now

22. #5:
services
(may)
scale
development

23. #6: Bad Services << No Services << Good
Services

24. Service layer evolution
• HTTP + serialized java
• Protocol Buffers + TCP
• REST + JSON

25. #7:
the
service
contract
is
binary

26. #8:
isolation
vs
utilization

27. scaling
software
engineering

28. cutting up the code base

29. #9:
build
your
process

30. we model
the complete lifecycle:
review
checkin
build
rollout

31. governance

32. communism vs capitalism

33. #10: treat code as property

34. #11: but you need effective government

35. #12: don’t
duplicate
integration layers

36. agility at scale

37. #14:
agility
requires
safety

39. two cheers
for testing

40. tests
are focused
on predicting things
that will go wrong

41. #15: measurement > prediction

42. how to measure cost?

43. #16:
reduce time in error
or
exposure to error

44. jay.kreps@linkedin.com
http://www.linkedin.com/in/jaykreps
@jaykreps