4. At a Glance
The Social Graph
120M+ active users
50B+ PVs per month
10B+ Photos
1B+ connections
50K+ Platform Apps
400K+ App Developers
5. General Design Principles
Use open source where possible
âȘ
Explore making optimizations where needed
âȘ
Unix Philosophy
âȘ
Keep individual components simple yet performant
âȘ
Combine as necessary
âȘ
Concentrate on clean interface points
âȘ
Build everything for scale
âȘ
Try to minimize failure points
âȘ
Simplicity, Simplicity, Simplicity!
âȘ
7. PHP
Good web programming language
âȘ
Extensive library support for web development
âȘ
Active developer community
âȘ
Good for rapid iteration
âȘ
Dynamically typed, interpreted scripting language
âȘ
8. PHP: What we Learnt
Tough to scale for large code bases
âȘ
Weak typing
âȘ
Limited opportunities for static analysis, code optimizations
âȘ
Not necessarily optimized for large website use case
âȘ
E.g. No dynamic reloading of files on web server
âȘ
Linearly increasing cost per included file
âȘ
Extension framework is difficult to use
âȘ
10. MySQL
Fast, reliable
âȘ
Used primarily as <key,value> store
âȘ
Data randomly distributed amongst large set of logical instances
âȘ
Most data access based on global id
âȘ
Large number of logical instances spread out across physical nodes
âȘ
Load balancing at physical node level
âȘ
No read replication
âȘ
11. MySQL: What We Learnt (ing)
Logical migration of data is very difficult
âȘ
Create a large number of logical dbs, load balance them over varying
âȘ
number of physical nodes
No joins in production
âȘ
Logically difficult (because data is distributed randomly)
âȘ
Easier to scale CPU on web tier
âȘ
12. MySQL: What we Learnt (ing)
Most data access is for recent data
âȘ
Optimize table layout for recency
âȘ
Archive older data
âȘ
Donât ever store non-static data in a central db
âȘ
CDB makes it easier to perform certain aggregated queries
âȘ
Will not scale
âȘ
Use services or memcache for global queries
âȘ
E.g.: What are the most popular groups in my network
âȘ
13. MySQL: Customizations
No extensive native MySQL modifications
âȘ
Custom partitioning scheme
âȘ
Global id assigned to all data
âȘ
Custom archiving scheme
âȘ
Based on frequency and recency of data on a per-user basis
âȘ
Extended Query Engine for cross-data center replication, cache
âȘ
consistency
14. MySQL: Customizations
Graph based data-access libraries
âȘ
Loosely typed objects (nodes) with limited datatypes (int, varchar, text)
âȘ
Replicated connections (edges)
âȘ
Analogous to distributed foreign keys
âȘ
Some data collocated
âȘ
Example: User profile data and all of userâs connections
âȘ
Most data distributed randomly
âȘ
15. Memcache
High-Performance, distributed in-memory hash table
âȘ
Used to alleviate database load
âȘ
Primary form of caching
âȘ
Over 25TB of in-memory cache
âȘ
Average latency < 200 micro-seconds
âȘ
Cache serialized PHP data structures
âȘ
Lots and lots of multi-gets to retrieve data spanning across graph edges
âȘ
16. Memache: Customizations
Memache over UDP
âȘ
Reduce memory overhead of thousands of TCP connection buffers
âȘ
Application-level flow control (optimization for multi-gets)
âȘ
On demand aggregation of per-thread stats
âȘ
Reduces global lock contention
âȘ
Multiple Kernel changes to optimize for Memcache usage
âȘ
Distributing network interrupt handling over multiple cores
âȘ
Opportunistic polling of network interface
âȘ
18. Under the Covers
Get my profile data
âȘ
Fetch from cache, potentially go to my DB (based on user-id)
âȘ
Get friend connections
âȘ
Cache, if not DB (based on user-id)
âȘ
In parallel, fetch last 10 photo album ids for each of my friends
âȘ
Multi-get; individual cache misses fetches data from db (based on photo-
âȘ
album id)
Fetch data for most recent photo albums in parallel
âȘ
Execute page-specific rendering logic in PHP
âȘ
Return data, make user happy
âȘ
20. LAMP is not Perfect
PHP+MySQL+Memcache works for a large class of problems but not for
âȘ
everything
PHP is stateless
âȘ
PHP not the fastest executing language
âȘ
All data is remote
âȘ
Reasons why services are written
âȘ
Store code closer to data
âȘ
Compiled environment is more efficient
âȘ
Certain functionality only present in other languages
âȘ
21. Services Philosophy
Create a service iff required
âȘ
Real overhead for deployment, maintenance, separate code-base
âȘ
Another failure point
âȘ
Create a common framework and toolset that will allow for easier
âȘ
creation of services
Thrift
âȘ
Scribe
âȘ
ODS, Alerting service, Monitoring service
âȘ
Use the right language, library and tool for the task
âȘ
23. Thrift
Lightweight software framework for cross-language development
âȘ
Provide IDL, statically generate code
âȘ
Supported bindings: C++, PHP, Python, Java, Ruby, Erlang, Perl, Haskell
âȘ
etc.
Transports: Simple Interface to I/O
âȘ
Tsocket, TFileTransport, TMemoryBuffer
âȘ
Protocols: Serialization Format
âȘ
TBinaryProtocol, TJSONProtocol
âȘ
Servers
âȘ
Non-Blocking, Async, Single Threaded, Multi-threaded
âȘ
24. Hasnât this been done before? (yes.)
SOAP
âȘ
XML, XML, and more XML
âȘ
CORBA
âȘ
Bloated? Remote bindings?
âȘ
COM
âȘ
Face-Win32ClientSoftware.dll-Book
âȘ
Pillar
âȘ
Slick! But no versioning/abstraction.
âȘ
Protocol Buffers
âȘ
25. Thrift: Why?
Itâs quick. Really quick.
âą
Less time wasted by individual developers
âą
No duplicated networking and protocol code
âą
Less time dealing with boilerplate stuff
âą
Write your client and server in about 5 minutes
âą
Division of labor
âą
Work on high-performance servers separate from applications
âą
Common toolkit
âą
Fosters code reuse and shared tools
âą
26. Scribe
Scalable distributed logging framework
âȘ
Useful for logging a wide array of data
âȘ
Search Redologs
âȘ
Powers news feed publishing
âȘ
A/B testing data
âȘ
Weak Reliability
âȘ
More reliable than traditional logging but not suitable for database
âȘ
transactions.
Simple data model
âȘ
Built on top of Thrift
âȘ
27. Other Tools
SMC (Service Management Console)
âȘ
Centralized configuration
âȘ
Used to determine logical service -> physical node mapping
âȘ
28. Other Tools
ODS
âȘ
Used to log and view historical trends for any stats published by service
âȘ
Useful for service monitoring, alerting
âȘ
29. Open Source
Thrift
âȘ
http://developers.facebook.com/thrift/
âȘ
Scribe
âȘ
http://developers.facebook.com/scribe/
âȘ
PHPEmbed
âȘ
http://developers.facebook.com/phpembed/
âȘ
More good stuff
âȘ
http://developers.facebook.com/opensource.php
âȘ
31. NewsFeed â The Work
friendsâ
actions
Leaf Server
web tier
Html
Leaf Server
Actions (Scribe)
PHP
home.php Leaf Server
user
Leaf Server
return
view state
view aggregators
state
storage friendsâ
actions?
aggregating...
- Most arrows indicate thrift calls ranking...
33. Search â The Work
Thrift
search tier
slave slave master slave
index index index index
user
web tier
Scribe live db
PHP change index
logs files
Indexing service
DB Tier
Updates