Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (16)
Semelhante a Scaling Twitter with Cassandra
Semelhante a Scaling Twitter with Cassandra (20)
Último
Último (20)
Scaling Twitter with Cassandra
- 2. Scaling Twitter with Cassandra Ryan King Storage Team
- 4. Legacy • vertically & horiztonally partitioned mysql • memcached (rows, indexes and fragments) • application managed
- 5. Legacy Drawbacks • many single-points-of-failure • hardware-intensive • manpower-intensive • tight coupling
- 6. Apache Cassandra • Apache top level project • originally developed at Facebook • Rackspace, Digg, SimpleGeo, Twitter, etc.
- 7. Why Cassandra? • highly available • consistent, eventually • decentralized • fault tolerant • elastic • flexible schema • high write throughput
- 8. What is Cassandra? • distributed database • Google's BigTable's data model • Amazon's Dynamo's infrastructure
- 9. Cassandra Data Model • keyspaces • column families • columns • super columns
- 10. Cassandra Infrastructure • partitioners • storage • querying
- 11. Partitioners • order-preserving • random • custom
- 12. Storage • commit log • memtables • sstables • compaction • bloom filters • indexes • key cache • row cache
- 13. Querying • get • multiget • range • slice
- 14. Consistency
- 15. Consistency • N, R, W
- 16. Consistency • N, R, W • N = number of replicas
- 17. Consistency • N, R, W • N = number of replicas • R = read replicas
- 18. Consistency • N, R, W • N = number of replicas • R = read replicas • W = write replicas
- 19. Consistency • N, R, W • N = number of replicas • R = read replicas • W = write replicas • send request, wait for specified number
- 20. Consistency • N, R, W • N = number of replicas • R = read replicas • W = write replicas • send request, wait for specified number • wait for others in background and perform read- repair
- 21. Consistency Levels • ZERO • ONE • QUORUM • ALL
- 22. Strong Consistency • If W + R > N, you will have consistency • W=1, R=N • W=N, R=1 • W=Q, R=Q where Q = N / 2 + 1
- 23. Eventuality • Hinted Handoff • Read Repair • Proactive Repair (Merkle trees)
- 25. Potential Consistency • causes • write-through caching • master-slave replication failures
- 26. Example
- 27. Read Repair • send read to all replicas • if they differ, resolve conflicts and update (in background)
- 28. Hinted Handoff • A wants to write to B • B is down • A tells C, "when B is back, send them this update"
- 29. Proactive Repair • use Merkle trees to find inconsistencies • resolve conflicts • send repaired data • triggered manually
- 31. How we’re moving? • parallel deployments • incremental traffic shifting
- 32. Parallel Deployment 1. build new implementation 2. integrate it alongside existing 3. ...with switches for dynamically move/mirror traffic 4. turn up traffic 5. break something 6. Fix it 7. GOTO 4
- 33. ?
Notas do Editor
- * storage team * personal background
- * began working on this problem last june * complexity had grown unmanageable * multiple internal customers * error domain grows as data size and complexity grow
- * every master db is a SPOF (failover is hard to pull off without strong coordination) * SPOFs lead to expensive hardware * app-managed hosts is tight coupling
- * our application is already tolerant of eventual consistency (actually more tolerant...) * in addition to scale, we want more flexibility than relational data models give us
- keyspace: database CF: table column: attribute SC: collection of attributes
- [insert diagrams of ring + tokens] nodes are arranged on a ring keys are mapped to the ring and written to the next N machines partitioners map keys to the ring
- [flow chart of how updates happen]
- if OPP, rows are ordered columns are ordered [diagram of range and slice]
- insert to mysql insert into memcache replicate to slave update mysql insert into memcache fails replication to slave fails
- Launching is shifting from roll back to roll forward