Spilgames Storage Platform
•
0 gostou•570 visualizações
A presentation of the Spilgames Storage Platform, a massively scalable solution to abstract your storage with transparent sharding and expanding your datacenters worldwide.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Spilgames Storage Platform
Semelhante a Spilgames Storage Platform (20)
Último
Último (20)
Spilgames Storage Platform
- 1. Introduction Spilgames & Me Another Storage Motivation Spilgames Storage Platform System Properties Overview Internals Versions & Shards Enrique Paz Worldwide Scaling Datacenters Senior Backend Developer Disaster Scenarios Lessons Learned 21/03/2013 Currents Status Contributions 1/42
- 2. Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Introduction Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 2/42
- 3. About Me Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards • Passionate Erlang developer Worldwide Scaling • Testing enthusiast Datacenters Disaster Scenarios • Love beautiful code! Lessons Learned Currents Status Contributions 3/42
- 4. Spilgames Introduction Spilgames & Me Another Storage Motivation • Gaming Platform System Properties • Serving data to 190+ countries world-wide Overview Internals Versions & Shards • 200+ million unique users per month Worldwide • Multiple Platforms: Desktop, Mobile Native & Web Scaling Datacenters Disaster • 300+ employees Scenarios Lessons • Offices in The Netherlands & China Learned Currents Status • Revenue: Advertising & EUM Contributions 4/42
- 5. Gaming Portals Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 5/42
- 6. Gaming Portals Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 6/42
- 7. Games Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 7/42
- 8. Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Another Storage Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 8/42
- 9. Another Storage Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 9/42
- 10. LAMP Stack & MySql Introduction Spilgames & Me Another Storage Motivation System Properties • Not all developers are DB experts Overview Internals • Difficult to shard the databases Versions & Shards Worldwide • Storage model all over the place Scaling Datacenters Disaster • Security Scenarios Lessons • Performance Learned Currents Status • Caching Contributions • ... 10/42
- 11. Our Ambition Introduction Spilgames & Me Another Storage Motivation • Transparent sharding layer System Properties Overview • Sharding on data ownership Internals Versions & Shards • High availability Worldwide Scaling • Centralized caching layer Datacenters Disaster Scenarios • Storage engine agnostic Lessons • One strict data model Learned Currents Status Contributions • Transparent storage changes • Scaling geographically 11/42
- 12. Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards System Properties Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 12/42
- 13. Mindset Introduction Spilgames & Me Another Storage Motivation System • Be always available Properties Overview • Avoid global locks Internals Versions & Shards • Accept change as the only constant Worldwide Scaling • Embrace inconsistencies Datacenters Disaster Hardware breaks down (power failures) Scenarios Version mismatches (upgrading system not atomic) Lessons Learned State mismatches (adding new machines) Currents Status Contributions 13/42
- 14. A Key-Value Store With Schema Introduction Spilgames & Me • Buckets Another Storage Largely generated OTP applications Motivation Offer a CRUD-like interface (with filters) System Properties Overview • GIDs (64 bits) identify the data owner Internals Versions & Shards user Worldwide game Scaling Datacenters • Buckets can use different storage engines Disaster Scenarios Several MySql tables in different databases Lessons Just a binary storage (SWIFT) Learned Currents Status ... Contributions • Data for a bucket/GID is cached • Requests can be atomic per bucket/GID 14/42
- 15. Optimistic Operations Introduction Spilgames & Me Another Storage Motivation System • Speed > Consistency Properties Overview Internals • Losing some updates in case of crash is affordable Versions & Shards • Act first on cache and then on disk Worldwide Scaling Datacenters • No warranties of eventual consistency upon crashes Disaster Scenarios i.e. Activity feeds Lessons i.e. Popular games list Learned Currents Status Contributions 15/42
- 16. Pessimistic Operations Introduction Spilgames & Me Another Storage Motivation System Properties • Consistency is key and confirmation is required Overview Internals Versions & Shards • Dealing with critical data Worldwide • Persist data and, upon success, update cache Scaling Datacenters i.e. Payments Disaster Scenarios i.e. Personal information Lessons Learned Currents Status Contributions 16/42
- 17. How does it work? Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 17/42
- 18. System Components Introduction Spilgames & Me Another Storage Motivation System • lookup application in all nodes. Properties Overview Uses a hashing ring (mnesia): Internals Versions & Shards replicated in all nodes Worldwide ram_copies Scaling Datacenters dirty reads Disaster Scenarios transactional writes Lessons Learned • Buckets have Pipeline Factories Currents Status Contributions • Buckets register PFs in lookup 18/42
- 19. Tracing Pessimistic Operations Introduction Spilgames & Me Another Storage Motivation System Properties Overview 1. Bucket/GID request in a node Internals Versions & Shards 2. Local lookup to find a PF Worldwide Scaling 3. PF receives request Datacenters Disaster Scenarios 4. PF builds job Lessons Learned 5. PF ensures Pipeline for GID Currents Status Contributions 6. PF queues operation in Pipeline 19/42
- 20. Wait a minute. . . Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals • “Why do we need pipelines?” Versions & Shards Worldwide • “Sequential == Bottleneck !!!” Scaling Datacenters • “Don’t you guys know Erlang is Disaster Scenarios about parallelizing work?” Lessons Learned Currents Status Contributions 20/42
- 21. About Pipelines Introduction Spilgames & Me Another Storage Motivation • CONS System Properties Sequential (read) access for hotspots Overview Internals i.e. Popular games Versions & Shards Optimization: read from SSP cache in PF Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 21/42
- 22. About Pipelines Introduction Spilgames & Me Another Storage Motivation • CONS System Properties Sequential (read) access for hotspots Overview Internals i.e. Popular games Versions & Shards Optimization: read from SSP cache in PF Worldwide Scaling • PROS Datacenters Disaster No need for storage engines to support global locks Scenarios A bucket can combine several engines Lessons Learned Currents Status Contributions 21/42
- 23. About Pipelines Introduction Spilgames & Me Another Storage Motivation • CONS System Properties Sequential (read) access for hotspots Overview Internals i.e. Popular games Versions & Shards Optimization: read from SSP cache in PF Worldwide Scaling • PROS Datacenters Disaster No need for storage engines to support global locks Scenarios A bucket can combine several engines Lessons Learned Currents Status • Requests to most GIDs (users) are evenly distributed Contributions 21/42
- 24. Schema Versions Introduction Spilgames & Me Another Storage Motivation System Properties Overview • Schema Versions determine allowed operations and storage(s) Internals Versions & Shards • Client is not aware of them Worldwide Scaling • Max 2 schema versions of a bucket at the time Datacenters Disaster Scenarios • Schema version can be changed at bucket/GID level Lessons Learned Currents Status Contributions 22/42
- 25. Shards Introduction Spilgames & Me Another Storage Motivation System Properties Overview • Useful for partitioning big blocks of data Internals Versions & Shards • Shard points to the physical location of the data Worldwide Scaling • Sharding rules are bucket specific. Default is GID % Shards Datacenters Disaster Scenarios • bucket/GID combinations can be migrated between shards Lessons Learned Currents Status Contributions 23/42
- 26. Working With Versions & Shards Introduction Spilgames & Me Another Storage Motivation 1. insert(Bucket, Gid, Value) System Properties 2. insert(Gid, Value) Overview Internals Versions & Shards 3. get_vs(Bucket, Gid) Worldwide 4. {v2, Shard1} Scaling Datacenters Disaster 5. build_job(insert, Gid, Shard1) Scenarios Lessons 6. {ok, InsertJob} Learned Currents Status 7. {ok, InsertJob} Contributions 24/42
- 27. One API To Rule Them All Introduction Spilgames & Me Another Storage • “Don’t care where it is, just want my data!!!” Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 25/42
- 28. One API To Rule Them All Introduction Spilgames & Me Another Storage • “Don’t care where it is, just want my data!!!” Motivation System • PIQI helps with the API Properties Overview Internals Versions & Shards Worldwide Scaling Erlang client + Protocol Buffers Datacenters Disaster Scenarios HTTP + JSON Lessons Learned HTTP + Protocol Buffers Currents Status Contributions 25/42
- 29. Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 26/42
- 30. Worlwide Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 27/42
- 31. Masters & Satellites Introduction Spilgames & Me Another Storage Motivation System • Master Datacenters Properties Have persistent storage Overview Internals Can own GIDs Versions & Shards GIDs can be migrated between Master DCs Worldwide Scaling Datacenters • Satellite Datacenters Disaster Scenarios Don’t have persistent storage Lessons Easy to setup and decommission Learned Virtual/Cloud-based Currents Status Contributions 28/42
- 32. Start With One DC Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 29/42
- 33. Scale Up A Satellite Where Needed Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 30/42
- 34. Turn Satellites Into Masters When Ready Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 31/42
- 35. Working With Multiple Datacenters Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 32/42
- 36. Disaster Scenarios Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 33/42
- 37. Losing A Satellite Datacenter Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 34/42
- 38. Losing A Master Datacenter Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 35/42
- 39. Losing A Master Datacenter Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 36/42
- 40. Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Lessons Learned Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 37/42
- 41. Where We Are Introduction Spilgames & Me Another Storage Motivation • Using simple buckets on LIVE in one DC System Properties • Added relup support for bucket only updates Overview Internals Versions & Shards • Hammering SSP using property based testing Worldwide • Integrating restricted search capabilities Scaling Datacenters Disaster • Testing the WAN protocol for inter DC communication Scenarios Lessons • More buckets to go live in H1 2013 Learned Currents Status • Satellite DCs coming on H2 2013 Contributions 38/42
- 42. What We’ve Used Introduction Spilgames & Me • Emysql (+) multi-database transaction support Another Storage (*) multi-timezone support Motivation • Eep0018/Jiffy System Properties • Estatsd Overview Internals • PropEr Versions & Shards Worldwide • Poolboy Scaling Datacenters • Lager Disaster Scenarios • Rebar Lessons (*) semantic versioning, i.e. [">=1.3.1", "<2.0.0"] Learned Currents Status (*) shared dependencies Contributions (*) xref fixes • Piqi • BashoBench (+) Several tests on the same plot support 39/42
- 43. Questions? Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 40/42
- 44. That’s All Folks! Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Thanks! Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions 41/42
- 45. Join Us! Introduction Spilgames & Me Another Storage Motivation System Properties Overview Internals Versions & Shards Worldwide Scaling Datacenters Disaster Scenarios Lessons Learned Currents Status Contributions http://www.spilgames.com/careers/job-openings/ 42/42