Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Squeezing scale out of MySQL with data tiering
Semelhante a Squeezing scale out of MySQL with data tiering (20)
Último
Último (20)
Squeezing scale out of MySQL with data tiering
- 1. data tiering Squeezing scale out of MySQL Julien, VP Engineering at HouseTrip github.com/mezis
- 2. disclaimer IANADBA* I’m not a database administrator 3
- 3. the scene 4
- 4. Load balancer (ELB) 3k rpm 30x10 web workers (Passenger/Rack) 6x20 job workers (DJ) 85k qpm Memcache 4x7GB MySQL 400GB 5x replica MondoDB 120GB 2x replica S3 TBs 5
- 6. search = [destination, start date, end date] ↓ [ [property, price], … ] property availability rate ! ! ! destination_id property_id rate_id start_date end_date price 7
- 7. search → booking → big bulky join long transaction + business logic + many small R/W property availability rate ! ! ! destination_id property_id rate_id start_date end_date price 8
- 8. the crisis 9
- 9. peak traffic 7pm - 10pm write queries ! read queries " write IO ⛅️ cpu load ⛅️ memory ☀️ 10
- 10. contention noun (kənˈtɛnʃən) 1. a struggling between opponents 2. competition for limited resources 11
- 11. slow reads ? poor use of indices during large write transactions http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-transaction-management.html http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_covering_index 12
- 12. slow writes ? load+locking on rollback segments http://dev.mysql.com/doc/refman/5.1/en/innodb-multi-versioning.html http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_rollback_segment 13
- 13. digging & deeper SHOW ENGINE INNODB STATUS ' ---TRANSACTION 72C, ACTIVE 755 sec 4 lock struct(s), …, 3 row lock(s), undo log entries 12 TABLE LOCK … RECORD LOCKS … RECORD LOCKS … locks rec but not gap RECORD LOCKS … lock_mode X locks gap before rec http://www.mysqlperformanceblog.com/2012/03/27/innodbs-gap-locks/ http://dev.mysql.com/doc/refman/5.1/en/innodb-monitors.html 14
- 14. 15
- 16. horizontal scaling “throw money at it” → does not work → ops + maintenance cost 17
- 17. fine-tune the DB engine “bring in the experts” → only works short-term 18
- 18. vertical scaling “throw more money at it” → bigger database instances ~ +4k$/mo → only 2 cartridges in that gun 19
- 19. put it in a service de-normalise the data use that noSQL thing webscale webscale websc ale 20
- 20. good solution ? → live in 1-2 weeks → buys 6-12 months http://xkcd.com/1205/ 21
- 23. frame tearing caused by “simple buffering” render scene in buffer draw to screen from buffer t t 24
- 24. frame(buffer) tiering aka “double buffering” render scene in buffer draw to screen from buffer t t 25
- 25. data tiering separate read and write tables read availabilities_front swap availabilities_back read/write copy availabilities http://en.wikipedia.org/wiki/File:Comparison_double_triple_buffering.svg 26
- 26. how it works 27
- 28. using tiered tables DataTiering::Switch.new.active_scope_for(Availability) equivalent one of Availability.scoped(:from => 'properties_0') Availability.scoped(:from => 'properties_1') read availabilities_front swap availabilities_back read/write copy availabilities 29
- 29. syncing DataTiering::Sync.sync_and_switch! regularly* scheduled task (every 5 min for us, takes ~ 60s) *depend on acceptable staleness read availabilities_front swap availabilities_back read/write copy availabilities 30
- 30. syncing: schema lazily (no migrations) : - create missing /.*_[01]/ tables - compare schemas with SHOW CREATE TABLE read availabilities_front swap availabilities_back read/write copy availabilities 31
- 31. syncing: bulk - run TRUNCATE TABLE then INSERT INTO … SELECT FROM - too slow at runtime (only for setup / after migrations) read availabilities_front swap availabilities_back read/write copy availabilities 32
- 32. syncing: deltas - deletions : SELECT id … LEFT JOIN … DELETE … WHERE id IN … - insertions & updates : read REPLACE INTO … row_touched_at > X availabilities_front swap - remember last sync in data_tiering_sync_logs availabilities_back - row_touched_at “magic” copy read/write TIMESTAMP columnavailabilities 33
- 33. swapping - renaming tables not transactional - atomically change a pointer instead (and cache it) → data_tiering_switches read availabilities_front swap availabilities_back read/write copy availabilities 34
- 35. outcome 36
- 36. outcome average DB time unchanged nominal at peak traffic deadlocks timeouts faster reads 37
- 37. epilogue 38
- 38. so long, data tiering ! you served us well 39