Scaling ScyllaDB Storage Engine with State-of-Art Compaction

1. Squeezing the Most Out of the Storage Engine with State of the Art Compaction Raphael S. Carvalho, Software Engineer

2. Raphael Carvalho ■ Syslinux, suite of bootloaders ■ OSv, an operating system for the cloud ■ Seastar, the framework powering ScyllaDB ■ ScyllaDB, the best database in the world

3. “In order to make good use of the computer resources, one must organize files intelligently, making the retrieval process efficient.” The Ubiquitous B-Tree paper, 1979

4. ■ Short & precise deﬁnition from aforementioned paper: ■ “allow users to store, update, and recall” Storage Engines

5. ■ Two approaches for handling updates ■ In-place structure (B+-tree) Storage Engines

6. ■ Two approaches for handling updates ■ In-place structure (B+-tree) Storage Engines (k1,v1)(k2,v2)

7. ■ Two approaches for handling updates ■ In-place structure (B+-tree) Storage Engines (k1,v1)(k2,v2) (k1, v3)

8. ■ Two approaches for handling updates ■ In-place structure (ex: B+-tree) Storage Engines (k1,v3)(k2,v2)

9. ■ Two approaches for handling updates ■ Out-of-place structure (ex: LSM-tree) Storage Engines

10. ■ Two approaches for handling updates ■ Out-of-place structure (ex: LSM-tree) Storage Engines (k1,v1)(k2,v2)

11. ■ Two approaches for handling updates ■ Out-of-place structure (ex: LSM-tree) Storage Engines (k1,v1)(k2,v2) (k1,v3)

15. ■ Out-of-place update isn’t new. ■ 1976 paper “Differential ﬁles” shows its applicability in the real world ■ “shown to be an eﬃcient method for storing a large and changing database” Storage Engines

16. ■ A good analogy is presented in the paper Storage Engines

17. ■ The Log-Structured Merge-Tree (LSM-Tree) paper is then published in 1996 Storage Engines

18. Storage Engines THE LSM-TREE writes C0 C1 C2 Ck MEMORY DISK merge sort

19. Storage Engines THE LSM-TREE C1 is T times bigger than C0. C(K) is T times bigger than C(K-1). C0 C1 C2 Ck MEMORY DISK merge sort

20. ■ Immutability of LSM tree components (ex: SSTables) simpliﬁes ■ Concurrency control ■ Recovery Storage Engines

21. Query on LSM Tree (k1, v2) (k1, v1) MEMORY DISK Query k1

22. ■ A compaction policy (or strategy) deﬁnes the shape of LSM tree ■ Any policy is composed of 4 primitives ■ Trigger (when to compact) ■ File picking policy (which data to compact) ■ Granularity (how much data at once) ■ Layout (how data is laid out) LSM-tree compaction policy

23. Pure Leveled in Original LSM Design ONLY 1 COMPONENT PER LEVEL! C0 C1 C2 Ck MEMORY DISK merge sort

24. Flexible Leveled in Modern LSM Design MEMORY DISK L0 L1

27. ■ Partitions the LSM-tree components into (usually ﬁxed-size) fragments ■ Subset of a level can be merged into the next one (partial merge) ■ Bounds: ■ compaction operation time ■ temporary disk space during compaction lifetime Partitioning Optimization for Leveled

28. Partitioning Optimization for Leveled MEMORY DISK L1 L2 KEY RANGE SST SST SST SST SST SST

29. Partitioning Optimization for Leveled MEMORY DISK L1 L2 KEY RANGE SST SST SST SST SST SST

30. Leveled Policy - Cost Analysis ■ Let T be the size ratio between adjacent levels ■ Let L be the number of levels for a given LSM tree ■ Write ampliﬁcation: ■ Space ampliﬁcation: O(T * L) O(T + 1) ------ = ~1.1 T

31. Stepped-Merge Algorithm ■ 1997 paper Incremental organization for data recording and warehousing -> a new approach to LSM tree layout ■ “Our goal is to design a technique that supports both insertion and queries with reasonable eﬃciency, and without the delays of periodic batch processing.” ■ Gives birth to the tiered compaction policy

32. Tiered Compaction Policy MEMORY DISK L0 L1 SST FILE SIZE

33. Tiered Compaction Policy MEMORY DISK L0 L1 SST FILE SIZE SST

34. Tiered Compaction Policy MEMORY DISK L0 L1 FILE SIZE SST

35. Tiered Policy - Cost Analysis ■ Let T be the size ratio between adjacent levels ■ Let L be the number of levels for a given LSM tree ■ Write ampliﬁcation: ■ Space ampliﬁcation: O(L) O(T * L)

36. Now ScyllaDB journey begins The database inherited all the LSM-tree improvements described so far… But they weren’t enough

37. Tiered - Temporary Space Problem! MEMORY DISK L0 L1 FILE SIZE SST SST

38. Tiered - Temporary Space Problem! MEMORY DISK L0 L1 FILE SIZE SST SST SST 100% TEMP SPACE OVERHEAD

39. Partitioning Optimization for Tiered MEMORY DISK L0 L1 FILE SIZE S S T S S T

40. Partitioning Optimization for Tiered MEMORY DISK L0 L1 FILE SIZE S S T S S T S

41. Partitioning Optimization for Tiered MEMORY DISK L0 L1 FILE SIZE S T S T S

42. Tiered Policy - Partitioning Optimization ■ Bounds temporary space overhead signiﬁcantly ■ Allows disk space usage from 50% to 80% and beyond. ■ Available in ScyllaDB as Incremental Compaction Strategy (ICS)

43. LSM tree - Efficiency Space SPACE OPTIMIZED WRITE OPTIMIZED

44. LSM tree - Efficiency Space SPACE OPTIMIZED WRITE OPTIMIZED PURE LEVELED

45. LSM tree - Efficiency Space SPACE OPTIMIZED WRITE OPTIMIZED PURE LEVELED PURE TIERED

46. But the world is not only black and white There are shades of gray in between…

47. Hybrid LSM-tree data layout ■ Largest level is space optimized ■ Other levels are write optimized ■ Addresses O(K) space ampliﬁcation in tiered in overwrite workloads ■ Where K = number of components per level

48. Hybrid LSM-tree data layout L1 L2 FILE SIZE L0 SST SST SST SST WRITE OPTIMIZED LEVELS SPACE OPTIMIZED LEVEL

49. Hybrid LSM-tree data layout L1 L2 FILE SIZE L0 SST SST WRITE OPTIMIZED LEVELS SPACE OPTIMIZED LEVEL SST

50. Hybrid LSM - Efficiency Space SPACE OPTIMIZED WRITE OPTIMIZED PURE TIERED PURE LEVELED HYBRID

51. Hybrid LSM - Efficiency Space SPACE OPTIMIZED WRITE OPTIMIZED PURE TIERED PURE LEVELED HYBRID

52. Hybrid LSM-tree data layout ■ Reduces space amplification in overwrite-intensive workloads ■ = less space amplification ■ = increased storage density per node ■ = more money in your pocket. ■ Available as space amplification goal (SAG) option of Incremental Compaction Strategy.

53. LSM-tree & tombstones MEMORY DISK L0 L1 FILE SIZE KEY A

54. LSM-tree & tombstones MEMORY DISK L0 L1 FILE SIZE KEY A KEY A TOMBSTONE

55. LSM-tree & tombstones MEMORY DISK L0 L1 FILE SIZE KEY A KEY A

56. Suboptimal LSM-tree tombstone handling MEMORY DISK L0 L1 FILE SIZE KEY A KEY A GARBAGE COLLECTION

57. Efficient LSM-tree tombstone handling MEMORY DISK L0 L1 FILE SIZE KEY A KEY A GARBAGE COLLECTION

58. Efficient LSM-tree tombstone handling ■ Piggyback on incremental compaction, to bound temporary disk space. ■ Triggers (avoids write ampliﬁcation issues): ■ File staleness ■ Tombstone density threshold ■ Available in Incremental Compaction Strategy (ICS) by default.

59. Thank You Stay in Touch Raphael Carvalho raphaelsc@scylladb.com @raphael_scarv raphaelsc

Scaling ScyllaDB Storage Engine with State-of-Art Compaction

Esté a ler uma amostra.

Confira estes a seguir

Scaling ScyllaDB Storage Engine with State-of-Art Compaction

Mais Conteúdo rRelacionado

Semelhante a Scaling ScyllaDB Storage Engine with State-of-Art Compaction (20)

Mais de ScyllaDB (20)

Mais recentes (20)