Why choose distributed, AP stores?

1. Databases
Sargun Dhillon
@Sargun

2. What is a database?
A database is an organized collection of data

3. Applications
What are databases for?

4. Internet Applications
Experiencing exploding growth

5. Internet Traffic vs. Penetration
0
25
50
75
100
0
10000
20000
30000
40000
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
IP Traffic (PB/mo) Global Penetration (%)

6. Number of Internet Users in 2012

7. Average Distance to Every Human

8. Extrapolating
We have not yet Peak “Web” and we wont see it for
some time

9. Applications
How are they built?

10. Basic Application

11. Useful Application
Add Persistence

12. Scale Out

13. Scale Out with Correctness

14. What is a Transaction?
A Unit of Work

15. Transaction Scheduling
Concurrent Operations

16. Non-Conflicting Concurrency
Parallel Execution

17. ACID

18. ACID = Atomicity
A transaction executes or it does not

19. ACID = Consistency
Correctness; Require the database to follow set of
invariants

20. ACID = Isolation
Prevent inter-actor visibility during concurrent operations

21. ACID = Durability
Once you write, it will survive

22. Lifecycle of a Transaction

23. Vertically Scalability
Moore’s Law can take us places

24. Biggest AWS Database
• vCPUs: 32
• Memory: 244
• Storage: 3TB
• IOPs: 30,000 IOPs
• Networking: 10 Gigabit
• Resiliency: Multi-AZ
• SLA: 99.95%
• Backend: Postgresql

25. $141,052.66/yr

26. Scaling Beyond

27. Sharding?

29. Do we have a natural
sharding key?

30. Add a Coordinator?

31. Two-phase commit?
Three-phase commit?
Paxos?
Enhanced Three-phase commit?
Wat?
Egalitarian Paxos?

32. Do we really want to
run NxM databases?

33. Partial Availability

34. Failure detectors are
hard

35. Database Failure

36. Cascading App Failure

37. Recovery

38. Hotspots?
(The “Beiber” problem)

39. Scaling SSI databases
is a hard problem

40. What if want
multidatacenter?

43. No latency win for
mutable data

44. Must sacrifice recency
for latency win

45. Complex Routing
Semantics

46. Multi-master requires
at least 1 RTT

47. 80ms+ writes!

48. “Average partition duration ranged from 6 minutes for
software-related failures to more than 8.2 hours for
hardware-related failures (median 2.7 and 32 minutes;
95th percentile of 19.9 minutes and 3.7 days,
respectively).”
-The Network is Reliable
WANs Fail

49. Is there another way?

50. Into Riak

51. Design Requirements

52. Incremental Scalability
Must be able to add nodes for greater reliability, or
throughput

53. High Availability
Must be able to seamlessly handle failures, and always
respond to operations

54. Efficiency
Meet stringent latency requirements

55. Implementation

56. “Experience at Amazon has shown
that data stores that provide
ACID guarantees tend to have
poor availability.”
Dynamo: Amazon’s Highly Available Key-value Store

57. The Ring
A cluster composed of set of virtual nodes (vnodes)

58. The Ring

59. Virtual Node Placement

60. The Ring

61. Data Placement

62. Data Placement

63. Fault Tolerance

64. Hinted Handoff

65. Fallback Virtual Nodes

66. Hinted Handoff

67. Read Repair

68. Replicas

69. Partial Failure

70. Divergence

71. Read Repair

72. Read Repair

73. Read Repair

74. Active-Anti Entropy

75. Merkle Tree

76. Compare Trees

77. Compare Trees

78. Compare Trees

79. Compare Trees

80. Repair Trees

81. Fault Tolerance
• Read Repair
• Active Anti-Entropy
• Hinted Handoff

82. Eventual Consistency

83. CAP Theorem

84. “A shared-data system can have at most
two of the three following properties:
Consistency, Availability, and tolerance to
network Partitions.”
-Dr. Eric Brewer

85. On Consistency
• ACID Consistency: Any transaction, or operation
will bring the database from one valid state to
another
• CAP Consistency: All nodes see the same data at
the same time (synchrony)

86. On Partition Tolerance
• The network will be allowed to lose arbitrarily many
messages sent from one node to another.
• Databases systems, in order to be useful must
have communication over the network
• Clients count

87. There is no such thing as
a 100% reliable network:
Can’t choose CA
http://codahale.com/you-cant-sacrifice-partition-tolerance

88. Very “AP”

89. Weak Consistency

90. Weak Consistency

91. “This is a specific form of weak
consistency; the storage system
guarantees that if no new
updates are made to the object,
eventually all accesses will
return the last updated value.”
Definition of “Eventual Consistency” from “Eventually
Consistency Revisited” - Werner Vogels

93. Tunable CAP Controls
• R (Read Acks) tunable: Default Quorum
• W (Write Acks) tunable: Default Quorum
• PR (Primary Read Acks) tunable: Default 0
• PW (Primary Write Acks) tunable: Default 0
• N (replicas) tunable: Default 3

94. Strong Eventual Consistency
PW+PR>N

95. How do you even use this?

96. Vector Clocks

97. Vector Clocks
• Extension of Lamport Clocks
• Used to detect cause and effect in distributed
systems
• Can determine concurrency of events, and
causality violations

98. CRDTs

99. • CRDTs:
• Convergent Replicated Data Types
• Commutative Replication Data Types
• Enables data structures to be always writeable on both sides of a partition,
and replay after healing a partition
• Enable distributed computation across monotonic functions
• Two Types:
• CvRDTs
• CmRDTs
CRDTs

100. CvRDTs
• State / value based CRDTs
• Minimal state
• Don’t require active garbage collection

101. Set CvRDT

102. CmRDTs
• Op / method based CRDTs
• Size grows monotonically
• Uses version vectors to determine order of
operations

103. Counter CmRDT

104. CRDTs in the Wild
• Sets
• Observe-remove set
• Grow-only sets
• Counters
• Grow-only counters
• PN-Counters
• Flags
• Maps

105. Data structures that are
CRDTs
• Probabilistic, convergent data structures
• Hyper log log
• Bloom filter
• Co-recursive folding functions
• Maximum-counter
• Running Average
• Operational Transform

106. CRDTs
• Incredibly powerful primitive
• Not only useful for in-database manipulation but
client-database interaction
• You can compose them, and build your own
• Garbage collection is tricky

107. RAMP: Read Atomic
Multi-Partition
Transactions

108. Multikey Transaction

109. Potential Consistency Violation

110. Add Metadata

111. Uncommitted State

112. Uncommitted State

113. Committed State

114. Have your availability
and consistency too
RAMP

115. Eventual Consistency
in the WAN

116. Low-latency
everywhere

117. Write Anywhere
Beat the speed of the light

118. MDC Replication

119. Hybrid Topologies

120. Bidirectional Replication

121. Unidirectional Replication

122. Replication Hooks

123. Tied Writes

124. Hook on Replication

125. Hook on Replication

126. Replicate Hook Return Data

127. Build for WAN locality

128. Eventual Consistency
In Summary

129. Invariant Operation AP / CP
Specify unique ID Any CP
Generate unique ID Any AP
> INCREMENT AP
> DECREMENT CP
< INCREMENT CP
< DECREMENT AP
Secondary Index Any AP
Materialized View Any AP
AUTO_INCREMENT INSERT CP
Linearizability CAS CP
Operations Requiring
Weak Consistency
vs.
Strong Consistency

130. BASE not ACID
•Basically Available: There will be a response
per request (failure, or success)
•Soft State: Any two reads against the system
may yield different data (when measured
against time)
•Eventually Consistent: The system will
eventually become consistent when all
failures have healed, and time goes to infinity

131. Deploying Riak

132. AWS Deployment
• 6 x i2.4xlarge
• 732GB of RAM
• 19TB of storage
• 960,000 IOPs
• 96 vCPUs
• 3 x Replication
• 10 Gigabit networking
• 99.9999999997% availability

133. $74,790/yr

134. Real World Use Case

135. Ad Network
• Sell targeted ads with minimum latency
• Two datasets:
• Ads
• Users

136. Deployment

137. Deployment

138. Overselling Ads is
Okay

139. Choose Random Ad
Based on Weight of
Outstanding Impressions

140. Batch System

141. Batch System
Generated targeted ads in offline process

142. Ad Graph

143. Ad Store

144. Initial Visit

145. Fetch All Ads

146. Choose Ad
Based upon weighted random

147. Decrement Value

148. Test Model
• 50 actors
• 5 Ads with inventory between 1000, and 1200
• Actors randomly get [1,3] times to choose per
round
• Rounds continue until entire inventory is exhausted

149. Test Model
OutstandingImpressions
-300
0
300
600
900
1200
Round Number
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76
Ad 1 Ad 2 Ad 3 Ad 4 Ad 5

150. Garbage Collection

151. Garbage Collection
Utilizes secondary indexes in batch process to delete
exhausted ads from user records

152. Ad Serving
• Requires batch generation of targets
• Requires external GC
• Allows for multidatacenter operation

153. In Summary

154. Riak

155. Distributed

156. Fault-Tolerant

157. Scalable
Scalability
Processors

158. Toolchest

159. Why
Distributed
Databases?
Sargun Dhillon
@Sargun