SlideShare uma empresa Scribd logo
1 de 40
Baixar para ler offline
Dr. Roland Kuhn
Akka Tech Lead
@rolandkuhn
Distributed in Space and Time
The Future is Bright
2
source: http://nature.desktopnexus.com/wallpaper/58502/
The Future is Bright
• Computing Resources: Effectively Unbounded
• 18-core CPU packages, 1024 around the corner
• on-demand provisioning in the cloud
• Demand for Web-Scale Apps: Ravenous
• everyone has the tools
• successful ideas are bought for Giga-$
3
How do I make a Large-Scale App?
• figure out a service to provide to everyone
• this will require resources accordingly:
• storage
• computation
• communication
• scalability implies no shared contention points
• need to be elastic ➟ use the cloud
4
How do I make an Always-Available App?
• must not have single point of failure
• all functions and resources are replicated
• protect against data loss
• protect against service downtime
• regional content delivery
• need to distribute ➟ use the cloud
5
How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central database is not tenable
• persist “locally” and replicate changes
6
How do I make a Persistent App?
7
How do I make a Persistent App?
8
How do I make a Persistent App?
• runtime replication usually not sufficient
• system of record needed
• one central database is not tenable
• persist “locally” and replicate changes
• temporally distributed change records
• spatially distributed change log
• granularity can be one Aggregate Root (Event Sourcing)
• extract business intelligence from history
• scale & distribute persistence ➟ in the cloud
9
As Noted Earlier: The Future is Bright
10
source: http://nature.desktopnexus.com/wallpaper/58502/
… or is it?
11
Photograph: Patrick Cromerford, 2011
Distribution: a Double-Edged Sword
• did that message make it over the network?
• is that other machine running?
• are our data centers in sync?
• did we lose messages in that hardware crash?
12
The Essence of the Problem
A distributed system is one whose parts can fail
independently of each other.
13
Fighting Back!
14
source: www.hear2heal.com (Kokopelli Rain Dance)
Transactional Storage
Consensus
Message Delivery
…
Transactional Storage
• Serializability simplifies reasoning about behavior
• … but is at odds with distribution (CAP theorem)
• Eventual Consistency is often sufficient

(PeterBailis:ProbabilisticallyBoundedStaleness)
• Scalability can be regained by partitioning

(PatHelland:LifebeyondDistributedTransactions)
• consistency has a cost
• you need to choose how much is necessary
16
PAXOS and Friends
• problem: distributed consensus
• hard solutions favor Consistency
• 2PC, PAXOS, RAFT
• if a node is unreachable then consensus cannot be reached
• soft solutions favor Availability
• quorum, allow nodes to drop out
• partitions can lead to more than one active set
• consistency always has a cost
• you need to choose which one is appropriate
17
Fighting Message Loss
• resending is the only cure
• sender stores the message and retries
• recipient must send acknowledgement eventually
• fighting unreliable medium is sender’s obligation
• sender’s buffer must be persistent
• resending must begin anew after a crash
18
At-Least-Once Delivery with Akka
19
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}
At-Least-Once Delivery with Akka
20
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}
At-Least-Once Delivery with Akka
21
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}
At-Least-Once Delivery with Akka
22
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}
At-Least-Once Delivery with Akka
23
class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery {
val IDs = Iterator from 1
def receiveCommand = {
case Command(x) =>
persist(ImportantResult(x)) { ir =>
sender() ! Done(x)
deliver(other, FollowUpCommand(x, IDs.next(), _))
}
case rc: ReceiptConfirmed =>
persist(rc) { rc => confirmDelivery(rc.deliveryId) }
}
def receiveRecover = {
case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _))
case ReceiptConfirmed(_, id) => confirmDelivery(id)
}
}
Mysterious Double-Bookings
• acknowledgements can be lost
• sender will faithfully duplicate the message
• therefore named at-least-once
• recipient is responsible for dealing with this
24
Exactly-Once Delivery
• processing occurs only once for each message
• all middleware does it*
• EIP: Guaranteed Delivery just means persistence
• JMS: configure session for AUTO_ACKNOWLEDGE
• Google’s MillWheel
25
Exactly-Once Delivery with Akka
26
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
Exactly-Once Delivery with Akka
27
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
Exactly-Once Delivery with Akka
28
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
Exactly-Once Delivery with Akka
29
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
Exactly-Once Delivery with Akka
30
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
persist(rc) { rc =>
sender() ! rc
doTheAction()
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
Exactly-Once Delivery with Akka
31
class PersistThenAct extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
doTheAction()
persist(rc) { rc =>
sender() ! rc
nextId += 1
}
} else if (id < nextId) sender() ! rc
// otherwise an older command was lost, so wait for the resend
}
def receiveRecover = {
case ReceiptConfirmed(id, _) => nextId = id + 1
}
private def doTheAction(): Unit = () // actually do the action
}
The Hard Truth
CPU can stop between any two instructions, therefore
exactly-once semantics cannot be Guaranteed*.
*with a capital G, i.e. with 100% absolute coverage
32
Mitigating that Hard Truth
• probability of loss or duplication can be small
• many systems are fine with almost-exactly-once
• still need to decide on which side to err
• persist-then-act ➟ almost-exactly-but-at-most-once
• act-then-persist ➟ almost-exactly-but-at-least-once
33
We Can Do Better!
• an operation is idempotent if multiple application
produces the same result as single application
• (side-)effects must also be idempotent
• unique transaction identifiers etc.
• this achieves effectively-exactly-once
• can save persistence round-trip latency by
optimistic execution
34
Effectively-Exactly-Once with Akka
35
class Idempotent(other: ActorRef) extends PersistentActor {
var nextId = 1
def receiveCommand = {
case FollowUpCommand(x, id, deliveryId) =>
val rc = ReceiptConfirmed(id, deliveryId)
if (id == nextId) {
// optimistically execute
if (isValid(x)) {
updateState(x)
other ! Command(x) // assuming idempotent receiver
} else sender() ! Invalid(x)
nextId += 1
// then take care of reliable delivery handling
persistAsync(rc) { rc =>
sender() ! rc
}
} else if (id < nextId) sender() ! rc
}
...
}
The Future is Bright Again!
36
source: http://freewallpapersbackgrounds.com/
To Recapitulate:
37
Problem Choices
Transactional Storage
monolithic or distributed/partitioned,
serializable or eventually consistent
Cluster Management
strong consensus or convergent gossip,
consistency or availability
Message Delivery
at-most-once, at-least-once,
almost-exactly-once, effectively-exactly-once
… …
Conclusion: Consistency à la Carte
• not all parts of a system have the same needs
• expend the effort only where necessary
• idempotency does not come for free
• confirmation handling cannot always be abstracted away
• incur the runtime overhead only where needed
• persistence round-trips take time
• consensus protocols take more time
• include required semantics in system design
38
Remember that the trade-off is always between
consistency and availability.
And latency.
©Typesafe 2014 – All Rights Reserved

Mais conteúdo relacionado

Último

Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.Sharon Liu
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeNeo4j
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 

Último (20)

Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
20240319 Car Simulator Plan.pptx . Plan for a JavaScript Car Driving Simulator.
 
IA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG timeIA Generativa y Grafos de Neo4j: RAG time
IA Generativa y Grafos de Neo4j: RAG time
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 

Distributed in Space and Time

  • 1. Dr. Roland Kuhn Akka Tech Lead @rolandkuhn Distributed in Space and Time
  • 2. The Future is Bright 2 source: http://nature.desktopnexus.com/wallpaper/58502/
  • 3. The Future is Bright • Computing Resources: Effectively Unbounded • 18-core CPU packages, 1024 around the corner • on-demand provisioning in the cloud • Demand for Web-Scale Apps: Ravenous • everyone has the tools • successful ideas are bought for Giga-$ 3
  • 4. How do I make a Large-Scale App? • figure out a service to provide to everyone • this will require resources accordingly: • storage • computation • communication • scalability implies no shared contention points • need to be elastic ➟ use the cloud 4
  • 5. How do I make an Always-Available App? • must not have single point of failure • all functions and resources are replicated • protect against data loss • protect against service downtime • regional content delivery • need to distribute ➟ use the cloud 5
  • 6. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes 6
  • 7. How do I make a Persistent App? 7
  • 8. How do I make a Persistent App? 8
  • 9. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes • temporally distributed change records • spatially distributed change log • granularity can be one Aggregate Root (Event Sourcing) • extract business intelligence from history • scale & distribute persistence ➟ in the cloud 9
  • 10. As Noted Earlier: The Future is Bright 10 source: http://nature.desktopnexus.com/wallpaper/58502/
  • 11. … or is it? 11 Photograph: Patrick Cromerford, 2011
  • 12. Distribution: a Double-Edged Sword • did that message make it over the network? • is that other machine running? • are our data centers in sync? • did we lose messages in that hardware crash? 12
  • 13. The Essence of the Problem A distributed system is one whose parts can fail independently of each other. 13
  • 16. Transactional Storage • Serializability simplifies reasoning about behavior • … but is at odds with distribution (CAP theorem) • Eventual Consistency is often sufficient
 (PeterBailis:ProbabilisticallyBoundedStaleness) • Scalability can be regained by partitioning
 (PatHelland:LifebeyondDistributedTransactions) • consistency has a cost • you need to choose how much is necessary 16
  • 17. PAXOS and Friends • problem: distributed consensus • hard solutions favor Consistency • 2PC, PAXOS, RAFT • if a node is unreachable then consensus cannot be reached • soft solutions favor Availability • quorum, allow nodes to drop out • partitions can lead to more than one active set • consistency always has a cost • you need to choose which one is appropriate 17
  • 18. Fighting Message Loss • resending is the only cure • sender stores the message and retries • recipient must send acknowledgement eventually • fighting unreliable medium is sender’s obligation • sender’s buffer must be persistent • resending must begin anew after a crash 18
  • 19. At-Least-Once Delivery with Akka 19 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  • 20. At-Least-Once Delivery with Akka 20 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  • 21. At-Least-Once Delivery with Akka 21 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  • 22. At-Least-Once Delivery with Akka 22 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  • 23. At-Least-Once Delivery with Akka 23 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  • 24. Mysterious Double-Bookings • acknowledgements can be lost • sender will faithfully duplicate the message • therefore named at-least-once • recipient is responsible for dealing with this 24
  • 25. Exactly-Once Delivery • processing occurs only once for each message • all middleware does it* • EIP: Guaranteed Delivery just means persistence • JMS: configure session for AUTO_ACKNOWLEDGE • Google’s MillWheel 25
  • 26. Exactly-Once Delivery with Akka 26 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 27. Exactly-Once Delivery with Akka 27 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 28. Exactly-Once Delivery with Akka 28 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 29. Exactly-Once Delivery with Akka 29 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 30. Exactly-Once Delivery with Akka 30 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 31. Exactly-Once Delivery with Akka 31 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { doTheAction() persist(rc) { rc => sender() ! rc nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  • 32. The Hard Truth CPU can stop between any two instructions, therefore exactly-once semantics cannot be Guaranteed*. *with a capital G, i.e. with 100% absolute coverage 32
  • 33. Mitigating that Hard Truth • probability of loss or duplication can be small • many systems are fine with almost-exactly-once • still need to decide on which side to err • persist-then-act ➟ almost-exactly-but-at-most-once • act-then-persist ➟ almost-exactly-but-at-least-once 33
  • 34. We Can Do Better! • an operation is idempotent if multiple application produces the same result as single application • (side-)effects must also be idempotent • unique transaction identifiers etc. • this achieves effectively-exactly-once • can save persistence round-trip latency by optimistic execution 34
  • 35. Effectively-Exactly-Once with Akka 35 class Idempotent(other: ActorRef) extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { // optimistically execute if (isValid(x)) { updateState(x) other ! Command(x) // assuming idempotent receiver } else sender() ! Invalid(x) nextId += 1 // then take care of reliable delivery handling persistAsync(rc) { rc => sender() ! rc } } else if (id < nextId) sender() ! rc } ... }
  • 36. The Future is Bright Again! 36 source: http://freewallpapersbackgrounds.com/
  • 37. To Recapitulate: 37 Problem Choices Transactional Storage monolithic or distributed/partitioned, serializable or eventually consistent Cluster Management strong consensus or convergent gossip, consistency or availability Message Delivery at-most-once, at-least-once, almost-exactly-once, effectively-exactly-once … …
  • 38. Conclusion: Consistency à la Carte • not all parts of a system have the same needs • expend the effort only where necessary • idempotency does not come for free • confirmation handling cannot always be abstracted away • incur the runtime overhead only where needed • persistence round-trips take time • consensus protocols take more time • include required semantics in system design 38
  • 39. Remember that the trade-off is always between consistency and availability. And latency.
  • 40. ©Typesafe 2014 – All Rights Reserved