How to choose the right messaging service
•
0 gostou•135 visualizações
At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a How to choose the right messaging service
Semelhante a How to choose the right messaging service (20)
Mais de Yan Cui
Mais de Yan Cui (20)
Último
Último (20)
How to choose the right messaging service
- 1. How to choose the right messaging service for your workload
- 6. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
- 8. AWS services are the new data structures. Understanding their trade-offs is the new big O notation.
- 13. EventBridge
- 17. event
- 19. event FIFO Archiving Replays Schema Discovery Scalability Cost Performance Debugging Error Handling
- 21. picking the right AWS service is a valuable skill
- 23. Scaling
- 25. What are the scaling constraint for the messaging service?
- 26. Google “{service name} quotas”
- 30. How does Lambda’s concurrency scale with throughput?
- 38. more concurrency is not always better…
- 41. msg/s time max throughput erred and retried SNS EventBridge
- 42. msg/s time max throughput erred and retried SNS EventBridge
- 44. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 45. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 46. Downstream System SNS Lambda use Reserved Concurrency to limit concurrency
- 47. Reserved Concurrency are taken out of the available regional Lambda concurrency
- 48. managing Reserved Concurrency for many functions is dif fi cult and error prone, easy to create more problems than it solves
- 49. Costs
- 50. always factor scale into the equation
- 51. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296
- 52. Kinesis on-demand mode pricing
- 53. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis Provisioned 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296 Kinesis On-Demand 1kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $28.998
- 54. $47.088 1,000 msg/s for a month, 1KB per msg 1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2592.00 SNS SQS EventBridge Kinesis Provisioned 1000 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1036.80 $1296.00 Kinesis On-Demand 1000kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $226.55
- 55. services that charge by uptime are order(s) of magnitude cheaper when running at scale
- 56. National broadcaster of Finland
- 57. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min
- 58. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 59. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova decisions driven by cost ef fi ciency
- 60. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 61. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova fallback in case Kinesis is down
- 62. Error Handling
- 64. 0, 1, 2
- 65. DLQs only capture the failed event
- 66. DLQ
- 73. prefer Lambda destination over DLQs (you can use both together, but there’s no clear reason for doing so)
- 77. DLQ
- 78. Observability
- 79. A measure of how well the internal state of an application can be inferred from its external outputs
- 80. X-Ray
- 83. X-Ray doesn’t trace through many popular messaging services.
- 84. X-Ray doesn’t trace through many popular messaging services. Need separate solution (e.g. correlation IDs) for Lambda logs.
- 86. all the indirect invocations are accounted for
- 88. all the Lambda logs from the transaction in chronological order
- 91. Step 1. Step 2.
- 92. No manual instrument. Better support for messaging services. Supports containers and Lambda.
- 93. “which messaging service should I use?”
- 95. task vs event
- 96. Task Event “Something happened” “Go do a thing”
- 97. Task Event “Something happened” “Go do a thing” Intended for a target receiver.
- 98. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Often expects an answer.
- 99. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Publishers are obvious to subscribers. Often expects an answer.
- 106. EventBridge SNS FIFO Do Stuff!
- 107. EventBridge SNS FIFO Do Stuff! order is lost
- 108. EventBridge SNS FIFO Do Stuff! order is lost
- 109. EventBridge SNS FIFO SQS FIFO Do Stuff (in order)! Other subscribers (not my problem!)
- 111. Kinesis Lambda
- 112. EventBridge Kinesis Lambda Other subscribers (not my problem!) send domain events
- 117. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead