Amazon Web Services provides multiple messaging options that you can use to create scalable, distributed systems, implement event sourcing to unlock hidden context and utilise CQRS for efficient data access. In this session we will look at various messaging patterns and discuss techniques and use cases for Amazon SQS, Amazon SNS, Amazon Kinesis, Amazon DynamoDB and Amazon Web Services IoT in your application. Speaker: Stephen Liedig, Solutions Architect, Amazon Web Services

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stephen Liedig – Solutions Architect
September 2016
Messaging in the AWS Cloud
Choices for Distributed Systems

2. “Loosely coupled systems”
The looser they are coupled,
the bigger they will scale,
the more fault tolerant they will be,
the less dependencies they will have,
the faster you will innovate.

3. AWS Messaging Building Blocks
Amazon SQS Amazon SNS
Amazon Kinesis AWS IoT
Amazon
EC2
AWS
Lambda

4. Amazon SQS
• Message queue service that store
messages waiting to be processed
(60 sec -­14 days)
• Separate parts of an application
• Perform tasks asynchronously
• Batch and burst processing

5. Amazon SQS
Customer Order
Queue
Order
message
Amazon
CloudWatch
Scale independently
• Queue length
• ApproximateAgeOfOldestMessage (New!)
Priority Order
Queue
Client
Extend your business requirements
without impacting your application logic
Dead letter
queue
Priority Order
message
Amazon
RDS
Amazon
EC2

6. Amazon SQS
DEMO

7. Amazon SNS
• Reliable storage of messages for immediate
or delayed processing
• Publish / Subscribe – Direct, Broadcast,
Targeted “push” messaging
• Multiple subscribers protocols – SQS,
HTTP/S, Email, SMS, Mobile Push, Lambda
{
"default": "Message body text here.",
"email": "Message body text here.",
"sms": "Message body text here."
}

8. Amazon SNS
Customer Order
Queue
Order
message
Priority Order
Queue
Client
Priority Order
message
Amazon
RDS
Amazon
EC2
Violating Design Principles!

9. Amazon SNS
Client
Amazon
Redshift
AWS
Lambda
Amazon
SNS
Amazon
EC2
Order
notification
Developers
Order
Queue
Developer
Queue
Fan-­out Pattern
Marketing

10. • Focused on high volume data
streaming
• Multiple consumers
• Re-­playable (24hrs up to 7days)
• FIFO (per shard)
Amazon Kinesis Streams

11. Serverless Sentiment Analysis
AWS
Lambda
AWS
Lambda
Kinesis
StreamShard / Partition
Multiple Adapters
Amazon Elasticsearch
Service
Social Media
Sources
Producers Consumers
Amazon
CloudWatch
5min
#food
Amazon
EC2

12. Amazon SNS
DEMO

13. • Ephemeral publish / subscribe
• Supports non-­IoT applications
• MQTT over Websockets
AWS IoT Message Boker
wss://<endpoint>.iot.<region>.amazonaws.com/mqtt

14. • Need to provision topic
• Only publishes to subscribed
endpoints
• Multiple Protocols / Mobile
• 64-­256 KB
Pub / Sub Messaging Options Compared
• Ephemeral topics, no
provisioning necessary
• Wildcard subscription
• MQTT over Websockets
• NodeJs, iOS, Android SDK’s
for non-­IoT use cases
• 128 KB

15. Pub / Sub Messaging Options Compared
Wildcard Description
# Matching the current tree and all subtrees.
Subscribe: Sensor/#
Publish: Sensor/
Sensor/temp
Sensor/temp/room1
+ Matches exactly one item in the topic hierarchy.
Subscribe: Sensor/+/room1
Publish: Sensor/temp/room1
Sensor/moisture/room1

16. • Need to provision topic
• Only publishes to subscribed
endpoints
• Multiple Protocols / Mobile
• 64-­256 KB
Pub / Sub Messaging Options Compared
• Ephemeral topics, no
provisioning necessary
• Wildcard subscription
• MQTT over Websockets
• NodeJS, iOS, Android SDK’s
for non-­IoT use cases
• 128 KB

17. CQRS
Command Query Responsibility Segregation

18. CQRS
• Based on CQS (Command Query Separation) pattern
[Myer]
• CQRS [Young, Dahan] extends CQS pattern by providing
physical separation of Command and Queries.
• CQRS is a pattern where you use one model to update
information and a separate model to read information

19. CQRS
• Benefits
• Distribute and scale reads and writes
• Take advantage of eventual consistency
• Flexibility in read and write models (polyglot persistence)
• Not a general purpose architecture, will add complexity!
• Specific bounded context
• Designed for multi-­user collaborative applications.

20. CQRS
Amazon
RDS
(3NF DB)
Amazon
DynamoDB
(1NF View Model)
Client
Commands Queries
Domain Model
Command Handlers
ORM < Event >
Separate read and write models
(Eventually Consistent)
View models:
• one table = one page view
• Select * from table
• Does not require layered
architecture and multiple data
transformations.
Commands:
• Asynchronous
• Designed not to fail
De-­normaliser
AWS
Lambda
Amazon
SQS
Thin Read Layer
< DTO >

21. Event Sourcing

22. Event Sourcing
• Rebuild domain state by re-­playing event sequence.
• Allows for projections and what-­if scenarios.
• CQRS pattern lays the foundations for an event sourced application.
• Capture all changes to an application domain state as a sequence of
events.
• As events come in, they are appended to an immutable, append-­only
log and can trigger automation workflows, much like a accounting
ledger.

23. Amazon
RDS
(3NF DB)
CQRS + Event Sourcing
Amazon
DynamoDB
(1NF)
Client
Commands Queries
Domain Model
Command Handlers
ORM
< Event >
< DTO >
Immutable,
append-­only event
log / journal
• Files / Aggregate
• RDS
• DynamoDB
• Event Store
De-­normaliser
Amazon
SQS
Amazon
SNS
AWS
Lambda
Thin Read Layer
Amazon
DynamoDB
(Event Log)
AWS
Lambda

24. Summary
Use messaging to build reliable, scalable, distributed
applications
• Amazon SQS – simple, scalable, decoupling of components
• Amazon SNS – reliable publish subscribe scenarios
• Amazon Kinesis – scalable stream data processing
• AWS IoT – ephemeral event bus and message broker

25. Thank You!