Massive Message Processing with Amazon SQS and Amazon DynamoDB (ARC301) | AWS re:Invent 2013
•
25 likes•25,438 views
This document provides an overview of a presentation on using Amazon SQS and DynamoDB to process massive amounts of messages. The presentation discusses using SQS to reliably queue messages at scale, processing the messages in parallel using auto-scaled EC2 instances, and storing results in DynamoDB. It also describes an example application built by the presenter to handle promotional votes for a Super Bowl winner by scaling up to process millions of SMS votes within a tight 10 minute window.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Massive Message Processing with Amazon SQS and Amazon DynamoDB (ARC301) | AWS re:Invent 2013
Similar to Massive Message Processing with Amazon SQS and Amazon DynamoDB (ARC301) | AWS re:Invent 2013 (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Recently uploaded
Recently uploaded (20)
Massive Message Processing with Amazon SQS and Amazon DynamoDB (ARC301) | AWS re:Invent 2013
- 1. ARC301 - Controlling the Flood: Massive Message Processing with Amazon SQS and Amazon DynamoDB Ari Dias Neto, Ecosystem Solution Architect November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
- 2. Who am I? • The Mailman from Brazil – Delivering messages around the world!
- 4. How many Mailmen? When? How long?
- 5. Who am I? Ari Dias Neto – Ecosystem Solutions Architect What are we going to do? We are going to design and build an application to handle any volume of messages! Right now!
- 6. Scenario – Super Bowl Promotion: who is going to win?
- 7. Promotion • Subscription based on SMS – Cellphone number is the key Requirements • • • • • We cannot lose any message We need to process all the valid messages Log all the invalid messages and errors Beautiful dashboard at the end We must process all the messages during the event!
- 8. Who is going to be in the front-line? Fast! Scalable! Reliable! Simple! Fully managed!
- 9. Amazon Simple Queue Service Fully Managed Queue Service SQS
- 10. Any volume of data At any level of throughput
- 11. We cannot lose any message
- 12. No up-front or fixed expenses
- 13. Architecture – Starting with SQS We have received all the messages SQS Now we need to process all of it
- 14. Architecture – Amazon EC2 Instances SQS BUT! how many Instances?
- 15. Architecture – Multithread application EC2 Instances Reduce the costs and increase performance
- 17. But how many instances do we need? 01 instance EC2 m1.xlarge 10 instances 10 instances 100k msgs/minute 1M msgs/minute 5M messages 5 minutes
- 18. Architecture Auto Scaling based on the number of msgs in the queue SQS Auto Scaling Group
- 19. Architecture Where should we save all the messages? High Throughput Needed SQS Auto Scaling Group
- 20. Amazon DynamoDB
- 23. The Dashboard
- 24. Final Architecture Workers Web Dashboard DynamoDB AWS Elastic Beanstalk Container SQS Auto Scaling Group
- 25. Benefits • Ready for any level of throughput • SQS • Ready for any required SLA • Auto Scaling and EC2 • Low Cost • Fully managed queue service • Infrastructure is based on the required SLA • Infrastructure needed for an small period of time
- 26. The challenge! Process all the messages from the queue in 10 minutes!
- 27. Let’s go deep! Let's code!
- 28. Each thread Connect to SQS queue Queue Set “read” in the queue Save as valid or invalid DynamoDB Read up to 10 msgs Queue Validate each message
- 29. Each thread Connect to SQS queue Queue Set “read” in the queue Save as valid or invalid DynamoDB Read up to 10 msgs Queue Validate each message
- 30. Steps to deploy it on AWS ✔Create the queue. Queue name: votes ✔Upload application to S3: s3-sa-east-1.amazonaws.com/arineto/processor.jar ✔Create AMI with JRE. Image ID: ami-05355a6c Create bootstrap script: userdata.txt Create launch configuration Create Auto Scaling Group Create alarms Launch it!
- 32. The Company • BigData Corp. was founded to help companies solve the challenges associated with big data, from collection to processing to information and knowledge extraction.
- 33. The Challenge • “How many e-commerce websites exist in your continent? Can we monitor them on a consistent basis?” – Build a crawling process that can answer this question in a cost effective and speedy manner.
- 34. Architecture • Spot Instances + SQS + S3 = Magic – Spot Instances allow us to optimize processing costs – Amazon SQS allows us to orchestrate the process in a distributed and asynchronous manner – Amazon Simple Storage Service (S3) facilitates the storage of intermediate and final processing results
- 35. Architecture Maestro (reserved instance) List of crawl URLs Main Workers Secondary Workers (queue listeners) Spot Instances Spot Instances Command and Control Queue Execute crawling and process data Secondary work queues – processed data Reprocess data, query additional services, store data on MongoDB MongoDB cluster
- 36. Architecture (3) • Message Volumes – Processing starts by uploading 10MM+ messages – Each processed message may generate up to 10 new intermediate messages – Peak processing of 70K messages / second • Command & Control Queue – This queue enables us to adjust processing as we go and request status checks from instances
- 37. Results (1) $900,000.00 $800,000.00 $700,000.00 $600,000.00 $500,000.00 $400,000.00 $300,000.00 $200,000.00 $100,000.00 $- 0 1 2 3 4 Estimated cost without AWS 5 6 7 Cost with AWS 8 9 10 11 12
- 38. Results(2) 2+ PB of data processed 40+ Bi web pages visited and parsed 500+ services and technologies mapped A complete new view of the web market
- 39. Please give us your feedback on this presentation ARC301 As a thank you, we will select prize winners daily for completed surveys!