Technologies for the capture and analysis of streaming data has changed over the years and cloud technologies have taken us to a new level. Many people are not aware of the new technologies and architectural paradigms that are available today for near-real-time capture and analysis of high-volume data. This presentation will examine Amazon Web Services’ offerings for streaming data analysis, compare how it’s changed over the years, and take a look at what might be coming in the future. Real-life case-studies and architectures will be shared to demonstrate how these technologies can, and have been, used to successfully meet customer needs.

1. How Amazon Web Services is Changing Our Approach
Presented by: Michael Krouze, CTO & VP Analytics, Charter Solutions, Inc.
High Volume Streaming Data

2. 2

3. Act I
1990
3

4. 4
Improve the quality of the manufacturing process
Build something usable with minimal education
Distributed system
High-visualization
Intuitive

5. 5
Data Collection
Server
Archive Server
Visualization
Station
Sensors
Data Collection
Server
Process
Control
Data Collection
Server
Manual
Entry
Visualization
Station
Visualization
Station

6. Client 1
6
Visualization
Software
GDBM Client
Client 2
Visualization
Software
GDBM Client
Source
Integration
Code
GDBM Client
Archive
Server

7. Observations
7
u Met the needs of the organization
u Was a cool, cutting edge project
u Took 5 internal staff and 3 external consultants years to implement
u Had measurable positive impact on product quality
u Very custom
u Lots of manual labor in scaling up
u Costly

8. Act II
2011
8

9. 9
Support a new set of services around HVAC management
Improve customer satisfaction
Build better products
High availability
Linear scalability
Fast

10. 10
Amazon
EC2
Amazon
SQS
Amazon
S3
Instance
Queue
Bucket
DB on
Instance
Amazon
VPC
Internet
Gateway
Router
Elastic Load
Balancing

11. 11
Bucket
Queue
Vertica
Database
3..n
Instances
Catching
Mitt
Custom
Java App
2..n
Instances
Data Prep
Custom
Java App
2..n
Instances
Viz Server
SAS Server
Real Time
Analytics
Custom
Java/R App
AWS cloud
Building
Device
user
user
Corporate Data
Center

12. Observations
12
u 2 part-time resources took 2 months to implement initial version
u Increased performance by over 60x
u Cost improvements of over 50%
u Productivity increase of over 40%
u Faster time from data collection to visualization
u Scales linearly and quickly
u Highly failure resilient
u Must maintain capacity for bursts of data
u Delay of up to 5 minutes to scale

13. Act III
2014
13

14. 14
Provide insight into how users engage with the application
Provide insight into content usage for ordering and recommendation models
Low cost of entry
Fast
Can scale quickly, easily, and in small increments
Can handle “bursty” traffic

15. 15
Amazon
Kinesis
AWS
Lambda
Amazon
S3
Amazon
Redshift
Amazon
CloudWatch
Amazon
Kinesis Streams
Bucket AlarmDense
Storage
Node
AWS KMS Amazon
VPC
Internet
Gateway
Router

16. 16
Core
Application
Event
Stream
Data
Processing
Data Store
Long-Term
Storage
Analytics ClientMobile
Browser
AWS cloud

17. Observations
17
u 1 person implemented in 2-3 weeks
u Very low cost for low data rates
u Scales in very small increments
u Scales fast
u Handles small and large bursts of data easily
u Minimal held capacity – primarily pay when processing data

18. 18

19. A Note on Scale
19
Rate per Second Rate per Month Rate per Year
1 KB 2.6 GB 31 GB

20. A Note on Scale
20
Rate per Second Rate per Month Rate per Year
1 KB 2.6 GB 31 GB
10 KB 26 GB 310 GB
100 KB 260 GB 3.1 TB
1 MB 2.6 TB 31 TB
10 MB 26 TB 310 TB
100 MB 260 TB 3.1 PB

21. 21

22. 22

23. 23

24. 24

25. 25

26. Act IV
Today and Beyond
26

27. 27

28. 28
AWS Import/
Export Snowball
Amazon
DynamoDB
Amazon
Kinesis
Amazon Machine
Learning
Amazon
QuickSight
AWS IoTAmazon Elasticsearch
Service
import/
export
Amazon
Kinesis
Firehose
Amazon
Kinesis
Analytics
IoT HTTP
protocol
IoT
certificate
IoT
rule

29. 29

30. Final Words
30
u IoT will drive the future of streaming data analytics
u While AWS has a strong lead, there are competitors both in the general cloud services arena
and individual niches
u Batched data for analysis will decrease over time – giving preference to streaming data
u Even for transactional systems

31. Thank you!

32. u Contact me:
u michael.krouze@chartersolutions.com
u @mjkrouze
u Resources:
u www.chartersolutions.com
u github.com/awslabs/amazon-redshift-utils
u AWS YouTube channel
u AWS on SlideShare