Mais conteúdo relacionado Semelhante a Accelerate database development and testing with Amazon Aurora - ADB208 - New York AWS Summit (20) Mais de Amazon Web Services (20) Accelerate database development and testing with Amazon Aurora - ADB208 - New York AWS Summit1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Accelerate database development and
testing with Amazon Aurora
Vlad Vlasceanu
Principal Database Specialist SA
AWS
A D B 2 0 8
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database development and testing perspective
Responsibilities of today’s developers, DevOps, and database admins:
• Database architecture
• Deployment & automation
• Improve dev/test outcomes
• Performance and availability focused development
• Cost effective development
Aurora provides native capabilities to meet these goals
3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora: Reimaging the relational database
Scale-out, distributed architecture using purpose-built storage
system
Service-oriented architecture leveraging AWS services
Fully managed service, automating administrative tasks
1
2
3
Cloud-native capabilities simplify use4
Aurora is fast…
5x more throughput than MySQL; 3x more throughput than PostgreSQL
5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Aurora – key characteristics
1 Log-structured storage volume shared
between all cluster nodes
2
Optimized for high throughput3
Readers strictly read-only
6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Core use cases: Read scaling architecture
Automatic scaling optional
All readers treated the same
Any reader can be a failover
target
Application should use cluster
and reader endpoints
Dev. tip: Monitor cluster
topology for faster failover
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Core use cases: Separated workloads architecture
Each reader used for a specific
purpose
Ordered failover targeting
Application uses cluster, custom
or even database instance
endpoints
Caveat: Readers share the same
storage and undo log
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automating deployments for Aurora
Why?
• Reduce the risk of human errors
• Repeatable deployments
• Accountability and management of change
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Automating deployments for Aurora
Why?
• Reduce the risk of human errors
• Repeatable deployments
• Accountability and management of change
How?
+
10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Deployment automation using AWS CloudFormation
AWS CloudFormation:
Outcome-driven deployment automation, templates define desired state
… but how do you define the state of an Aurora database cluster?
Supplement capabilities to match desired processes:
Check capability gaps in AWS CloudFormation versus API1
Use AWS CloudFormation custom resources or a master snapshot to bootstrap
database (initial users, permissions, schemas)
2
Create alternative automation for workflows that need to be completed with
minimal disruption
3
11. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Risk free dev/test, data extracts, and analytics
Bad practices we’ve seen
• App development and testing on tiny subsets of non-production data
• Bulk data exports from live production instances
• Running long, unoptimized queries on cluster readers
• Lack of outcome focus
The right way
• Know your features: Point-in-Time Restore, Database Cloning, Backtrack
• Proper workload isolation
• Test; don’t hope for the best
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database cloning
Get faster access to copies of your data so you can:
Test changes in pre-production on relevant
datasets
Reorganize a database with minimal
production impact
Save a point-in-time snapshot for data
analysis without impacting production
systems
1
2
3
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Fast database cloning with Aurora
Available for both MySQL and PostgreSQL compatible versions
of Aurora
Creation of clone takes a few minutes
Data copy happens only on write, when the original and
cloned volume start to differ
Operations on clone do not affect the source cluster
Up to 15 clones from the same source
Pay only for the data storage difference on the clone PRODUCTION DATABASE
CLONE CLONE
CLONE
DEV/TEST
APPLICATIONS
BENCHMARKS
PRODUCTION
APPLICATIONS
PRODUCTION
APPLICATIONS
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Backtracking databases
Easily undo changes to your data:
Reduce the risk of database changes at
scale
Undo unintentional DML and DDL
changes
Mitigate risk of malicious changes to your
data
1
2
3
Avoid time consuming data restore from
backups
4
16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does Backtrack work?
• We keep periodic snapshots of each segment; we also preserve the logs
• For Backtrack, we identify the appropriate segment snapshots
• Apply log streams to segment snapshots in parallel and asynchronously
SEGMENT SNAPSHOT
CHANGE RECORDS
RECOVERY POINT
SEGMENT 1
SEGMENT 2
SEGMENT 3
TIME
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Going back in time with Backtrack
t0 t1 t2
t0 t1
t2
t3 t4
t3
t4
Rewind to t1
Rewind to t3
Invisible Invisible
• Backtrack is not destructive
• You can backtrack multiple times to find the right point in time
• Pay for the volume of change records retained for the desired duration (up to 72 hours)
• Available for Aurora MySQL 5.6 compatible
18. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Development focused on production scale
Bad practices we’ve seen:
• App development and testing on tiny subsets of non-production data
• My database is always there for me and never fails me
• Running long, unoptimized queries on cluster readers
• Lack of outcome focus
The right way:
• Be aware of architectural differences between Aurora and equivalent engines
• Know Aurora capabilities and operational procedures: automated failovers, upgrades
• Extend Aurora features with application side cluster awareness
• Test, don’t hope for the best: failure injection
20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Connection management and availability
Minimize failure downtime, and connection management overhead
Common MySQL and PostgreSQL drivers are not aware
of the cluster or topology
Use a client-side connection pool; recycle connections
periodically; avoid connection storms
Use a smart driver, or build topology awareness in your
client data layer, validate connections
1
2
3
Understand the behavior of the Aurora DNS endpoints,
manage DNS caching
4
21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Monitor the performance of your queries
Establish a baseline of acceptable or
preferred query performance
Assess performance impact of workload
changes
Troubleshoot poor performance, identify
bottlenecks
1
2
3
Effective capacity planning4
22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database health at a glance
Aurora comes with comprehensive monitoring built-in:
23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Database health at a glance
Amazon Aurora comes with comprehensive monitoring built-in:
24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Performance Insights
Available for both MySQL and PostgreSQL compatible versions
of Aurora
Easy and powerful dashboard showing load on your database
Helps you identify source of bottlenecks: top SQL queries, wait
statistics
Adjustable time frame (hour, day week, month)
Seven days of performance data history free. Perfect for
developers; up to two years of long-term retention for
production use cases
Max CPU
25. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Cost optimized development and testing
Use database resources at appropriate scale only when they are needed
1 Development and test activities are rarely continuous
2
Development and test environment isolation3
Scale of development and test activities vary
27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora features helping with cost optimization
28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora features helping with cost optimization
29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora Serverless use cases
Development and test databases:
• Easily provisioned
• Cost savings when databases are not in use
• Simplify development and test pipelines
Infrequently used applications
(e.g., low-volume blog site)
Applications with variable load; peaks of activity that are
hard to predict (e.g., news site)
30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora Serverless architecture
• Application talks to MySQL-compatible endpoint
• Fleet of routers manage queue, client connections, and
route DB traffic
• Instance handles database operations
• Data kept durable and highly available on Aurora storage
volume
• When scaling thresholds are reached, we scale the
instance substituting with capacity from warm pool
• Scaling operations are transparent to applications
• You configure min. and max. capacity, whether and when
to pause database if there’s no activity
INSTANCE
DATABASE
STORAGE
APPLICATION
WARM POOL
DATABASE TRAFFIC ROUTERS
31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Aurora Serverless pricing
• You pay for the capacity your database consumes while it
is active
• Database capacity is measured in Aurora Capacity Units
(ACUs): 1 ACU ~ 2 Gb of memory with corresponding CPU
and network capacity
• Flat rate per second of ACU usage, with a minimum of 5
minutes of usage each time the database is activated
• Storage and I/O are billed the same as Aurora instances
Duration ACUs ACU-Hours Rate Charges
1 hour, 10 minutes, and 15
seconds (1.020 hours)
4 4.08 $0.06 $0.25
40 minutes and 34 seconds
(0.676 hours)
8 5.41 $0.06 $0.32
Usage total for 24 hours 9.49 $0.06 $0.57
Usage Rate Charges
9.49 ACU-hours $0.06 per ACU-hour $0.57
80 GB of storage for 24 hours $0.10 per GB-month $0.26
90,000 I/O requests $0.20 per 1 million requests $0.02
Total charges for 24 hours $0.85
32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Introducing the RDS Data API
• Public HTTP API endpoint, integrated with AWS authentication
• Automated server-side connection pooling
• Ideal for AWS Lambda or serverless apps reusing connections
across multiple invocations
• Fully integrated with AWS Secrets Manager and
AWS AppSync
• Optional session construct for efficient prepared statement
executions
• Interactive query editor in the console for
ad-hoc queries
• Available for Aurora Serverless (MySQL)
33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Parting thoughts
Use automation wisely and map out your
operational processes in detail
Know Aurora’s features and how they can
improve on your development and testing
process
Develop and test with performance in
mind
1
2
3
Quantify the expected baseline
performance of your workload
4
34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vlad Vlasceanu
vladv@amazon.com