There is a lot of interest these days in migrating data from commercial relational databases to open-source relational databases. PostgreSQL is a great choice for migration, offering advanced features, high performance, rock-solid data integrity, and a flexible open-source license. PostgreSQL is compliant with ANSI SQL. It supports drivers for nearly all development languages, and it has a strong community of active committers and companies to provide support. In this talk, we demonstrate an overall approach for migrating an application from your current Oracle database to an Amazon Aurora PostgreSQL database.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Accelerate Oracle to Aurora
PostgreSQL Migration
Jatin Singh
Solutions Architect
AWS/AWS Partner Program
G P S T E C 3 1 3
Ashok Sundaram
Solutions Architect
AWS/AWS Partner Program

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What this session is all about:
Planning heterogeneous database migrations
Using DMS & SCT for Oracle to Aurora PostgreSQL
Lessons from the field & best practices when moving into production

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Existing state of commercial databases
 Very expensive
 Proprietary
 Lock in
 Punitive licensing
 Audits
Open source is an alternative…
How do we get control & predictability without sacrificing speed, scale, or features?

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Speed and availability of high-end commercial databases
 Simplicity and cost-effectiveness of open source databases
 Drop-in compatibility with MySQL and PostgreSQL
 Simple pay as you go pricing
Delivered as a managed service
Amazon Aurora database re-imagined for cloud

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Heterogeneous database migration methodology
SELECT *
FROM GPS_Attendees
WHERE Session_id = ‘GPSTEC313’
AND Clue > 0;
No Records Found

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A typical database migration lifecycle
Data
migration
Application
code
Schema
objects
Cutover
Analysis
Testing

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration assessment
Keep the following assessment separate:
Technical
Processes
Personnel & tools
Create an “inventory” for the database migration, including features:

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
One way to classify the database workload can be:
• Online transaction processing (OLTP):
Several small/fast transactions and queries by indexes that return few rows
• Data warehouse (DW):
Interactive and long-running queries that scan millions of rows and batched updates
Workload qualification framework - Example 01
Category 1 ODBC/JBDC workloads Lightweight applications with no PL/SQL
Category 2 Light, proprietary feature workloads Custom web application with some PL/SQL
Category 3 Heavy, proprietary feature workloads Custom web application with heavy PL/SQL
Category 4 Engine-specific workloads Remedy or other proprietary applications
Category 5 Non-portable, high-risk or lift-and-shift Legacy software
Other way to classify the database workload can be:

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Other way to classify the workload can be:
Workload qualification framework - Example 02

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Estimate the cost using schema complexity only:
Factor Weightage No. of objects Comments
Tables 0.5 n Most Non-Partitioned tables will be auto converted. Ex, like default sysdate
will need to be handled
MViews 4 n All Mviews will be exported as snapshot mviews & are only updated when
fully refreshed
Triggers 2 n Easy conversion
DB Link 1 n Easy conversion using foreign data wrapper
PLSQL
Functions/Procedures
3 n Pl/PgSQL Can be used, for complexity, will also need to evaluate
proprietary dbms_* feature dependencies
PLSQL Packages 12 n Pl/PgSQL Can be used
Partitions 0.1 n Create child partitions + functions & triggers, pg_partman
LOBs 1 to 5 n LOBs will need to be handled differently; consider Max LOB Size
Workload qualification framework - Example 03

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Estimating “level of effort” for DB migration
Complexity Level Simple Medium Difficult Complex Hard blocker
Weightage 1-300 300-700 700-1200 1200+ Hash partitioning
or mandatory
BLOBS
Redo/hr 1-5 GB 5-10 GB 10-20 GB 20+ GB
Simple Medium Difficult Complex
Total 85 55 15 3
Estimated effort per DB 4-6 weeks 6-8 weeks 8-12 weeks 12-14 weeks

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Estimating “level of effort” for DB migration (cont.)

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Master plan

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Convert schema & migrate data
Step 1: Convert or copy your schema
Source DB or DW AWS SCT Destination DB or DW
Step 2: Move your data
Source DB or DW Destination DB or DW
AWS DMS
Copy or convert
Data
schema

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Schema Conversion Tool
• Assessment report
• Project interface
• Code browser
• Automates many conversion tasks
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
IndexesViews
Sequences
Synonyms
Tab with the assessment report
Manual conversion tips
Side by side code view

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Schema Conversion Tool tips
Save as an SQL file
• Allows you to apply only
Table DDL & PK
• Save the secondary DDL for
after migration
Memory management
• Global settings
• JVM settings

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SCT migration assessment report
Connect SCT to “source” &
“target”
Run “assessment report”
Read “executive summary”
& follow detailed
instructions

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SQL scripts
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
IndexesViews
Sequences
Synonyms
SCT does a great job of
converting your schema and
code objects
Users, roles, grants
https://aws.amazon.com/blogs/database/use-sql-to-map-users-roles-and-grants-from-oracle-to-postgresql/

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Oracle to Amazon Aurora PostgreSQL
Migration Playbook – Example

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Customer
On-premises
Application users
AWS
AWS Database Migration Service
 Start a replication instance
 Connect the source and target endpoints
DMS Replication Instance
Oracle Aurora
PostgreSQL
End Point Connection
Full Load
Change Data Capture
 DMS FULL LOAD the data from Source to Target
 DMS Change Data Capture to replicate ongoing changes
 At steady state – take an outage, validate & redirect connection
Amazon
VPC
VPN

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Working with DMS components
AWS Database Migration Service
Replication Instance
C – compute
R – memory
T – burstable
Endpoint
Connect source / target from replication instance
Choose advanced override settings
Task
Full Load
Change Data Capture

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choosing a replication instance
AWS Database Migration Service
How to choose between different instance types.
Things to know about EBS storage on replication instance
Important Amazon CloudWatch metrics to watch on the replication instance

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Designing the source & target endpoint
AWS Database Migration Service
Oracle (source)
When to use ‘binary reader’
When to use ‘log miner’
Using NumberDataTypeScale
Using standbyDelayTime
PostgreSQL (target)
Using ‘maxFileSize’

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Components of a DMS task
AWS Database Migration Service
 Choosing a “migration type”
Full Load
Change Data Capture (CDC)
 Choosing a target table prep mode
- DMS can create tables, however, better pre-create table with SCT
- Flexibility to drop or truncate tables.
 Include LOB columns in replication
- Why limited LOB mode is a better choice?
 Selection & transformation rules, logging, exceptions, and others
- Flexible in selection but limited transformations
- How to work around transformations

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Database Migration Service
Sample task
The following table mapping does these things:
(1) It includes the DMS_SAMPLE schema in the migration.
(2) It excludes the tables
NFL_DATA,
MLB_DATA,
NAME_DATE and
STADIUM_DATA.
(3) It converts the dictionary objects to lower case:
Schema
Tables
Columns

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: DMS migrate existing data
Initially create only tables & primary keys
Use transformation & selection rules
Truncate data or Do nothing

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: DMS replicate ongoing changes
Oracle CDC: Oracle LogMiner or Binary Reader
• Read changes from redo logs/archived redo logs
• Used for crash recovery, but can replay transactions to target
Tell DMS where to start picking up changes
• Custom Start time
• *SCN: SELECT current_scn FROM V$DATABASE
• *DMS Checkpoint

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why key performance indicators (KPIs) are important:
Define your own KPIs for
• Scale
• Uptime
• Backup/Restore
• Maintenance
• Capacity
• Resource usage
• Performance
• Throughput/TPS and others
Key performance indicators for production

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: Scale & performance
Infrastructure planning, performance benchmarking & key parameters
• Infrastructure planning
•Storage
•Compute & memory
Aurora storage daemon uses CPU
• Important PostgreSQL database parameters for scale
• Example: Understanding DB Cache Size
• Example: Consider PostgreSQL Query Planning:
https://www.postgresql.org/docs/9.6/static/runtime-config-query.html

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CloudWatch metrics
• Some CloudWatch metrics for Aurora are different from traditional EBS-backed
RDS PostgreSQL instance
•Commit throughput
•Commit latency
Enhanced monitoring
• Over 50 system metrics and aggregated process information
• Granularity of up to 1 second
Example: Scale & performance (cont.)
Amazon CloudWatch & Enhanced Monitoring

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Do not turn off autovacuum
•Routine vacuuming is required in PostgreSQL -
https://www.postgresql.org/docs/9.6/static/routine-vacuuming.html
•A case study of tuning autovacuum in Amazon Relational Database
Service (Amazon RDS) for PostgreSQL:
https://aws.amazon.com/blogs/database/a-case-study-of-tuning-
autovacuum-in-amazon-rds-for-postgresql/
• Maintenance operations:
•Set appropriate maintenance_work_mem for maintenance operations
•Watch locally attached EBS volume usage & performance
Example: Maintenance
Maintenance best practices

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: Availability
Fast failover best practices
1. Use read & write cluster endpoints to connect
2. Aggressively set TCP keepalives so queries don’t rely on read timeout only
3. Aggressively set Java DNS caching timeouts
4. Set timeout in JDBC connection string lower
TIP - Use separate connection objects for short and long running queries
5. Test
Use RDS APIs to test application response on server side failures and use a packet dropping tool to test
application response for client-side failures
Reference: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/AuroraPostgreSQL.BestPractices.html

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PostgreSQL is a
lowercase data
dictionary
Use “exception handlers”
when needed, not by
default
PostgreSQL has six
different index types.
Store your BLOBs in
Amazon S3 instead of
the database
search_path replaces
PUBLIC SYNONYM
PostgreSQL has 64
datatypes
Understand basic database engine differences

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Enable CloudWatch
Logs
Increase the logging
level for your task
Metrics for replication
instance and tasks
Look for latency in
CDC tasks
Stream buffer
settings
AWS DB Blog
https://aws.amazon.com/blogs/database/debugging-your-aws-
dms-migrations-what-to-do-when-things-go-wrong-part-1/
Best practices on troubleshooting DMS tasks

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DMS RDS PostgreSQL
Oracle RDS
Legacy Oracle DB
1
2
Legacy to Aurora PostgreSQL
Aurora PostgreSQL to RDS Oracle
1
2
AWS DMS
HIGH LEVEL Plan:
 Create Tables with SCT
 Full Load
 Apply Indexes
 Apply cached changes
during Full Load
 Enable FKs
 Start CDC
 Enable triggers once you
stop DMS tasks for the DB
cutover
 Sync Sequences and any
PL/SQL that was modified
 Stop the applications
 Validate source/target
 Cutover
Example: Planning a cutover & rollback

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Use appropriate tools for planning
• Use SCT & DMS for schema design & data migration
• Understand the difference in source & target DB engines
• Define KPIs & benchmark the performance, availability, & other key aspects
• Plan ahead for production / operations.
Summary…

41. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jatin Singh
sinjatin@amazon.com
Ashok Sundaram
sunashok@amazon.com

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.