The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Databases on AWS
The Right Tool for the Right Job

Migration & Streaming Services
Data Movement
Data
Warehousing
Big Data
Processing
Interactive
Query
Operational
Analytics
Real time
Analytics
Serverless
Data processing
Analytics
Infrastructure
Data Catalog
& ETL
Security &
Management
Data Lake Infrastructure & Management Ledger
Blockchain Central Trust
Dashboards Predictive Analytics
Visualization & Machine Learning
Databases
Relational Key-value Document In-memory Graph Time-Series
Broad and deep portfolio, purpose-built for builders

Data Movement
Analytics
Data Lake Infrastructure & Management Ledger
Visualization & Machine Learning
Databases
+ 10 more
QuickSight SageMaker Comprehend Lex Polly Rekognition
The image part
with relationship
ID rId5 was not
found in the file.
Translate Transcribe
Deep Learning
AMIs
Aurora
MySQL,
PostgreSQL
RDS and RDS on VMWARE
MySQL, PostgreSQL,
Oracle, SQL Server
MariaDB
DynamoDB
Key-value
ElastiCache
Redis,
Memcached
Timestream
Time Series
Neptune
Graph
DocumentDB
Documents
Redshift
EMR
Spark,
Hadoop
Athena
Elasticsearch
Service
Kinesis Data
Analytics
Glue
Scala, Python
S3/Glacier Lake
Formation
Glue
Crawl, ETL
Managed Blockchain Quantum Ledger
Database (QLDB)
Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka
Broad and deep portfolio, purpose-built for builders

Common data categories and use cases
Relational
Referential
integrity, ACID
transactions,
schema-
on-write
Lift and shift, ERP,
CRM, finance
Key-value
High
throughput, low-
latency reads
and writes,
endless scale
Real-time bidding,
shopping cart,
social, product
catalog, customer
preferences
Document
Store
documents and
quickly access
querying on
any attribute
Content
management,
personalization,
mobile
In-memory
Query by key
with
microsecond
latency
Leaderboards,
real-time analytics,
caching
Graph
Quickly and
easily create
and navigate
relationships
between
data
Fraud detection,
social networking,
recommendation
engine
Time-series
Collect, store,
and process
data sequenced
by time
IoT applications,
event tracking
Ledger
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record, supply
chain, health care,
registrations,
financial

AWS Database Services
DynamoDB ElastiCache Neptune
Key value | Document Graph
Non-relational
Database Migration Service
Relational
Aurora RDS

Two fundamental areas of focus
“Lift and shift” existing
apps to the cloud
Quickly build new
apps in the cloud

“Lift and shift” existing apps to the cloud
apps to the cloud
Quickly build new
apps in the cloud

Amazon Aurora
MySQL and PostgreSQL-compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the cost
Performance
and scalability
Availability
and durability
Highly secure Fully managed
5x throughput of standard
MySQL and 3x of standard
PostgreSQL; scale-out up to
15 read replicas
Fault-tolerant, self-healing
storage; six copies of data
across three Availability Zones;
continuous backup to Amazon S3
Network isolation,
encryption at
rest/transit
Managed by RDS:
No hardware provisioning,
software patching, setup,
configuration, or backups

Amazon Relational Database Service (RDS)
Managed relational database service with a choice of six popular database engines
Easy to administer Available and durable Highly scalable Fast and secure
No need for infrastructure
provisioning, installing, and
maintaining DB software
Automatic Multi-AZ data
replication; automated backup,
snapshots, failover
Scale database compute
and storage with a few
clicks with no app
downtime
SSD storage and guaranteed
provisioned I/O; data
encryption at rest and in
transit

AWS Database Migration Service (AWS DMS)
M I G R A T I N G
D A T A B A S E S
T O A W S
Migrate between on-premises and AWS
Migrate between databases
Automated schema conversion
Data replication for
zero-downtime migration
100,000+
databases migrated

Quickly build new apps in the cloud
apps to the cloud
Quickly build new
apps in the cloud

Modern apps create new requirements
Users: 1 million+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Web, mobile, IoT, devices
Scale: Up-down, Out-in
Economics: Pay for what you use
Developer access: No assembly requiredSocial mediaRide hailing Media streaming Dating

Let’s take a closer look at…
Relational Key-value Graph

Relational data
• Divide data among tables
• Highly structured
• Relationships established via
keys enforced by the system
• Data accuracy and consistency
Patient
* Patient ID
First Name
Last Name
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
Adverse Outcome
Contraindication
Doctor
* Doctor ID
First Name
Last Name
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating

Relational use case
// Doctors affiliated with Mercy
hospital
Patient
* Patient ID
First Name
Last Name
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
Adverse Outcome
Contraindication
Doctor
* Doctor ID
First Name
Last Name
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating
SELECT
d.first_name, d.last_name
FROM
doctor as d,
hospital as h
WHERE
d.hospital = h.hospital_id
AND h.name = ‘Mercy';
// Number of patient visits each doctor
completed last week
SELECT
d.first_name, d.last_name, count(*)
FROM
visit as v,
hospital as h,
doctor as d
WHERE
v.hospital_id = h.hospital_id
AND h.hospital_id = d.hospital
AND v.t_date > date_trunc('week’,
CURRENT_TIMESTAMP - interval '1 week')
GROUP BY
d.first_name, d.last_name;

Key-value data
• Simple key value
pairs
• Partitioned by keys
• Resilient to failure
• High throughput,
low-latency reads
and writes
• Consistent
performance at scale
Gamers
Primary Key Attributes
GamerTag Level Points High Score Plays
Hammer57 21 4050 483610 1722
FluffyDuffy 5 1123 10863 43
Lol777313 14 3075 380500 1307
Jam22Jam 20 3986 478658 1694
ButterZZ_55 7 1530 12547 66
… … … … …
Gamers
Hammer57
21
4050,
483610,
1722
GET {
TableName:"Gamers",
Key: {
"GamerTag":"Hammer57“,
“ProjectionExpression“:”Points”
} }

Gamers
Primary Key
Attributes
Gamer Tag Type
Hammer57
Rank
Level Points Tier
87 4050 Elite
Status
Health Progress
90 30
Weapon
Class Damage Range
Taser 87% 50
FluffyDuffy
Rank
Level Points Tier
5 1072 Trainee
Status
Health Progress
37 8
Key-value use case
// Status of Hammer57
GET {
TableName:"Gamers",
Key: {
"GamerTag":"Hammer57",
"Type":"Status” } }
// Return all Hammer57
Gamers
GamerTag = :a
:a Hammer57

Amazon DynamoDB
Fast and flexible NoSQL database service for any scale
Performance at
scale
Handles millions of requests per
second
Delivers microsecond latency
Automated global replication
Enterprise
ready
ACID transactions
Encryption at rest
On-demand backup and restore
Serverless
Maintenance free
Auto scaling
On-demand capacity mode

Amazon DynamoDB
Document or Key-Value Scales to Any WorkloadFully Managed NoSQL
Access Control Event Driven ProgrammingFast and Consistent

Graph data
• Relationships are first-class
objects
• Vertices connected by Edges
PURCHASED PURCHASED
FOLLOWS
PURCHASED
KNOWS
PRODUCT
SPORT
FOLLOWS

Graph use case
// Product recommendation to a user
gremlin> V().has(‘name’,’sara’).as(‘customer’).out(‘follows’).in(‘follows’).out(‘purchased’)
( (‘customer’)).dedup() (‘name’) ('name')
KNO
W
S
PURCHASED PURCHASED
FOLLOWS
PURCHASED
KNOWS
PRODUCT
SPORT
FOLLOWS
FOLLOWS
// Identify a friend in common and
make a recommendation
gremlin> g.V().has('name','mary').as(‘start’).
both('knows').both('knows’).
where(neq(‘start’)).
dedup().by('name').properties('name')

AMAZON Neptune
Fully managed graph database
FAST RELIABLEOPEN
Query billions of
relationships with
millisecond latency
6 replicas of your data
across 3 AZs with full
backup and restore
Build powerful
queries easily with
Gremlin and SPARQL
Supports Apache
TinkerPop & W3C
RDF graph models
EASY

Open Source Apache TinkerPop™
Gremlin Traversal Language
W3C Standard
SPARQL Query Language
RESOURCE DESCRIPTION
FRAMEWORK (RDF)
PROPERTY GRAPH
Graph Models and Frameworks

Common customer use cases
Ledgers with centralized control
Healthcare
Verify and track hospital
equipment inventory
Manufacturers
Track distribution of a
recalled product
HR & Payroll
Track changes to an
individual’s profile
Government
Track vehicle title
history

Challenges with building ledgers
Adds unnecessary
complexity
BlockchainRDBMS - audit tables
Difficult to
maintain
Hard to use
and slow
Hard to build
Custom audit functionality using
triggers or stored procedures
Impossible to verify
No way to verify changes made
to data by sys admins

ID Manufacturer Model Year VIN Owner
ID Version Start End Manufacturer Model Year VIN Owner
How it works
1 Tesla Model S 2012 123456789 Traci Russell
INSERT INTO cars <<
{ 'Manufacturer': 'Tesla',
'Model': 'Model S',
'Year': '2012',
'VIN': '123456789',
'Owner': 'Traci Russel' }
>>
FROM cars WHERE VIN = '123456789' UPDATE owner = 'Ronnie Nash'FROM cars WHERE VIN = '123456789' UPDATE owner = 'Elmer Hubbard'
J
1 1 07/16/2012 NULL Tesla Model S 2012 123456789 Traci Russell
current.cars
C
history.cars
H ID Version Start End Manufacturer Model Year VIN Owner
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 NULL Tesla Model S 2012 123456789 Ronnie Nash
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 09/02/2016 Tesla Model S 2012 123456789 Ronnie Nash
1 3 09/02/2016 NULL Tesla Model S 2012 123456789 Elmer Hubbard
1 Tesla Model S 2012 123456789 Ronnie Nash
1 Tesla Model S 2012 123456789 Elmer Hubbard
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
H (x) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (x) UPDATE cars
ID:1
Owner: Elmer Hubbard
Metadata: {
Date: 09/02/2016
}
H (x)

Amazon Quantum Ledger Database (QLDB)
Fully managed ledger database
Track and verify history of all changes made to your application’s data
Immutable
Maintains a sequenced record of
all changes to your data, which
cannot be deleted or modified;
you have the ability to query and
analyze the full history
Cryptographically
verifiable
Uses cryptography to
generate a secure output
file of your data’s history
Easy to use
Easy to use, letting you
use familiar database
capabilities like SQL APIs
for querying the data
Highly scalable
Executes 2–3X as many
transactions than ledgers
in common blockchain
frameworks

Time-series data
What is time-series
data?
What is special about a
time-series database? A sequence of data points
recorded over a time interval
Time is the
single primary axis
of the data model
t

Time-series use case
Application events
IoT Sensor
Readings
DevOps data
Humidity
% WATER VAPOR
91.094.086.093.0

Existing time-series databasesRelational databases
Difficult to
maintain high
availability
Difficult to scale Limited data
lifecycle
management
Inefficient
time-series data
processing
Unnatural for
time-series
data
Rigid schema
inflexible for fast
moving time-series
data
Building with time-series data is challenging

Amazon Timestream
Fast, scalable, fully managed time-series database
1,000x faster and 1/10th the
cost of relational databases
Collect data at the rate of
millions of inserts per
second (10M/second)
Trillions of
daily events
Adaptive query processing
engine maintains steady,
predictable performance
Time-series analytics
Built-in functions for
interpolation, smoothing,
and approximation
Serverless
Automated setup,
configuration, server
provisioning, software patching

Purpose-built databases
Relational Key-value Document In-memory Graph Time-series Ledger
DynamoDB NeptuneAmazon RDS
Aurora CommercialCommunity
Timestream QLDBElastiCache

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Migration
Introduction

Agenda
Challenges
Migration patterns
AWS Database Migration Service
AWS Schema Conversion Tool
Database Migration Overview
Summary
Questions

Database Migration Challenges
Cost
Complexity
Application downtime
Engine specific database code

Value Proposition – What determines value?
From an Organizational Perspective…
• Value enables a positive outcome
• Value IS NOT preventing a negative outcome
• Value is a function of Capability, Risk, and Cost.
Value = f(Capability, Risk, Cost)

Value Proposition – Operations
DevOps NoOpsSustainment
Value

Value Proposition – Hosting
IaaS PaaSOn-Premises
Value

Value Proposition – Migration
Re-Host Re-Platform Re-Factor
Value
Linux à Linux
Oracle à Oracle
Solaris à Linux
Oracle à Aurora
Linux à Serverless
Oracle à DynamoDB

Migration Patterns
Re-Host (Homogenous)
• “Lift-and-Leverage”
• Same database engine / Same operating system
Re-Platform (Heterogeneous)
• Different operating system
• Different database engine
• Convert the database and/or application code
Re-Factor
• Decompose into smaller, logical components
• Right tool for the workload

Sources* Targets*
Oracle Oracle
SQL Server SQL Server
Azure SQL PostgreSQL
PostgreSQL MySQL
MySQL Amazon Redshift
SAP ASE SAP ASE
MongoDB Amazon S3
Amazon S3 Amazon DynamoDB
IBM DB2 Amazon Kinesis
Amazon ElasticSearch
• Start your first migration in 10
minutes or less
• Keep your apps running during the
migration
• Replicate from within, to, or from
AWS
• Move data to the same or different
database engine
Consult CHAP_Source.html and CHAP_Target.html pages
for latest DMS sources and targets

Keep your application running during migration
On-Premise AWS Cloud
Generic database DB Instance
Start a replication instance
Connect to the source and target
Select tables, schemas, or databases
Let DMS create the target objects
Move data and synchronize objects
Switch applications when ready

Database Migration Options
Load is table by table
• Configurable number of tables in parallel
One time load or Change Data Capture (CDC)
• Read from database log on the source and apply to the target
Filter criteria available for selective loading
• Select only a few tables or a subset of data in your tables
Multiple sources and targets. Mix and match.
• One side of the migration must be in AWS
Ongoing replication support
• Keep your replication going until your application is ready to cutover

Makes heterogeneous database migrations
predictable by automatically converting the source
database schema and a majority of the database
code objects, including views, stored procedures,
and functions, to a format compatible with the
target database
Features
Database Migration Assessment report for choosing the right target engine
Automatic conversion for eligible database objects and code
Code browser to highlight places where manual edits are required

Support for the following conversions
Source* Database Target* Database on AWS
Oracle database Amazon Aurora, MySQL, PostgreSQL, Oracle
Oracle data warehouse Amazon Redshift
Azure SQL Amazon Aurora, MySQL, PostgreSQL
Microsoft SQL Server Amazon Aurora, Amazon Redshift, MySQL PostgreSQL
Teradata Amazon Redshift
IBM Netezza Amazon Redshift
Greenplum Amazon Redshift
HPE Vertica Amazon Redshift
MySQL and Maria DB PostgreSQL
PostgreSQL Amazon Aurora, MySQL
Amazon Aurora PostgreSQL
IBM DB2 LUW Amazon Aurora, MySQL, PostgreSQL
Apache Cassandra Amazon DynamoDB
Consult CHAP_Source.html and CHAP_Target.html pages
for latest DMS sources and targets

AWS SCT converts objects and code

Pricing and platform support
Microsoft Windows
Apple Mac
Fedora Linux (rpm)
Ubuntu Linux (deb)
$0You can download AWS Schema Conversion Tool for your platform of choice

Database Migration Overview
Data
migration
Application
code
Schema
objects
Cutover
Analysis
Testing

Example migration plan
Analysis
Assessment 2%
Planning 7%
Schema
conversion
TESTING
Database conversion 14%
Procedure, function, and db script conversion 25%
Application
conversion
Application conversion/remediation 5%
Integration with 3rd party applications 2%
Data
migration
Data migration 3%
Functional end to end testing 28%
Performance testing 2%
Cutover
Integration and deployment 5%
Training and knowledge transfer 2%
Documentation and version control 2%
Post production support 3%

Resources and skills required
Database expertise is required
• Working knowledge of target database engine
Basic networking knowledge
• Familiarity with AWS VPC concepts
AWS knowledge is required
Software architecture knowledge is an advantage

Educate yourself and your team
Getting started
FAQS
Migration best practices
Database migration playbook

Plan your time
• Migration projects can take from several weeks to
several months to complete depending on the
complexity of the code conversion
• It may take several iterations and may take longer than
you anticipate
• Good planning will increase the success and reduce the
number of iterations required

Understand your database
Size
• Uncompressed size
• Number schemas and objects
• Relationships between schema
objects
• Number large tables (>500GB)
Datatypes
• LOB column number and size
• Custom or unusual datatypes
• Tables without PKs
Transaction profile
• Transactions per second
• Transaction boundaries
Authentication
• User authentication
• Roles and permissions
Integrations
• ETL jobs, interfaces, applications

Understand your network
Database access
• Firewalls, tunnels, VPN, Direct Connect
VPC
• Identify the target VPC and subnets you will use
IAM permissions
• Security groups and IAM permissions needed for DMS
Do you have enough bandwidth to move all your data?

Clarify your requirements
Can you afford downtime?
Do you need the source database to stay alive after migration?
Do you know why you prefer one target engine over another?
What are your high availability requirements?
Does all the data need to move?
Does it need to move to the same place?
Do you understand RDS benefits (backups, HA, etc.)
Do you understand RDS limitations? (storage size, admin user, etc.)

Create your target schema
SCT
• For heterogeneous migrations
• Orchestrate your migration through SCT by configuring your access keys
DMS
• Creates only tables and primary keys
• Uses transformation rules to convert case, filter, or change schema
Native tools for homogeneous migrations

Start your data migration
• Check prerequisites on the source
• Check the logs
• Check CloudWatch metrics
• Proper instance sizing
• Migration can take time, especially if your database is
large or your network connection is small

Things that affect your migration speed
Size of the source
Transaction pressure on the source
• Volume of transactions
• Run length of transactions
Size of the target
DMS task settings (# tasks, # tables, settings)
Bandwidth available
Size of the replication instance
Schema configuration
LOBs in the schema

When to choose native migration tools
Target supports native replication
Moving ALL the data
No transformations required
Target is a new database

When to use backup/restore or import/export
Database is small
Downtime window is large enough
Moving all the data
No transformations required

Larger database migration
What is large?
• Is a relative term
• Moving all the data takes longer than you are willing to wait
What can I do?
• Use AWS Snowball
• Let AWS Professional Services help you

Lessons Learned
ü Plan, Plan, Plan
• 80+% of the time spent during the migration should be planning
• Each aspect should be well documented, scripted, and automated
ü Test, Test, Test
• Conduct multiple test runs of the migration process until it is muscle memory
• It is the customer’s responsibility to provide a test plan

Summary
Migrations are complex
AWS SCT & DMS can automate a lot of the work
Know your environment (existing and new)
Bring the right skills
Allocate enough time
Sometimes native tools are the right choice
Many successful migrations
AWS is here to help

Questions?

AWS
John Franklin
Solutions Architect – Database Specialist

Agenda
• Overview
• Getting started
• Menu navigation
• Settings

Overview

AWS Schema Conversion Tool (SCT)
The AWS Schema Conversion Tool helps
automate many database schema and code
conversion tasks when migrating to a new
database engine.
Features
Schema conversion between database engines
Database Migration Assessment report for choosing the best target engine
Code browser that highlights places where manual edits are required

Getting Started

Connecting to your database
1. Create a new project
2. Connect to your source database
3. Enter the connection details

Migration assessment report
4. Run assessment
report
5. Review executive
summary
6. Review detailed
instructions

Convert schema and code objects
• Functions
• Indexes
• Packages
• Tables
• Triggers
• Schemas
• Sequences
• Stored Procedures
• Synonyms
• User Defined Types
• Views

Tree view of objects
• Tree view of
source
• Lazy loaded
objects

Actions at a glance
• Action items to
help with manual
conversion

Target database schema
• Tree view of
target
• Lazy loaded
objects

Source object code
• SQL code to
create chosen
source database
object
• Properties of
chosen source
database object

Target object code
• Editable SQL
code to create
chosen object in
the target
database
• Properties of
chosen target
database object

Apply to target database
Select Apply to
Database from the
Action menu
or
Right click on the
target schema to
apply

Menu navigation

Work with projects
Save your work
Open an existing project from a
list of recent ones
Connect to a source or target

Context sensitive actions
Menu options change when you work with the source versus the target

Switch between views
• Main view shows the working window for code conversion
• Assessment report view shows the summary and action item detail
• Database migration view for orchestrating your migration from within SCT

Convert an application
Use the AWS Schema Conversion Tool to convert your application code

Get help
Update AWS SCT
User Guide

Closer look at Settings

Global settings
Applies to your installation of SCT
Control
• Logging
• Drivers
• Performance and Memory
• Assessment Report
• AWS Service Profiles
• Security
• Notifications
• Tree View

Global settings - logging
• Application activity is
logged and file size is
retained to 100 MB up
to 20 files by default
• Increase the
information in the log
by setting Debug
mode to true

Drivers
• Add or update
drivers for your
databases
• Drivers are used
to connect to
the source and
target
• Use the
recommended
drivers in the
user guide

AWS Service Profiles
Add your AWS
Credentials to initiate
an AWS DMS
migration from within
AWS SCT
Enter your AWS
Access Key and AWS
Secret Key
Test the connection
to check permissions

Control your application and project settings
Default Project Settings
Indicates where your project
will be stored by default

Control your application and project settings
Project Settings
Settings specific to the open
project
Control
• Conversion settings
• Project environment
• SQL Scripting
• Tree View

Modify or add mapping rules
Import or export
transformation rules as JSON

Agenda
Overview
DMS components
Database migration options
Migration use cases
Best practices
Questions

Overview
• Simple to use
• Minimal downtime
• Supports widely used
databases
• Low cost
• Fast and easy to setup
• Reliable

Simple to use
No drivers or applications to install
No changes to the source database in most cases
Just a few clicks to start a migration from the console
DMS manages the complexities of migration for you
Automatically replicate changes
Can be used for continuous replication

Minimal downtime
Source
All changes can be
replicated to the target
Source database stays
fully operational during
the migration
Target
Target database stays
in sync with the source
for as long as you
choose
Switch over when the
target is fully sync’d
and without lag

Supports widely used databases
Sources* Targets**
Oracle Oracle
SQL Server SQL Server
PostgreSQL MySQL
MySQL Amazon Redshift
SAP ASE SAP ASE
MongoDB Amazon S3
Amazon S3 Amazon DynamoDB
IBM DB2 Amazon Kinesis
On-premises
database
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html
*
**

Low cost
Compute
resource
Additional
log
storage
Instance
Type
Hourly
Rate
Duration Activities Total
t2.small $0.036 2 weeks Testing $12.096
c4.large $0.154 2 weeks Initial
Load &
CDC Until
Cutover
$51.744
Migrate a 1 TB DB for under $65 ($63.84)
Pricing Example
https://aws.amazon.com/dms/pricing/

Fast and easy to setup
Set up a migration task in minutes
Connect to
the source
database
Connect to
the target
database
Create a
replication
instance to
run the
migration
Create a task Run the task
You can use different tasks with different settings for different environments

Reliable
• Highly resilient and self–healing
• Continual monitoring
• Source and target databases
• Network connectivity
• Replication instance
• In case of interruption, it automatically restarts the
process and continues the migration from where it was
halted
• Multi-AZ option for high-availability
CloudWatch

DMS Components

Database Migration Service components
Replication instance
Endpoint
Task

Replication instance
• T2, C4, or R4 instance types
Dedicated EC2 instance
• Private: Source or Target inside VPC / VPN / DC
• Public: Source or Target outside VPC
Public or private IP address
• Moves the data from the source to the target
• Support for multiple tasks
Task execution
Replication
Instance

Endpoint
Connection information to source or target database
• Endpoint type, identifier, engine (source or target)
• Server name, port, SSL mode
• Username, password
Advanced attributes
• Extra connection information
• KMS key
Test with a replication instance

Endpoint sources
Oracle
SQL Server
MySQL
SAP ASE
MongoDB Amazon S3
IBM DB2
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.html

Endpoint targets
Targets
Oracle
SQL Server
PostgreSQL
MySQL
Amazon Redshift
SAP ASE
Amazon S3
Amazon DynamoDB
Amazon Kinesis
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.html

Task
Name your
task
No spaces
or
underscores
Connect to
endpoints
Source
Target
Choose
migration type
Existing data
Existing data
and replicate
changes
Replicate
changes
only
Target
preparation
Do nothing
Drop tables
on target
Truncate
Include LOBs
Don't include
LOBs
Full LOB
mode
Limited LOB
mode*✓ Enable validation
✓ Enable logging

Task – migration type
Choose
migration type
Existing data
Existing data
and replicate
changes
Replicate
changes
only
Creates files or tables in the target database
Populates the tables with data from the source.
Migrate existing data option in the AWS console and Full Load in the
API.
Captures changes on the source during migration
Once initial migration completes, changes are applied to the target as
units of completed transactions
Migrate existing data and replicate ongoing changes option in the AWS
console and full-load-and-cdc in the API.
Reads the recovery file on the source database
Groups together transactions and applies them to the target. Buffering
as needed
Replicate data changes only option in the AWS console

In Do nothing mode, AWS DMS assumes target tables are pre-created. In
full load or full load plus CDC, ensure that the target tables are empty
before starting the migration.
In Drop tables on target mode, AWS DMS drops the target tables and
recreates them before starting the migration. This ensures that the
target tables are empty when the migration starts.
In Truncate mode, AWS DMS truncates all target tables before the
migration starts.
Task – Target preparation
Target
preparation
Do nothing
Drop tables
on target
Truncate

LOB columns are excluded from the migration.
Migrate complete LOBs regardless of size. AWS DMS migrates LOBs
piecewise in chunks controlled by the Max LOB size parameter. This
mode is slower than using Limited LOB mode.
Truncate LOBs to the value of the Max LOB size parameter. This mode is
faster than using Full LOB mode.
Task – Include LOBs
Include
LOBs
Don't
include
LOBs
Full LOB
mode
Limited
LOB mode*

More about replication tasks
• Tasks are the workhorse of the migration
• Tasks run on a replication instance
• Multiple tasks can be run in parallel
• Task settings JSON gives you control over performance
and debugging

Migration scenarios

Possible migration preparation scenarios
Using DMS alone
• DMS Creates tables and primary keys using Drop tables on target
• Manually add secondary indexes and foreign key objects after initial load has
completed.
Using SCT and DMS together
• Use Schema Conversion Tool to create all objects
• Disable secondary indexes and foreign key objects
• Load using DMS using Do nothing
• Re-enable secondary indexes and foreign key objects
Using Native Tools and DMS
• Use database scripts to create tables, primary keys, views, and sequences
• Load tables with DMS using Truncate
• Manually add secondary indexes and foreign key objects after initial load has
completed.

Steps to migrate - overview
Start the full load
While loading data, also capture changes
Load complete - apply captured changes
Changes reach steady state
Shutdown apps and apply remaining changes
Change your application endpoint

Replication
Instance
Source Target
Start full load

Source TargetReplication Instance
App
While loading data also capture changes

App
Load complete - apply captured changes

Changes reach steady state
App

App
Shutdown apps and apply remaining changes

Change your application endpoint
App

Load is table by table
Replication Instance
Source Target

Replication Server
Source Target
App
t
1
t
2
t
1
t
2
Changes are transactional - from db logs

DMS – change data capture (CDC)
“No Touch” design
• Reads recovery log of source database
• Using the engine’s native change data capture API
• No agent required on the source
• Changes captured as transactions and applied in order
• Activated when load starts
• Changes are applied after initial load is complete

DMS – change data capture (CDC)
Some requirements
• Oracle: Supplemental logging required
• MySQL: Full image row level bin logging required
• SQL Server: Recovery model bulk logged or full
• Postgres: wal_level = logical; max_replication_slots >= 1;
max_wal_Senders >=1; wal_sender_timeout = 0

Replication
Instance
Source Target
Multiple targets
Target
Target

Migration
Server
Source Target
Multiple Sources
Source
Source

Migration
Server
Source
Target
Multiple sources and targets
Source
Source
Target

You don’t have to take everything
Source
L
Target
Replication Instance

Homogenous or heterogeneous
Replication
Instance
SQL Server MySQL
Replication
Instance
Oracle Oracle
Replication
Instance
Oracle PostgreSQL

Size in GB Transaction volume
Size
Transaction Load
DMS limits
Too big
Toomanychanges

Setup and configuration tips
• Enable CloudWatch logs (not enabled by default)
• Choose LOB mode carefully
• Replication instance security group is default for VPC –
change after creation
• Extra connection attributes can alter how the migration
task operates
• Provide transformation rules for changing case

Source configuration tips
• Oracle specific settings for CDC
• Enable supplemental logging on tables with no primary key
• If you add a table mid-migration add supplemental logging
• Even if the table has a primary key
• To have DMS automatically configure supplemental logging in
Oracle add an extra connection attribute
• addSupplementalLogging=Y
• Check for unsupported datatypes
• Can use a read replica or standby as a migration source

Performance tips
• Use larger DMS instance for maximum throughput
• CPU for type conversions
• Check network throughput
• Split load across multiple tasks and/or DMS instances
• Remember transaction boundaries when capturing changes
• Reduce contention on your target
• Turn off logging
• Run in single AZ
Further reading on the AWS Database Blog

Questions

The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft

Semelhante a The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

The Future of Database Migration is Cloud, AWS Federal Pop-Up Loft