Migrate Oracle to Aurora PostgreSQL Best Practices

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate from Oracle to Aurora PostgreSQL:
Best Practices, Design Patterns, & Setup
Mahesh Pakala
Solutions Architect
Amazon Web Services
G P S C T 4 0 6

Agenda
• Introduction
• Planning Phase
• DB Freedom Migrations
• Source Server Setup/Config Recommendations
• Oracle Database: Source
• Oracle Key Parameter Setup/Config Recommendations
• AWS Database Migration Service (AWS DMS) Process
• DMS - Oracle RAC, ASM configurations
• PostgreSQL Database : Target
• Key Parameter Setup/Config Recommendations
• Design Guidelines
• Summary
+

Workload Qualification Framework (WQF)
• The WQF includes self-service tools, a questionnaire, and design review
guidance that solutions architects, partners, and consultants can use to
classify workloads to determine the ease of migration, staff-hour
consumption and to which appropriate AWS Service to migrate
• There are two main types of database workloads
Category Workload type Ease of
migration
Staff-hour
consumption
Category 1 ODBC/JDBC workload Easy Light
Category 2 Light Oracle feature workload Easy Medium
Category 3 Heavy Oracle feature workload Hard Heavy
Category 4 Oracle specific application
framework
Difficult N/A
Category 5 OCI workloads (Non-portable,
high-risk or lift-and-shift )
Hard Heavy
• Online Transaction Processing (OLTP): Many
small/fast transactions and queries via indexes that
return few rows
• Data Warehouse (DW): Interactive and long-running
queries that scan millions of rows and batched
updates
• WQF classifies OLTP and DW workloads into five
categories:

Planning : Schema Migration Complexity
Factors Wt Comment
Tables 0.5 Non Partitioned tables are converted automatically, Datatype and character set
issues
Materialized Views /
Triggers
3 Only Fully refreshed MV supported / Rewrite the logic
Partitions 0.1 Create child partitions Manually. Apply Partitioning arch with help of functions / trig
PL/SQL Packages 15 Packages with >= procedures or functions, review, or rewrite the logic
PL /SQL Procedures 4 Review or rewrite logic
LOB 1 Special Handling
DB
NAme
TA
B
MV DBL
INK
PRC PKG TRG TOTAL
WT
PRO 40 0 327
FACTORS SMALL MEDIUM LARGE TOTAL
WT
Complexity 1-500 500-
1000
1000-
2000
2000+
Redo /GB /
Hr
0-4 4-10 10-20 20+

Source: Oracle DB setup
• Server setup
• Infrastructure security
• Network I/O rates, transfers, and read/write ratios
• Database
• Oracle account access for the replicate user
• Archive logging
• Supplemental logging

Source server preparation
• Based on observability (data-driven)
• Observe workload and resource usage
• Analyze results
• Identify software and hardware components in use
• Study in-use components for tunables
• Quantify expected speed-up
• Tune, including experimental, the highest value targets
• Example methodology: USE Method
• Utilization, Saturation, & Errors

Source server metrics
• CPU utilization
• Memory use
• Kernel statistics and run queue information
• Disks I/O rates, transfers, and read/write ratios
• Free space on file systems
• Disk adapters
• Paging space and paging rates
• Network I/O rates, transfers, and read/write ratios

Network traffic I/O
• Network sniffer
• # ifconfig –a
• The primary Ethernet network interface is eth0. To find details of a
specific interface, for example eth0, you can use
• # ifconfig eth0
• /proc/net/dev, netstat, mii-tool – port speeds
• netstat –s or netstat –su and look for either “udpInOverflows”, “packet
receive errors” or “fragments dropped” – If you observe a consistent
increase in metrics that indicates an issue
• NIC Version : ethtool –driver eth0 - Port/packet speeds setup

Tracepath for jumbo frames
In this example, the path MTU is 1500

AWS DMS methods to read Oracle logs
• Oracle LogMiner
• Utility provided by Oracle, provides methods of querying
logged changes made to an Oracle database, principally
though SQL commands referencing data in Oracle redo logs.
• BinaryReader
• Binary Reader is a proprietary redo log parser for parsing the
Oracle redo logs.
• BinaryReader provides greatly improved performance and
reduces the load on the Oracle server when compared to
LogMiner

• Captures change data from operational system(s) as it occurs
• Delivers change data when you need it, where you need it
• Offers several tradeoffs between timely change delivery vs. source
system overhead (sync, async hotlog, async autolog, and so on)
• Assumes complete change management responsibility
• Captures all change data along with transaction information – See all
changes a given transaction made and who made them
• Transactional consistency for changes across multiple source tables is
guaranteed
• Transparently coordinates sharing of change data across users and
applications
What is Change Data Capture (CDC) ?

Oracle source environment
• Full load replication
• Creates files or tables at the target endpoint, automatically defines the
metadata that is required at the target, and populates the tables with data
from the source
• Change processing / Change Data Capture (CDC)
• Captures changes in the source data or metadata as they occur and applies
them to the target endpoint as soon as possible in near- real time
• Full load plus CDC
• Replication is log based
• Which means that it reads only the changes. This reduces the impact on the
source endpoints

AWS Database Migration Service (AWS DMS)

Reading redo from ASM
• Oracle permissions required
• Binary Reader and the "Copy redo logs to temporary folder" options
• Deleting the processed redo logs from temp folder
• Oracle ASM access permissions
• Setting up file share if the "direct access" option was chosen
• Configuring the "Copy to temp folder" option in replicate
• Accessible from ALL RAC nodes
• Multiple tasks using the same temporary folder may lead to
conflicts
• One task needing to access a redo log that another task has already deleted
• One replicate task access the same temp folder
• Different sub folders for different tasks

Advantages of Binary Reader with AWS DMS
• Migrations with high volume changes
• LogMiner has I/O or CPU impact on Oracle DB Server
• Binary Reader has less chance of having I/O or CPU impact
• Supports CDC for LOBS in Oracle version 12c
• LogMiner does not
• Supports Oracle transparent data encryption (TDE)
• Supports HCC compression types for both FULL LOAD and
CONTINUOUS REPLICATION (CDC)
• QUERY HIGH, ARCHIVE HIGH & ARCHIVE LOW
• Supports QUERY LOW compression type only for FULL LOAD
• Better performance and reduces load on Oracle than LogMiner

Encryption tips
AWS DMS TDE Tablespace TDE Column
Binary Reader Yes Yes
LogMiner Yes Yes
• BinaryReader supports CDC for LOBS in Oracle version 12c, but
NOT LogMiner
• To replicate encrypted columns in tables, need to specify the
Oracle Wallet encryption keys and their values

Target: Security infrastructure
• Is the cluster on VPC?
• Is it launched to a private subnet?
• Are security groups restricted only to specific IPs?
• Have you used a non-standard port?
• Are audit logs enabled?
• IAM ownerships are in place to secure snapshots?
• Is the cluster encrypted? Using HSM or AWS Key Management Service
(AWS KMS)?
• Are clients forced to use SSL?
• Are Amazon CloudWatch alarms in place?

Target: PostgreSQL DB setup
• Disable
• DB instance backups (set BACKUP_RETENTION to 0)
• Multi-AZ – To avoid latency due to sync replication
• SYNCHRONOUS_COMMIT BUT not FSYNC
• Do NOT Disable AUTOVACUUM
• Do NOT Modify VACUUM_FREEZE_MIN_AGE
• VACUUM ANALYZE after initial load to keep latest statistics
• Adjust the logging default to be more VERBOSE

Target : PostgreSQL DB setup
• SHARED_BUFFERS
• EFFECTIVE_CACHE_SIZE
• PG_BUFFERCACHE
• MAX_CONNECTIONS
• Hard Limit
• MAX_WAL_SIZE
• Effects checkpoint ( exceed maxl _wal_size or timing)
• WAL_LEVEL = LOGICAL; MAX_REPLICATION_SLOTS >= 1;
• MAX_WAL_SENDERS >=1; WAL_SENDER_TIMEOUT = 0
• Not adjustable in Aurora PostgreSQL; only Amazon RDS PostgreSQL

Target : PostgreSQL DB Setup
• Increase
• WAL_BUFFER, MAINTENANCE_WORK_MEM
• checkpoint_segments from 16(256M) to 256(256*16=4096=4G) at least, even
1024
• checkpoint_timeout to 1800 (default is 300).
• Not adjustable in Aurora PostgreSQL but only in RDS PostgreSQL.
• Maintenance Work Mem
• Free mem / max connections
• Remove FK
• Remove Cross reference tables
• AWS Reports to identify unused objects

Design guidelines
• Partitioning
• Prior PostgreSQL 10: Inheritance (Parent / Child Tables)
• From PostgreSQL 10: Declarative Partitioning (Create Table Partition Name)
• Oracle has Hash, Range, List, Composite <> PostgreSQL has Only List and Range
• Exchange / Split Partitioning not available
• Materialized Views
• FAST REFRESH : Not Available
• Automatic incremental refresh - NA
• Temporary tables are dropped after every SESSION and RECREATED
• Multiple Session cannot access the temp table
• Parameters
• Random_page_cost =1 (4 was the default : Index Scan if you are having table scan)
• Set ON - Log_connections, log_disconnections, log_lock_waits, huge_pages
• Shared_preload_libraries=‘pg_stat_statements,auto_explain.pgaudit’

Creating a list partitioned table : 5 Step Process
1. Create the parent table:
CREATE TABLE SYSTEM_LOGS
(EVENT_NO NUMERIC NOT NULL,
EVENT_DATE DATE NOT NULL,
EVENT_STR VARCHAR(500),
ERROR_CODE VARCHAR(10));
2. Create child tables (“partitions”) with check constraints:
#CREATE TABLE SYSTEM_LOGS_WARNING (
CHECK (ERROR_CODE IN('err1', 'err2', 'err3')))
INHERITS (SYSTEM_LOGS);
#CREATE TABLE SYSTEM_LOGS_CRITICAL (
CHECK (ERROR_CODE IN('err4', 'err5', 'err6')))
INHERITS (SYSTEM_LOGS);
3. Create indexes on each of the child tables (“partitions”):
#CREATE INDEX IDX_SYSTEM_LOGS_WARNING ON
SYSTEM_LOGS_WARNING(ERROR_CODE);
#CREATE INDEX IDX_SYSTEM_LOGS_CRITICAL ON
SYSTEM_LOGS_CRITICAL(ERROR_CODE);
4. Create a function to redirect data inserted into the Parent Table:
CREATE OR REPLACE FUNCTION SYSTEM_LOGS_ERR_CODE_INS()
RETURNS TRIGGER AS
$$
BEGIN
IF (NEW.ERROR_CODE IN('err1', 'err2', 'err3')) THEN
INSERT INTO SYSTEM_LOGS_WARNING VALUES (NEW.*);
ELSIF (NEW.ERROR_CODE IN('err4', 'err5', 'err6')) THEN
INSERT INTO SYSTEM_LOGS_CRITICAL VALUES (NEW.*);
ELSE
RAISE EXCEPTION 'Value out of range, check
SYSTEM_LOGS_ERR_CODE_INS () Function!';
END IF;
RETURN NULL;
END;
$$
LANGUAGE plpgsql;
5. Attach the trigger function created above to log to the table:
CREATE TRIGGER SYSTEM_LOGS_ERR_TRIG
BEFORE INSERT ON SYSTEM_LOGS
FOR EACH ROW EXECUTE PROCEDURE SYSTEM_LOGS_ERR_CODE_INS();
6. Insert data directly into the parent table:
demo=# INSERT INTO SYSTEM_LOGS VALUES(1, '2015-05-15', 'a...', 'err1');
demo=# INSERT INTO SYSTEM_LOGS VALUES(2, '2016-06-16', 'b...', 'err3');
demo=# INSERT INTO SYSTEM_LOGS VALUES(3, '2017-07-17', 'c...', 'err6');
7. View results from across all the different child tables:
demo=# SELECT * FROM SYSTEM_LOGS;

Design Guideline : Materialized ViewOracle PostgreSQL
Create Materialized
View
CREATE MATERIALIZED VIEW mv1 AS SELECT * FROM
employees
CREATE MATERIALIZED VIEW mv1 AS SELECT * FROM
employees
Manual refresh of a
Materialized View
DBMS_MVIEW.REFRESH('mv1', 'cf');
The--cf parameter configured the refresh method: c is complete
and f is fast
REFRESH MATERIALIZED VIEW mv1;
Online refresh of a
Materialized View
CREATE MATERIALIZED VIEW mv1 REFRESH FAST ON COMMIT
AS SELECT * FROM employees;
- Supports automatic incremental refresh
- Supports DML on the MV
Create a trigger that will initiate a refresh after every DML
command on the underlying tables:
CREATE OR REPLACE FUNCTION refresh_mv1()
returns trigger language plpgsql
as $$
begin
REFRESH MATERIALIZED VIEW mv1;
return null;
end $$;
CREATE TRIGGER refresh_ mv1
after insert or update or delete or truncate
on employees for each statement
cron job ->
execute procedure
refresh_mv1();
Automatic
incremental refresh of
a Materialized View
CREATE MATERIALIZED VIEW LOG ON employees…
INCLUDING NEW VALUES;
CREATE MATERIALIZED VIEW mv1 REFRESH FAST AS SELECT *
FROM employees;
Not Supported
DML on Materialized
View data
Supported Not Supported

Design guidelines
• Sequence Cache 20
• Oracle Sessions are contiguous
• PG : caches per session - Session A starts 1 and session B 120
• UNDO segments – ORA-01555 (Primary Key Filtering)
• Triggers - Oracle : After : DML / DDL/ DB ops
When
Fired
Database Event Row-Level Trigger
(FOR EACH ROW)
Statement-Level Trigger
(FOR EACH STATEMENT)
BEFORE INSERT, UPDATE, DELETE Tables and foreign
tables
Tables, views, and foreign
tables
TRUNCATE - Tables
AFTER INSERT, UPDATE, DELETE Tables and foreign
tables
Tables, views, and foreign
tables
TRUNCATE - Tables
INSTEAD OF INSERT, UPDATE, DELETE Views --
TRUNCATE -- --

Truncation and Performance of LOBS
• If max LOB size <=64 KB, we
read the LOBs in array mode
with default array size set to
1000
• If max LOB > 64 KB, we read
LOB in array mode :
• If LOB size <= 64KB, we migrate and
move on
• If LOB size > 64KB, we use GetData()
method for CLOBs and OCILobRead2
for BLOBs
• Check if LOB value is null. If
not null, send a signal to
lookup the LOB from source
• We then use OCILobRead2 to
read the first chunk of “full
lob chunk size” source
(default chunk size is 64 KB),
and send to the target. Then
we read the next chunk and
send to target and so on,
until the LOB is completely
read.

Customer data center AWS
Internet
Oracle to PostgreSQL migration using AWS DMS
VPN
AWS SCT
Migration agents
(DW)
Same AWS account & region
AWS DMS
Replication
instance
AWS DMS
Replication
instance
EC2 instance
web app
server
EC2 instance
web app
server
1
3
4
5
2
DX

AWS DMS best practices
• Improving performance of an AWS DMS migration
• Load multiple tables in parallel
• Remove bottlenecks on the target
• Use multiple tasks
• Improving LOB performance
• Optimizing change processing
• Determining the optimum size for a replication instance
• Reducing load on your source database
• Ongoing replication

Comparison of RDS & Aurora
Category RDS Aurora
Performance Good performance 5x faster for MySQL
3x faster for PostgreSQL
Scalability Up to 5 read replicas
Lag measured in seconds
Up to 15 read replicas
Lag measured in milliseconds
Failover Around 60 seconds Less than 30 seconds
Storage Scales up to 16 TB Scales up to 64 TB. Auto-scales in 10 GB increments.
High Availability Multi-AZ is available Writes 6 copies to 3 AZs
Backup Takes daily snapshot during
backup window & captures
transaction logs
Continuous, asynchronous backup to S3 (no backup
windows)
Instance Types M3, M4, R3, T2 R3, R4, T2 for MySQL
R4 for PostgreSQL
Innovations with Aurora Fast Database Cloning
Fast DDL
Advanced Auditing
Performance Insights (Preview)
Parallel Query (Preview)
Serverless (Preview)
Multi-Master (coming)

Alexa + Performance Insights
New integration between Alexa & Performance
Insights can be your assistant DBA
Uses the Performance Insights API to identify
bottlenecks in Amazon RDS
Get actionable suggestions such as on-demand
scaling, DBA notifications & paging
Go to the Slalom booth (#1438) to see a live demo
and learn more!

Thank you!
Mahesh Pakala
pakalam@amazon.com

Extras

AWS SCT guidance

AWS SCT helps with converting tables, views, &
code
Sequences
User-defined types
Synonyms
Packages
Stored procedures
Functions
Triggers
Schemas
Tables
Indexes
Views
Sort and distribution
keys

Migrate Oracle to Aurora PostgreSQL Best Practices

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Migrate Oracle to Aurora PostgreSQL Best Practices

Semelhante a Migrate Oracle to Aurora PostgreSQL Best Practices (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Migrate Oracle to Aurora PostgreSQL Best Practices