Oracle to Netezza Migration Casestudy

1. Oracle to Netezza
Migration
Whitepaper
This white paper provides key challenges faces and resolved
and the approach followed in Oracle to Netezza migration for
one of the fortune 100 Retailer.
Asis Mohanty, asismohanty@gmail.com
11/11/2011

2. Table of Contents
1. Introduction .......................................................................................................................................... 3
2. Migration Approach and Process .......................................................................................................... 3
3. Impact Analysis ..................................................................................................................................... 3
3.1 Database Impact Analysis: .................................................................................................................. 3
3.2 ETL/ELT Impact Analysis: ..................................................................................................................... 4
3.3 Applications and Reports Impact Analysis: ......................................................................................... 4
4. Design and Development ...................................................................................................................... 4
4.1 Database Objects Design and Development Considerations: ............................................................ 4
4.2 ETL/ELT Design and Development Considerations: ............................................................................ 5
4.3 Report Design and Development Considerations: .............................................................................. 5
5. History Load .......................................................................................................................................... 5
6. Testing ................................................................................................................................................... 6
7. Lesson Learnt and Recommendations .................................................................................................. 7
8. Appendix ............................................................................................................................................... 9
Automation Recommendations ................................................................................................................ 9
About the Author ...................................................................................................................................... 9
| Oracle to Netezza Migration 2

3. 1. Introduction
Every organization wants to have advance analytics, real time or near real time data which will help then
in Business Intelligence and competitive Intelligence. High volume growth, scalability, Maintainability
cost is the critical factor for any organization. Netezza provides a platform which is highly Scalable with
availability, easy to integrate and maintain, cost effective (Pay per usage), Asymmetric Massively Parallel
Processing architecture for performance.
This white paper contains the approach followed, key challenges faces and resolved during the Oracle to
Netezza migration program for one of the Fortune 100 Retailer.
2. Migration Approach and Process
Oracle to Netezza migration has various SDLC process and steps as given below (Figure-1).
Figure-1: Various steps involved in Oracle to Netezza Migration
3. Impact Analysis
This is the most critical phase to identify all impacted Data Base Objects, ETLs and applications/Reports
required for downstream.
3.1 Database Impact Analysis:
Identify all impacted Data Base Objects such as Tables, Views, Stored procedures, Index, Partitions,
Volumetric (Volume of Data to be migrated) to be migrated from existing Oracle platform to Netezza
Platform. Create the impact inventory list of DB Objects.
| Oracle to Netezza Migration 3

4. Figure-2: Example of Existing DB & ETL Environment
For example in the above figure (Figure-2), as part of Data Base Object impact identify all Tables, Views,
Stored procedures, Index, Partitions, Volumetric being used in Stage, Warehouse and Mart
environment.
3.2 ETL/ELT Impact Analysis:
Identify all ETL mapping or Jobs and ELT scripts being used by all impacted DB Objects. Create the
inventory list of all impacted ETL mappings or Jobs and ELT Scripts along with patterns.
For example in the above figure (Figure-2), as part of ETL/ELT impact identify all ETL/ELT Scripts being
used from Source to Stage, Stage to Warehouse and Warehouse to MART.
3.3 Applications and Reports Impact Analysis:
Identify all Reports or applications which are using the Warehouse or Mart Data. Create the inventory
list of all impacted Reports and applications.
Figure-3: Example of Existing Reporting and Application Environment
For Example as shown in the above figure (Figure-2) identify all Reports and various application which is
using the Warehouse or Mart DB Objects (Tables, Views .etc).
4. Design and Development
This section contains the design and development of Database Objects, ETL/ELT and
Reports/Applications.
4.1 Database Objects Design and Development Considerations:
• As-Is migration verses re-architecture of Data model is most critical factor.
| Oracle to Netezza Migration 4

5. • Identify all Oracle Vs Netezza DB Object data type, Size and compatibility. For example the Date
time stamp format, CLOB, LONG Data type etc.
• Identify all Oracle Vs Netezza DB functions. Data type and function will be helpful while
converting and Oracle SQL to Netezza SQL.
• Netezza does not support Index where as it supports Distribution Key. While creating the new
Netezza Tables the Oracle Table Index must be considered.
4.2 ETL/ELT Design and Development Considerations:
• Identify different patterns of ETL Mapping/Jobs.
• Identify all impacted ETL where Source connection to be modified from Oracle to Netezza.
(Example: All Sources to Stage ETL will fall under this category. Refer Figure-1)
• Identify all impacted ETL where Source and Target connection to be modified from Oracle to
Netezza. (Example: All ETL Mapping/Jobs being used in Stage to Warehouse and Warehouse to
Mart load)
• Identify all embedded SQL queries used in ETL mappings/Jobs which need to be converted from
Oracle to Netezza specific SQL Query.
• Identify all ELT Scripts where embedded Oracle SQL queries to be converted to Netezza SQLs.
• Make the Netezza specific changes in ETL Mappings/Jobs and do unit testing. Early Unit testing
identifies various issues.
4.3 Report Design and Development Considerations:
• Identify different patterns of Reports.
• Identify all embedded SQL queries used in reports by report.
• Identify all matrices which used Oracle specific Functions and Data type which needs to be
converted to Netezza specific functions and data type.
• Make Netezza specific changes in Reports. There might be some extra space, rounding, Special
Character issue which needs to be addressed during development phase.
5. History Load
History load from Oracle to Netezza can be done in following methods.
• Extract Oracle Table to flat file using Oracle Database utilities. Load the flat file to corresponding
Netezza table using NZ Load. (Always use Bulk load/insert/update utility which is faster)
• Use the ETL tool to create one time mapping which will load history data from Oracle to
Netezza.
• Use Unix/Perl script to extract data from Oracle and load it to Netezza.
• Use Third Party tool which uses the Bulk Read and Write mechanism to migrate history data
from Oracle to Netezza. Wisdomforce Fastreader (www.wisdomforce.com/products-
FastReader.html) supports the bulk/fast load from Oracle to Netezza.
| Oracle to Netezza Migration 5

6. • Define the History data Delta if the history data volume is high, for example the existing
Enterprise Data Warehouse has 6 Years worth of Data. Define two to three deltas each 2 Years
worth of data to load.
6. Testing
Testing is the most important phase to identify any issue. Following are few of the recommendations:
• Database Object Testing
o Compare the DB Objects (Oracle Vs Netezza) along with distribution Key.
o Create automated script which can compare the DB Objects by querying System table
scans.
• SQL Testing
o Compare the Oracle SQL results with Netezza SQL results. The SQL used in ETL, Scripts,
Reports must be compared during the development phase.
o There are third party tolls available for Oracle to Netezza SQL conversion such as Sqissql
(www.swissql.com). Even automated Perl or UNIX scripts can be created to convert the
SQL.
• ETL Testing/Data Testing:
o Run the existing Oracle ETL Job and new Netezza ETL Job. Compare the target Oracle
table Vs Netezza table data.
o If the Table volume is small, do a byte to byte data comparison of the data (Except Data
Time Stamp fields)
o If the volume is high use mathematical function (Min, Max and Average of additive field)
along with count match. Do a sampling using Min/Max functions.
o Created automated script to compare the Data. This script and method will be used for
History data comparison too.
• Report/Application Testing:
o Run the report/Application in Oracle and Netezza environment. Compare the report
manually or use Reporting environment specific tool to compare the result. For Example
Microstrategy has ‘Microstrategy integrity manager’ which compares the reports.
o There are third party tools available which compared Byte-to-Byte of .csv, pdf, xls, html
reports, for example Beyondcompare (www.scootersoftware.com) .
• Performance Testing:
o Capture the ETL run statistics while running in Oracle environment and Netezza
Environment. Compare the ETL load run statistics to know the performance. This must
be done for the load window too.
o Capture the Report run statistics for Oracle and Netezza environment. Compare the run
statistics for Report performance in new Netezza environment.
| Oracle to Netezza Migration 6

7. 7. Lesson Learnt and Recommendations
Following are few of the lessons learnt from the Oracle to Netezza Migration Project.
POC
o Do POC for early identification of issues and approach of Migration.
Product Issue Resolution
o There might be product related issue (Compatibility) which must be resolved quickly.
o Netezza Special Characters (Ex: Trademark) might appear differently in reports.
o Rounding of matrices in Reports.
o Extra space in matrices, field of Reports.
Resource
o Cross skilled resource (resource having Oracle and Netezza Knowledge)
Automation accelerators
o Automation is the key to avoid any manual mistake during the Analysis, Development
and Testing. Following Figure-4 shows the automations recommendations for Oracle to
Netezza Migration.
Figure-4: Automation Recommendations in various Phases
Upgrade of Reporting and ETL Environment
| Oracle to Netezza Migration 7

8. o Upgrade of ETL environment or Reporting environment along with DB Platform
migration is not recommended.
Performance
o Perform Performance testing for any early indication of system performance.
Usage of Advance Feature of Netezza
o Use the advance feature such as NZ Load utilities for better performance for high
volume of data load.
| Oracle to Netezza Migration 8

9. 8. Appendix
Automation Recommendations
Tools to Convert Oracle to Netezza SQLs and DDLs: Create Unix/Perl script which converts the Oracle
specific DDL or SQL to Netezza specific DDL/SQL. This script will use the Oracle Vs Netezza Data Type,
Functions and patterns to convert.
Tools to Convert the Oracle ETL to Netezza ETL: Create a tool which takes ETL xml as input and changes
the Oracle tag to Netezza tag (connections, extra Tags) and converts Oracle embedded SQLs to Netezza
SQLs.
Data Testing Tool: Create a Unix/Perl script/tool which compares the Oracle Table data Vs Netezza Table
data. Following are the two patterns:
Low volume Table: Compare the byte-to-byte data.
High Volume Table: Compare the Count of records, use mathematical functions such as Min, Max, Sum,
Average .etc for numeric field. Use sampling of data comparison (Top 10 records using Rank functions
.etc)
About the Author
Asis Mohanty has more than 11 Years of Industry experience on Data Warehousing and Business
Intelligence field. He is a Certified Business Intelligence Professional from www.tdwi.org and Certified
Data Management Professional from www.dama.org and Netezza Silver Certified Professional. For any
query contact Asis Mohanty at asismohanty@gmail.com .
| Oracle to Netezza Migration 9