Oracle Database-Migration and -Upgrade with Transportable Tablespaces

1. Basel · Baden Bern · Brugg · Lausanne Zurich Düsseldorf · Frankfurt/M. · Freiburg i. Br. Hamburg · Munich · Stuttgart · Vienna
Database-Migration and –Upgrade
using Transportable Tablespaces –
report from a running project
Markus Flechtner
TechEvent April 2010

2. © 20102
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
Life, the universe
and everything.
Database-Migration and -Upgrade using Transportable Tablespaces

3. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 3
The setting (1) – customer situation
§ Our customer is one of the largest telecommunications equipment
suppliers in the world
§ They develop and sell hardware for cellular networks (GSM)
§ And they sell a „network performance management & capacity
management software“
§ A lot of mobile network operators worldwide
use this software

4. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 4
The setting (2) – the infrastructure
§ Current platform of the end customer:
ú Server Hardware: PA-RISC
ú Server OS: HP-UX
ú Oracle-Software: 10.2.0.4
ú File Storage: Filesystem
... But the PA-RISC processor architecture will be desupported in
the near future and they have to migrate to a new platform
§ Target platform of the end customer is:
ú Server Hardware: Intel
ú Server OS: Red Hat Linux 64bit
ú Oracle-Software: 11.2
ú RAC
ú File Storage: ASM

5. © 2010Database-Migration and –Upgrade using Transportable Tablespaces 5
The setting (3) – the project
§ First customer for data migration is the largest customer of the
application:
ú 4 databases, about 50.000 tables per database
ú Each DB Server: 32 CPUs, 96 GB RAM
ú DB Size between 3.2 and 5.5 TB, up to 1 TB redo log amount/day
ú Databases running in NOARCHIVELOG-mode (performance!)
ú New SAN
ú No chance to connect the old SAN to the new servers
ú Just a network connection available (Gigabit ethernet)
ú RAC with 42 nodes planned
§ Requirement:
Maximum downtime of the application of 5 – 6 hours!

6. © 2010Database-Migration and –Upgrade using Transportable Tablespaces 6
The setting (4) - summary
§ Large databases, running in NOARCHIVELOG-Mode
§ Platform change HP-UX à Linux
§ Oracle Upgrade: 10.2 à 11.2
§ From single instance to RAC
§ From file system to ASM
§ Limited downtime
§ Experiences from the current project will be used for future
customers

7. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 7
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
There's more
than one way to
skin a cat …

8. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 8
Why Transportable Tablespaces (TTS)?
§ Exp/imp and Datapump è Performance?
§ Tools like „Golden Gate“ or „Quest Shareplex“?
è Additional license costs
§ Streams? è ARCHIVELOG-mode required.
§ Data Guard / Physical Standby
è Not applicable due to platform change from HP-UX à Linux
§ Replication è Materialized views on 50.000 tables?
§ Self-made methods had bad performance

9. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 9
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
Data is always
part of the game.

10. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 10
The TTS approach (1) - Overview
§ „Transportable Tablespaces“ are used to transfer data between
databases by copying the data files and transfering the
corresponding metadata
§ Starting with Oracle 10g, this works even between different
operating systems (e.g. from HP-UX to Linux), changing the
„endianess“ of a datafile is possible using rman
§ It is a well known procedure

11. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 11
The TTS approach (2) – Specials
§ The set of tablespaces which is transferred must be „self-
contained“:
ú no reference to another database object in a tablespace outside the
tablespace set
ú This can be checked in advance and is usually no problem when
transferring a complete database
• Sequences, synonyms, temporary tables, privileges, objects
owned by SYSTEM or placed in the SYSTEM-tablespace etc.
must be transferred separately

12. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 12
The TTS approach (3) – The procedure (simplified)
1. Create an "empty" target database
2. Set the tablespaces to be transferred in „READ ONLY“ mode
3. Export the metadata using Oracle export or Oracle Datapump
(„which objects are in the tablespaces?“)
4. Transfer the export file and the datafiles to the target system.
- If necessary, convert the data files using rman
5. Import the metadata (import or Datapump)
6. Copy additional objects like sequences, synonyms etc. using
Datapump via Network Link
7. Set the tablespaces „READ WRITE“ in the target database

13. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 13
The TTS approach (4) – File Conversion
§ rman is used to change the endianess of the files and to move
the files from the file system into ASM
ú Big Endianess: HP-UX, Solaris (Sparc), AIX, Linux on zOS or
PowerPC
ú Little Endianess: Windows, Linux (Intel-based), VMS
rman> CONVERT DATAFILE 'copy_of_db_file_in_filesystem'
FROM PLATFORM 'HP-UX (64-bit)'
TO PLATFORM 'Linux x86 64-bit'
FORMAT 'file_location_in_ASM'
PARALLELISM 4;

14. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 14
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
You cannot
increase the
speed of light.

15. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 15
Tuning TTS (1) - Overview
§ Resize the files to the High Water Mark before the transfer
ú Less data to be read from the source system
ú Less data to be transferred via network
ú Less data to be converted using rman
§ Compression (for data transfer)
§ Parallelization
ú For all operations where possible (background jobs)
ú For transfer & conversion: transfer and convert multiple files in
parallel
§ Automation (using scripts)

16. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 16
Tuning TTS (2) – Reduce File Size
§ For all datafiles involved (pseudocode):
§ ORA-3297 "file contains used data beyond requested RESIZE
value"
§ Resize can be done "online" during normal business
§ Estimated result: Reduction of the database size to about 85 –
90 % of original size J
§ Files will be resized in target database after transfer
FOR ALL DATAFILES
SET AUTOEXTEND ON MAXSIZE 32G;
REPEAT
RESIZE FILE TO 95% of FILE_SIZE;
UNTIL ORA-3297;

17. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 17
Tuning TTS (3) – Compression
§ Facts:
ú When transfering a lot of large database files via network, network
bandwidth can be a bottleneck
ú Using a simple „gzip“, typical database files can be compressed to
less than 20% of their original size
Þ Why not use a „compressed transfer“?
§ Like:
ssh source_server "gzip remote_file –c"|gunzip >local_file

18. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 18
Tuning TTS (4) – Compression
Theoretical network bandwidth between old DB-Server and
new RAC-System
1 Gigabit/second
This is about 450 GB/hour
Let‘s take 75% of this theoretical value 337 GB/hour
Database Size (largest DB, without resize) 5.500 GB
Without SYSTEM- , SYSAUX- and UNDO-Tablespace 5.000 GB
Resized to 90 % original size 4.500 GB
Compressed to approx. 25% of original size 1.125 GB
Theoretical duration of the „compressed transfer“ 3,4 hours
+ metadata export/import

19. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 19
Tuning TTS (5) – Parallelization
§ Level 1:
ú Transferring and converting multiple files to one node in parallel
Source-Server
(32 CPU)
One RAC node Target Database
(on ASM)
File transfer
(ssh / gzip)
Conversion
(rman)

20. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 20
Tuning TTS (6) – Parallelization
§ Level 2:
ú Let's use the RAC and use all nodes for transfer & conversion
Source-Server
(32 CPU)
RAC nodes Target Database
(on ASM)
File transfer
(ssh / gzip)
Conversion
(rman)

21. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 21
Tuning TTS (7) – Automation
§ Basic database migration using TTS is a well-documented
straight-forward operation which can be scripted
§ One RAC node is the "target master" which coordinates the work
and starts all the scripts on the other nodes
§ Communication between the nodes and the scripts is done via a
table in the target database
ú contains information on file transfer status
ú contains information on tablespace transfer status (metadata)
ú files which could not be copied or converted using rman get an „ERROR“-
status and have to be transferred again

22. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 22
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
It‘s not as simple
as that!

23. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 23
Problems and challenges (1) – Metadata Export
§ In the lab exporting the metadata for the whole database using
DataPump took 12 hours
§ è Oracle said: "Use Original Export"
§ With "Original Export", it took just 6 hours J
§ But running the metadata export using "Original Export" in
another database resulted in an:
EXP-0008 / ORA-29336: Internal error … from DBMS_PLUGTS
§ è Oracle said: "Use DataPump"
§ L

24. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 24
Problems and challenges (2) – Metadata Export
§ Solution: Create self-contained subsets of the tablespaces and
export them in parallel using datapump
§ Preparation (Pseudocode):
§ The "grouping operation" can be very time-consuming (450+
tablespaces), but can be executed during normal operation
REPEAT
CREATE A NEW GROUP;
ADD TABLESPACE TO NEW GROUP;
REPEAT
CHECK TS-GROUP FOR SELFCONTAINMENT
(DBMS_TTS.TRANSPORT_CHECK_SET)
IF THERE ARE ROWS IN SYS.TRANSPORT_SET_VIOLATIONS
ADD THESE TABLESPACES TO GROUP;
UNTIL SYS.TRANSPORT_SET_VIOLATIONS IS EMPTY;
UNTIL ALL TABLESPACES ARE ASSIGNED TO A GROUP;

25. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 25
Problems and challenges (3) – Metadata Export
§ The „grouping operation“ resulted in about 120 small
tablespaces groups
§ These tablespace groups are grouped into larger "metagroups"
and every target node transfers a given number of metagroups:
ú 1. export metadata (in Background)
ú 2. transfer and convert datafiles (multiple jobs in parallel)
ú 3. import metadata
§ It takes just about 1,5 hours for all export jobs to complete J
§ Status information is stored in the database

26. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 26
Problems and challenges (4) – Job Coordination
§ Coordination of all the jobs between by the "target master"-node
ú Keeping control over all the parallel background jobs on source
server, RAC nodes, etc.
ú Communication via database table
§ The number of parallel transfer jobs is limited by:
ú CPU power on source
ú available disk space on the RAC nodes
èjob monitoring needed to avoid overloading or space problems
§ Error handling / Restart

27. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 27
Agenda
§ The setting
§ Why Transportable Tablespaces?
§ The TTS Approach
§ Tuning TTS
§ Problems and challenges
§ Outlook
What has the
future in store for
us?

28. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 28
Outlook (1) – The Procedure
§ The project is still ongoing ..
§ The procedure works J
§ The lab is too small for performance tests:
ú Target cluster has 4 nodes with 2 Quadcores and 32 GB RAM
ú But source server has just 2 CPUs and 8 GB RAM
ú Running more than 8 transfer jobs in parallel (2 per RAC node) kills
the source server
§ (End-)customer has a larger server (8 CPUs), but no RAC at the
moment L
§ Performance not tested so far …

29. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 29
Outlook (2) – The Plan
§ We're pretty confident that the procedure will work in production
§ For the small databases (2 databases with less than 4 TB) we'll
use the TTS-approach only
§ Depending on the results with these databases, we can change
the strategy for the larger databases:
ú 1. Use an offline backup of the database (volume split) as a source
for a TTS-based data transfer
ú 2. Use an already existing self-developed replication based on the
timestamps in the records for the remaining data
§ Plan B: Use "Golden Gate"

30. © 2010Database-Migration and -Upgrade using Transportable Tablespaces 30
References
§ Oracle MAA-Whitepaper „Platform Migration using Transportable
Tablespaces“
http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_platformmigrationtts.pdf
§ Oracle MAA-Whitepaper „Database Upgrade using
Transportable Tablespaces
http://www.oracle.com/technology/deploy/availability/pdf/maa_wp_11g_upgradetts.pdf
§ "Minimum Downtime Upgrade with Transportable Tablespaces"
http://blogs.oracle.com/UPDATE/
§ "Moving Oracle Databases Across Platforms without
Export/Import"
http://www.dbspecialists.com/files/presentations/changing_platforms.ppt

31. Basel · Baden Bern · Brugg · Lausanne Zurich Düsseldorf · Frankfurt/M. · Freiburg i. Br. Hamburg · Munich · Stuttgart · Vienna
Thanks for your attention!
.. any questions?