Oracle is the most powerful DBMS in the world. However, Oracle's expensive and complex replication makes it difficult to build highly available applications or move data in real-time to data warehouses and popular databases like MySQL. In this webinar you learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too – picture Oracle GoldenGate without the price tag! We demo constructing a highly available site using Oracle-to-Oracle replication. We then show you how to replicate data in real time from Oracle to MySQL as well as load a data warehouse.

1. ©Continuent 2013.
Tungsten University
Replicate between
MySQL and Oracle
Linas Virbalas, Senior Software Engineer
Robert Hodges, CEO

2. ©Continuent 2013
Introducing Continuent
2
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement

3. ©Continuent 2013.
Tungsten Replicator
3
Tungsten Replicator is a fast,
open source, database
replication engine
Designed for speed and flexibility
GPLV2 license
100% open source
Annual support subscription available from Continuent
“Golden Gate without the Price Tag”
®

4. ©Continuent 2013.
Master
Slave
THL
(Transactions + Metadata)
Login as
replication
client
Apply using JDBC
Replicator
MySQL
Binlog
THLReplicator
Store
forward
Store
Replicator Overview
(Transactions + Metadata)

5. ©Continuent 2012 5
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Extract Filter Apply
Stage
Pipeline
Master
DBMS
Transaction
History Log
Queue
Slave
DBMS
Tungsten Replication Service

6. ©Continuent 2013.
Multiple Services per Replicator
Service
frommysql
Service
fromoracle
frommysql
fromoracle
Replicator
Replicator
Replicator
Aggregated

7. ©Continuent 2013
star
master-slave Heterogeneous
fan-in slave all-masters
MySQL
Oracle
Oracle
MySQL Oracle
Oracle
MySQL MySQL

8. ©Continuent 2012.
Enabling
Heterogeneous
Replication

9. ©Continuent 2013.
Steps to Heterogeneous Replication
1. Prepare (translate) schema for
the slave DBMS
2. Set up replication
3. Provision initial data
4. Continue real-time replication
ddlscan
tungsten-installer
Parallel Apply

10. ©Continuent 2012.
Replicating from
MySQL to Oracle

11. ©Continuent 2013.
Use Case: Real-Time Sales/Order
Ful!llment
Real-Time Replication
Web-based Sales Purchase Ordering
3M
Order
Items
~1M
Xacts/
Day

12. ©Continuent 2012.
1.Translating schema
for the slave

13. ©Continuent 2012
Introducing ddlscan
13
• Helps to prepare schema for heterogeneous
replication
• Single command-line call
• Part of Tungsten Replicator 2.1, GPL v2

14. ©Continuent 2012
Translating Schema
14
• Goal - set up heterogeneous replication
Replication

15. ©Continuent 2012
Translating Schema
15
• Beginning - how to convert tables?
empty
Tables

16. ©Continuent 2012
Translating Schema
16
• Beginning - how to convert tables?
empty
Tables
•Data types?
•Column lengths?
•Naming conventions?
•Reserved words?

17. ©Continuent 2012
Translating Schema
17
• ddlscan automates these conversions
empty
Tables
ddlscan

18. ©Continuent 2012
Features
18
• Translates schema with replication in mind
• Provides errors and warnings proactively
• Can rename schema/tables/columns
• Easy
• Straightforward to extend (write new
templates)

19. ©Continuent 2012
Usage (MySQL to Oracle Example)
19
$ cd tungsten-replicator/bin
$ ./ddlscan
-db test
-template ddl-mysql-oracle.vm
-user tungsten
-pass secret

20. ©Continuent 2012
Translating Schema
20
• ddlscan looks into source schema
empty
Tables

21. ©Continuent 2012
Translating Schema
21
• ddlscan translates and renders DDL commands
empty
Tables
oracle-ddl.sql

22. ©Continuent 2012
Result of ddlscan
22
CREATE TABLE 3colors
/* ERROR: table starts with a number, use rename option and a
filter */
(
id NUMBER(10, 0),
color VARCHAR2(1) /* ENUM('R','G','B') */,
enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */,
acolumnthatdoesntfittooracleat NUMBER(10, 0)
/* WARN: truncated column name exceeding 30 characters
(acolumnthatdoesntfittooracleatall) */
/* ERROR: table must have a primary key! */
);
CREATE TABLE talks
...

23. ©Continuent 2012
Translating Schema
23
• You run resulting SQL file on Oracle
empty
Tables
oracle-ddl.sql

24. ©Continuent 2012
Translating Schema
24
• Tables are ready!
Tables Translated
tables (empty)

25. ©Continuent 2013.
Architecture
25
Tables
$ ./bin/ddlscan
JDBC
metadata
Libraries
Tungsten
Replicator
Template
engine
ApacheVelocity
Schema
as JAVA
objects
MySQL,
Oracle,
PostgreSQL
*.vm
template
of choice
Rendered
text output

26. ©Continuent 2012.
2. Set Up Replication

27. ©Continuent 2012
Master Installation
27
tools/tungsten-installer -a
--master-slave
--service-name=frommysql
--cluster-hosts=db1
--master-host=db1
--datasource-port=3306
--datasource-user=tungsten
--datasource-password=secret
--home-directory=/opt/continuent
--mysql-use-bytes-for-string=false
--mysql-enable-enumtostring=true
--mysql-enable-settostring=true
--start-and-report

28. ©Continuent 2012
Slave Installation
28
tools/tungsten-installer -a
--master-slave
--service-name=frommysql
--cluster-hosts=db2
--master-host=db1
--datasource-type=oracle
--datasource-oracle-service=ORCL
--datasource-user=tungsten_frommysql
--datasource-password=secret
--home-directory=/opt/continuent
--svc-applier-filters=dropstatementdata
--start-and-report

29. ©Continuent 2012.
3. Provisioning
Initial Data

30. ©Continuent 2012
Provisioning (Single Threaded)
30
• Restore a backup and let it replicate
Replication
Provisioning
Schema
Translated
Tables
Production
Schema

31. ©Continuent 2013.
Parallel Apply on Replicas
Extract
Filter
(Shard)
Apply
Extract/Filter/
Apply
Stage Stage Stage
Pipeline
Master
DBMS
Slave
DBMS
Transaction
History Log
Parallel
Queue
Extract/Filter/Apply
Extract/Filter/Apply
Extract/Filter/Apply
Provisioning
Multi-
tenant
source

32. ©Continuent 2012
Provisioning (Parallel Apply)
32
• Restore a backup and let it replicate
Production
Schema
Translated
tables
Replication
Replication
Replication
Replication
Replication
Replication
Replication
Provisioning
Schema

33. ©Continuent 2012.
4. Continue Real-Time
Replication

34. ©Continuent 2012
Continue Replicating Real-Time Data
34
• Discard provisioning schema and leave single
channel
Production
Schema
Provisioned
Tables
Replication
Replication
Replication
Replication
Replication
Replication
Replication
Provisioning
Schema

35. ©Continuent 2013.
How Does Tungsten Do That?
35
MySQL
Binlog
binlog_format=row
Service
frommysql
Tungsten Master
Replicator
MySQLExtractor
Special Filters
•Transform
ENUM to string
•Transform SET
to string
Service
frommysql
Tungsten Slave
Replicator
Special Filters
•Drop DDL
•Map names to upper case
•Rename too long or reserved
objects
•Optimize updates to remove
unchanged columns
•Ignore extra tables
OracleApplierMySQLExtractor

36. ©Continuent 2012.
Replicating from
Oracle to MySQL
* newly open-sourced!

37. ©Continuent 2013.
Use Case: Web Content Publishing
Real-Time Publication
Backend Office Web-Based Catalog

38. ©Continuent 2013.
How Does Tungsten Do That?
38
Service
fromoracle
Tungsten Master
Replicator
OracleCDCExtractor
No Special Filters
Service
fromoracle
Tungsten Slave
Replicator
Special Filters
•Map names to lower case
•Ignore extra tables
•Heartbeat table renaming
demo
schema
demo_pub
schema
Oracle CDC
(Synchronous or
Async Hotlog)
MySQLApplierOracleCDCExtractor

39. ©Continuent 2012.
Replicating from
Oracle to Oracle
* newly open-sourced!

40. ©Continuent 2013.
How Does Tungsten Do That?
40
Service
fromoracle
Tungsten Master
Replicator
OracleCDCExtractor
No Special Filters
Service
fromoracle
Tungsten Slave
Replicator
Special Filters
No Special Filters
demo
schema
demo_pub
schema
Oracle CDC
(Synchronous or
Async Hotlog)
Tungsten Slave
Replicator
OracleApplierOracleCDCExtractor

41. ©Continuent 2013.
More Replication
Possibilities!

42. ©Continuent 2013.
Oracle RAC and Continuent Tungsten
Continuent Tungsten
Cluster for MySQLOracle 11G RAC
Connect your favorite clusters!

43. ©Continuent 2013.
Oracle-Based Analytics and Reporting
Oracle Data Warehouse
Sharded MySQL
Operational Data
MySQL Fan-In
Aggregation Server

44. ©Continuent 2013.
Oracle-Based Analytics and Reporting
Oracle Data Warehouse
Sharded MySQL
Operational Data (HA)
MySQL Fan-In
Aggregation Server

45. ©Continuent 2013.
Oracle-Based Analytics and Reporting
Web-facing MySQL DBMS
with HA and scaling
Backend Oracle
app B
Backend Oracle
app A
Aggregate data

46. ©Continuent 2013.
46
More Replication Combinations
PostgreSQL
MongoDB
Vertica
MySQL

47. ©Continuent 2013
Support for Tungsten Replicator
47
• Community support is available on project
mailing list
• Continuent o!ers implementation services and
annual 24/7 support subscriptions
• We do not o!er hourly or per-incident support
(sorry!)
• For high availability or disaster recovery needs,
check out Continuent Tungsten clustering
• Meanwhile, have fun and try it out!

48. ©Continuent 2013
Tungsten University Sessions
48
• Introduction to Continuent Tungsten 2.0 (May
16 and May 21)
• Load a Vertica Data Warehouse with MySQL
Data (May 30 and June 4)
Send any feedback to: tu@continuent.com

49. ©Continuent 2013. 49
Continuent Web Page:
http://www.continuent.com
Tungsten Replicator:
http://code.google.com/p/tungsten-replicator
Our Blogs:
http://"yingclusters.blogspot.com
http://scale-out-blog.blogspot.com
http://datacharmer.blogspot.com
http://continuent-tungsten.blogspot.com
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128
Tel +1 (866) 998-3642
Fax +1 (408) 668-1009
e-mail: sales@continuent.com