Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle
âą
0 gostouâą552 visualizaçÔes
Gittigidiyor, a subsidiary of eBay, needed to replicate data in real time from their MySQL database to an Oracle database to power their data warehouse. Continuent Tungsten was used to provide heterogeneous replication between the databases. The schema was translated from MySQL to Oracle using ddlscan. Initial data was exported and loaded into Oracle using parallel apply. Ongoing real-time replication now occurs between MySQL and Oracle using Tungsten Replicator with custom filters to handle data type translations.
Recomendados
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle
Semelhante a Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle (20)
Mais de Continuent
Mais de Continuent (20)
Ăltimo
Ăltimo (20)
Tungsten Use Case: How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle
- 1. How Gittigidiyor (a subsidiary of eBay) Replicates Data In Real Time From MySQL To Oracle Levent Kurt, Site Operations Manager, Gittigidiyor Giuseppe Maxia, Director of Quality Assurance, Continuent Linas Virbalas, Senior Software Engineer, Continuent ©Continuent 2013.
- 2. Introducing Continuent ⹠The leading provider of clustering and replication for open source DBMS ⹠Our Product: Continuent Tungsten ⹠Clustering - Commercial-grade HA, performance scaling and data management for MySQL ⹠Replication - Flexible, high-performance data movement ©Continuent 2013 2
- 3. A Review of Tungsten Replicator ©Continuent 2013 3
- 4. Tungsten Replicator Overview Master Download transactions via network DBMS Logs Slave Replicator (Transactions + Metadata) Replicator Apply using JDBC ©Continuent 2013 THL THL (Transactions + Metadata) 4
- 5. Master Replication Service Pipeline Stage Extract Filter Stage Apply Extract Filter Apply tcp/ip Binlog MySQL Master ©Continuent 2013 Transaction History Log In-Memory Queue 5 Slave Replicators
- 6. Slave Replication Service Pipeline Stage Apply Extract Filter Stage Apply Extract Filter Apply tcp/ip Extract Filter Stage Master Replicator Transaction History Log ©Continuent 2013 In-Memory Queue 6 Slave DBMS
- 9. Steps to Heterogeneous Replication 1. Prepare (translate) schema for the slave DBMS 2. Set up replication ddlscan tpm 3. Provision initial data⚠⚠MySQL Blackhole Parallel Apply 4. Continue real-time replication Tungsten Replicator ©Continuent 2013
- 10. Replicating from MySQL to Oracle ©Continuent 2013
- 11. Use Case: Real-Time Sales/Order Fulfillment Web-based Sales ~1M Xacts/ Day ©Continuent 2013 Purchase Ordering Real-Time Replication 3M Order Items
- 12. 1. Translating schema for the slave ©Continuent 2013
- 13. Translating Schema ⹠Goal - set up heterogeneous replication Replication ©Continuent 2013 13
- 14. Translating Schema ⹠Beginning - how to convert tables? empty Tables ©Continuent 2013 14
- 15. Translating Schema âą Beginning - how to convert tables? empty âąData types? âąColumn lengths? âąNaming conventions? âąReserved words? Tables ©Continuent 2013 15
- 16. Translating Schema ⹠ddlscan automates these conversions ddlscan Tables ©Continuent 2013 16 empty
- 17. ddlscan ⹠⹠⹠⹠©Continuent 2013 Part of Tungsten Replicator, GPL v2 Translates schema with replication in mind Provides errors and warnings Can rename schema/tables/columns 17
- 18. Usage (MySQL to Oracle Example) $ cd tungsten-replicator/bin ! $ ./ddlscan -db test -template ddl-mysql-oracle.vm -user tungsten -pass secret ©Continuent 2013 18
- 19. Translating Schema ⹠ddlscan looks into source schema empty Tables ©Continuent 2013 19
- 20. Translating Schema ⹠ddlscan translates and renders DDL commands empty Tables oracle-ddl.sql ©Continuent 2013 20
- 21. Result of ddlscan CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */ ( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0) /* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */ ! /* ERROR: table must have a primary key! */ ); ! CREATE TABLE talks ... ©Continuent 2013 21
- 22. Translating Schema âą You run resulting SQL ïŹle on Oracle empty Tables oracle-ddl.sql ©Continuent 2013 22
- 23. Translating Schema ⹠Tables are ready! Tables ©Continuent 2013 Translated tables (empty) 23
- 24. 2. Set Up Replication ©Continuent 2013
- 25. Replicator Installation ./tools/tpm configure my_ora --enable-heterogenous-service=true --install-directory=/opt/mysql_to_ora/continuent --members=alpha,bravo --master=alpha ! ./tools/tpm configure my_ora --hosts=alpha --replication-user=tungsten --replication-password=secret ! ./tools/tpm configure my_ora --hosts=bravo --replication-user=tungsten_frommysql --replication-password=secret --datasource-type=oracle --datasource-oracle-service=ORCL --svc-applier-filters=dropstatementdata ! ./tools/tpm install my_ora ! ./tools/tpm start my_ora ! ©Continuent 2013 25
- 26. Replication Under the Hood MySQLExtractor OracleApplier Tungsten Master Replicator Service frommysql MySQL Binlog binlog_format=row ©Continuent 2013 Tungsten Slave Replicator Service frommysql MySQLExtractor Special Filters ⹠Transform ENUM to string ⹠Transform SET to string Special Filters ⹠Drop DDL ⹠Map names to upper case ⹠Rename too long or reserved objects ⹠Optimize updates to remove unchanged columns ⹠Ignore extra tables 26
- 27. 3. Provisioning an Oracle slave with auditing trail ©Continuent 2013
- 28. The Requirements ⹠Export data from MySQL master to the Oracle slave before starting replication ⹠The Oracle slave should also take trace of the changes ⹠Convert the MySQL tables to stream of raw data in THL format ⹠Data is about 400 GB ©Continuent 2013 28
- 29. The Practical Steps 1. Export data from master 2. Keep track of replication position 3. Start the replicator from the sandbox 4. Load data into slave with parallel apply 5. Switch replicator to real master 6. Dismiss the sandbox 7. Replication from MySQL to Oracle live ©Continuent 2013 29
- 30. Provisioning (1) MySQL Oracle master slave table structure table structure table data table structure table data table data ©Continuent 2013 30
- 31. Provisioning (2) MySQL Oracle master slave ddlscan table structure table structure table data table structure table data table data ©Continuent 2013 31 table structure table structure table structure
- 32. Provisioning (3) MySQL master table structure table structure table data table structure table data table data Oracle MySQL sandbox table structure table structure table structure CONVERT to BLACKHOLE ©Continuent 2013 32 slave
- 33. Provisioning (4) not replicating MySQL master MySQL sandbox table structure table structure table data table structure table data table data backup ©Continuent 2013 33 Oracle slave
- 35. How Parallel Apply Works Pipeline Stage Stage Filter Extract Apply (Shard) Stage Extract/Filter/Apply Extract/Filter/Apply Extract/Filter/Apply Extract/Filter/ Apply Parallel Queue Master DBMS ©Continuent 2013 Transaction History Log Provisioning Mutlitenant source Slave DBMS
- 37. Provisioning (6) MySQL MySQL sandbox master table structure table structure table data table structure table data table data Oracle slave tungsten replicator bin logs THL ©Continuent 2013 37 tungsten replicator
- 38. Gittigidiyor (subsidiary of eBay) architecture and needs ©Continuent 2013
- 39. ⹠Gittigidiyor ⹠⹠⹠⹠⹠⹠⹠⹠A subsidiary of eBay Founded in 2001, joined the eBay family in June 2011 Most widely-used e-shopping site in Turkey More than 9 million registered users Over 900 thousand daily visitors and 5 million listings at any given time. 700 thousand sellers all around Turkey Selling 1 item every 3 seconds ©Continuent 2013 1
- 40. Gittigidiyor Datawarehouse The Problem Replication ! ⹠⹠⹠⹠⹠400 GB of data 450 tables over 150 million rows 20GB binlog/day MySQL inefficiency for large data handling ⹠Migration to Oracle ⹠Need to keep auditing trail ⹠Tried Golden Gate, but ⹠technical problems ⹠too expensive ©Continuent 2013 Master CDC ETL DWH 40
- 41. Tungsten Solution ! ⹠DDL generation ⹠Replication to Oracle master ⹠Handling data correctly ⹠CDC generation ⹠Fail-safe Rep licat Master ion Tungsten Replication Replicator Tungsten Filters ETL ORA Master ORA CDC ©Continuent 2013 41 ORA DWH
- 42. The steps we took ! 1. Proof of Concept a. replicated 5 tables b. created the DDL Generator. 2. Set up the replication to Oracle Master 3. Set up filters for Oracle CDC generation. 4. Dumped all data in MySQL Master and MySQL CDC to Oracle equivalents. Rep licat Master ion Tungsten Replication Replicator Tungsten Filters ETL ORA Master ORA CDC ©Continuent 2013 42 ORA DWH
- 43. MySQL to Oracle Demo ©Continuent 2013
- 44. Support for Tungsten Replicator ⹠Community support is available on project mailing list ⹠Continuent offers implementation services and annual 24/7 support subscriptions ⹠For high availability or disaster recovery needs, check out Continuent Tungsten clustering ⹠Meanwhile, have fun and try it out! ©Continuent 2013 44
- 45. 560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com Continuent Web Page: http://www.continuent.com ! Tungsten Replicator: http://tungsten-replicator.org ©Continuent 2013 45