Find out more about our journey of migrating to MongoDB after using Oracle for our hotel search database for over ten years. - How did we solve the synchronization problem with the Master Database? - How to get fast search results (even with massive write operations)? - How other issues were solved

1. Our journey of Migration from the legacy datastore to
MongoDB
June 17, 2019
Akihiro Tatematsu
Data Management Group.
Rakuten, Inc.

2. 2
About me
2010-
• Oracle DBA
Exadata, RAC, TimesTen, etc
2016-
• NoSQL DBA
MongoDB, Elasticsearch, Redis, etc
Akihiro Tatematsu

3. 3
Introduction
Selection of Next Datastore
Optimization
Summary
Introduction
Selection of Next Datastore
Optimization
Summary

4. 4
About Rakuten

5. 5
About Rakuten

6. 6
About Rakuten
Total Global GTV (JPY tn)
Rakuten (2019.05.14) https://global.rakuten.com/corp/investors/documents/results/2018.html

7. 7
About Rakuten Travel
Rakuten Travel (2019.05.14) https://travel.rakuten.com/

8. 8
MongoDB in Rakuten Travel
Usage service
19services
Total Number of operations
260k/sec
Number of mongod
111nodes
MongoDB for Search
15nodes

9. 9
Necessity of changing the data store for the Search
Serviceability
License
Cost
Scalability
Data Sync
Labor
Cost

10. 10
Introduction
Selection of Next Datastore
Optimization
Summary

11. 11
Our Search system
We used TimesTen as a cache server in 2009
Search Engine
Front Service
Search Engine
Search Engine
Push the same data each cluster
Oracle TimesTen
Oracle

12. 12
Traffic Analysis and Forecast
License Cost No Horizontal Scale-Out Duplicated Data
2013 2014 2015 2016 2017
Search Query (TPS)
2013 2014 2015 2016 2017
Data Size (GB)

13. 13
Verifications by POC (Proof of Concept)
Low
Latency
Flexibility
Cost
Efficiency
SLA 99.95%
Horizontal
Scale Out
Trouble
Shooting
Fault
Tolerance
MongoDB
Oracle
Coherence
Cassandra
Redis
MEMSQL

14. 14
Movement to
Microservice
Architecture
Improvement of
Applications
Performance
Improvement of
User Interface
New challenge project with MongoDB
Develop search APIs
Synchronize Data from Oracle to MongoDB

15. 15
Data Synchronization from Oracle to MongoDB
How we synchronize data into MongoDB
Oracle Kafka Consumers MongoDB
Sync Tables High throughput
Low latency
Scalability
Centralized
Real-time
Operate hundreds of Instances
Write 500 million messages a day
Avg. 7ms per 10 message

16. 16
Let’s use MongoDB
Do you know how to use MongoDB?
We can be an expert in MongoDB

17. 17
Search MongoDB environment
Monitoring Grafana + Prometheus
Backup LVM snapshot + oplog
Log management logstash, logrotated, mtools
Enterprise function Audit, OpsManager
Config management Bitbucket + Confluence
tools
Shard
Physical
mongod
Physical
mongod
Physical
mongod
Routers
VM
mongos
VM
mongos
VM
mongos
Config Servers
VM
mongod
VM
mongod
VM
mongod
X 5
VM
mongos
API
VM
mongos
API
VM
API
VM
API

18. 18
What have we been through
2,000 QPS
Business Expected Load
8,000 QPS
Designed Load
100,000 QPS
Actual Load
1250%
3120%

19. 19
Problems with Mass Operations
170kops

20. 20
Introduction
Selection of Next Datastore
Optimization
Summary

21. 21
Search engineer s jurisdiction
Oracle Kafka Consumers MongoDB
Provider User
Mainly
Writes
Mainly
Reads

22. 22
What have we been through
Release
Retryable Writes setting
Bulk Write operation
Write Concern setting
Optimize
Over 14
Improvement

23. 23
Retryable Writes
Consumers
Primary
Secondary
Write
API services
Read
com.mongodb.MongoWriteException: not master
Secondary
Consumers
Primary
Secondary
à Primary
Write
API services
Read
Down
1 ~ 2 seconds
Secondary

24. 24
Retryable Writes
retryWrites : true
After retrying then send message to new master
Response time to write
Consumers
Primary
Secondary
à Primary
Write
API services
Read
Down
Secondary

25. 25
Bulk Write operation
db.collection.update()
MongoDB
PIC1 PIC2 PIC3
Media Table
Update sort order
Oracle
Collection 1
Collection 2db.collection.insert()

26. 26
Bulk Write operation
db.collection.bulkWrite()
db.collection.insert()
PIC1 PIC2 PIC3
Media Table
Update sort order
Oracle
Group by same collection messages
MongoDB
Collection 1
Collection 2

27. 27
Bulk Write operation
Before
After11.54ms
7.91ms
Improvement 31 %

28. 28
Write Concern settings (w Option)
w Option
The w option requests
acknowledgment that the write
operation has propagated to a
specified number of mongod
instances or to mongod instances
with specified tags.
Our initial setting
w : 1 (default)
Driver
Primary
Secondary
Secondary
write
Apply
Replication
w:0 w:1 w:2 (majority)
Response
Case of 3 nodes replica set
Apply

29. 29
Write Concern settings (w : 0)
We have to do something. Can we set w=0?
APP
DBA
If you want to take that risk, sure

30. 30
Trouble (w : 0)
Trouble
Primary nodes were downed by high load.

31. 31
Trouble (w : 0)
Driver
Primary
Secondary
Secondary
write
Apply
Response
'Dirty cache' size grew much larger
WiredTigerLAS.wt became too big
("LAS" == lookaside, a.k.a WT cache overflow.)
Apply

32. 32
Trouble (w : 0)
Dirty Bytes
150GB

33. 33
Investigation
Primary Secondary

34. 34
Write Concern settings (j : false)
journal
memory
disk
When w: <number>
default of j is false
(Acknowledgment requires
writing operation in memory)
j Option
The j option requests
acknowledgment from MongoDB that
the write operation has been written
to the on-disk journal.
Our initial setting
j : true
100ms
Writes (j:true)
Force sync
Writes (j:false)
writes
Journal Commit

35. 35
Write Concern settings (j : false)
Before After
Improvement 64%
4.96 ms 1.79 ms

36. 36
Reads operations
Operations Per Shard
25kops
Read latency
10ms

37. 37
Great new features of MongoDB 4.0
Multi-Document
ACID
Transactions
Aggregation
Pipeline Type
Conversions
Non-Blocking
Secondary Reads
40% Faster Data
Migrations
Extensions to
Change Streams
and so forth
MongoDB (2019.05.14) https://www.mongodb.com/blog/post/secondary-reads-mongodb-40
Multi-Document
ACID
Transactions
Aggregation
Pipeline Type
Conversions
Non-Blocking
Secondary Reads
40% Faster Data
Migrations
Extensions to
Change Streams
and so forth

38. 38
Upgrade to version 4
Ver3.6 Ver4.0
Operations
Read latency

39. 39
Current situation
Operations Per Shard
30kops
Writes Latency
1.5ms
Reads Latency
0.2ms
Reads Latency (max)
1.3ms

40. 40
Introduction
Selection of Next Datastore
Optimization
Summary

41. 41
Next Datastore
Oracle TimesTenOracle
Golden
Gate
Old architecture
New architecture
Oracle kafka Consumers MongoDB
Serviceability
License
Cost
Scalability
Data
Sync
Labor
Cost
MongoDB
Oracle
Coherence
Cassandra
Redis
MEMSQL

42. 42
Optimization
Operations
Writes Latency
Reads Latency
Write concern settings
w : 1, j : false
(default)
Latency
Writes : 1.5ms
Reads : 0.2ms
MongoDB version
4.0
(latest)
Retryable Writes
retryWrites=true
Bulk Write Operation

43. 43
Future outlook
DBA
APP
ORACLE
redis
Azure
Google Cloud Platform
Elasticsearch