Time to Talk about Data Mesh

1. It is time to talk about
DATA MESH
Distributed Data Management for
Microservices

2. MICROSERVICE’S CHALLENGE
01

3. 3
THE CHALLENGES OF MICROSERVICE DESIGN
TRANSFORM BUSINESS LOGIC AND DATA INTO MICROSERVICE ARCHITECTURE
Business flow combing,
Function and service separating
Data decoupling and fragmentation
management
◼ DDD (Domain-Driven Design)
◼ Split services by business flow
◼ EDA (Event-driven Architecture)
◼ Microservice component design
◼ Microservice horizontal scaling strategies
◼ Separating data by business flow
◼ Sharing data across domains
◼ Vast data deployment and caching strategies
◼ Data consistency(Data fragmentation management)
◼ The final data consistent
◼ Strong data consistent
◼ Absolute data consistent
CHALLENGE
DONE WELL
Business Logic
Data
CELL GATEWAY
Database
Service
Service
Analyze Decouple Aggregate
To implement the microservice architecture with
decision of design patterns, deployment and
maintenance strategies, integration of technical
components, planning and actual software
architectures.
Implement
API SERVICES/
EDGE SERVICES
COMPOSITE SERVICES/
INTEGRATION SERVICES
CORE SERVICES/
ATOMIC SERVICES
CLIENT APPLICATIONS
To implement the design and
choose an immediately available
solution to implement various
microservice design patterns and
basic architectures. Based on
stable software components and
platforms, determine service roles
and implement business logic.
Special design with stateful atomic services

4. 4
TRANSFORM TO DATABASE PER SERVICE
For Microservice Architecture
App
service
App
service
App
service
DATABASE
App
service
App
service
App
service
Database per service
Shared Database
How?
Risk?
Performance?
Plan?

5. 5
PROBLEMS WITH DATABASE PER SERVICE
WHEN WE IMPLEMENT IT
App
service
App
service
App
service
Round-trips Problem
Busy
App
service
App
service
App
service
Anti-Pattern
App
service
App
service
App
service
Confusion
OR
OR
How do we solve these problems ?

6. 6
PROVISION ONCE, SNAPSHOT MANY
CACHEING FOR MULTIPLE SNAPSHOTS
CLIENT
APPLICATIONS
Service API
SNAPSHOT
Service API
DATA FLOW
Source
VIRTIAL TABLE
VIRTUAL TABLE BUSINESS
BUSINESS
100M Records
100M Records
100M Records
Service
Owner
According to requirements of business task, selecting
database, table and fields to make a copy into a
virtual table.
Update
Database per service
1
Provision
2
3
Program A
Program B

7. 7
The scale of the supply pipelines and
the degree of deployment efficiency
THE KEY TO THE SMOOTH IMPLEMENTATION OF
MICROSERVICE ARCHITECTURE

8. 8
MANAGE DATA SUPPLY CHAIN WITH DATA PLATFORM
FAST TO BUILD DATA SUPPLY CHAIN FOR APPLICATIONS
CLIENT
APPLICATIONS
Service API
SNAPSHOT
Service API
DATA PLATFORM
Source
VIRTIAL TABLE
VIRTUAL TABLE BUSINESS
BUSINESS
DATA PROVISION AND CACHING
100M Records
100M Records
100M Records
Service
Owner
According to requirements of business task, selecting
database, table and fields to make a copy into a
virtual table.
Update
Database per service
1
Provision
2
4
Caching
3

9. 9
App
App
APPLICATION-DRIVEN DATA SUPPLY
DISTRIBUTED DATA INTEGRATION
VIRTUAL TABLE
App
App
App
App
DATA MESH
VIRTUAL TABLE
VIRTUAL TABLE
VIRTUAL TABLE
VIRTUAL TABLE
VIRTUAL TABLE
DATABASE
DATA SOURCE
CUSTOMIZATION
DATA SOURCE
FILES
DATA SOURCE

10. 10
On-Premise
APPLICATIONS
WHY NOT THE TRADITIONAL SOLUTIONS?
CENTRALIZED DATA MANAGEMENT CANNOT FIT TO MICROSERVICE ARCHITECTURE
Public
FILES
DATABASE
CUSTOMIZATION
DATA SOURCE
DATA SOURCE
DATA SOURCE
Cloud
Private Cloud
Cloud
VIRTUAL TABLE
Centralized
Data Process
VIRTUAL TABLE
VIRTUAL TABLE
Traditional ETL, Data Virtualization and Data Warehouse encounter problems with centralized system design.
ETL
Batch
Polling

11. 11
On-Premise
APPLICATIONS
DATA MESH CONCEPT
DISTRIBUTED DATA MANAGEMENT WITH DATA MESH
Public
FILES
DATABASE
DATA SOURCE
CUSTOMIZATION
DATA SOURCE
DATA SOURCE
Cloud
Private Cloud
Cloud
VIRTUAL TABLE
VIRTUAL TABLE
VIRTUAL TABLE
DATA MESH
Data Mesh focusing on supplying data to applications as fast as possible with distributed system design.
Provision
Provision
Provision

12. HOW TO BUILD
DATA MESH INFRASTRUCTURE
02

13. 13
Data Fabric
HOW TO BUILD DATA MESH
INFRASTRUCTURE
Data Virtualization and Relay
Data Caching
Presentation
Deployment and Management
Data Source
Data Mesh
Data Provision
Infrastructure
Services
Business
◼ Provision and manage data with pipelines
◼ Establish data relay mechanism（Caching）
◼ Rapidly deployment and management data pipeline clusters（Container）
◼ Manage vast multiplexed data channels and pipelines (Multiplexer)
Objective
Applications
Legacy and others
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Last
Mile
Infra needs to be able to
quickly deploy all data
management
mechanisms based on
application requirements.

14. 14
BUILD DATA NODE BASED ON KUBERNETES
YAML WAY TO DEPLOY DATA NODE AND PIPELINE INFRSTRUCTURE
Data Mesh Controller
YAML
Relay
Database
Relay
Database
Relay
Database
YAML YAML
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Service
BUSINESS
Container
Platform
Orchestrator
Kubernetes
◼ Declarative pipeline deployment with
integrated container platform
◼ Meet the demand for a large number
of microservice data supply
◼ Manage and maintain a large number
of provision pipelines
Objective
DATA PLATFORM
Hybrid Clouds

15. 15
Using YAML
SCENARIO: Starting from event data provision, caching,
building virtual table, and quickly generate APIs.
No need to write a single line of program, build the entire framework in a few minutes, from Oracle to MongoDB and generate usable data API
SERVICE
CDC
Protocol
Data Platform
Kubernetes
cluster VM / Container
Presenter
component
Event changed
Intercept database change events
The data source can be an application or a database
Select the required data fields
Other
applications
Customized
Query from cached data
Deploy a pipeline
mechanisms
Caching
API
Auto generated
Imported from
templates
1
Apply API settings
2
3
CLIENT APPLICATIONS
DATA SOURCE
INFRA AND PLATFORM
DATA PROVISION AND CACHING
11g R2

16. 16
DATABASE
RESULTS
Legacy
Program
USE CASE 1: Improve the efficiency of data interface to the
legacy system SOLVE PROBLEM THAT LEGACY SYSTEM BRINGS ON LOW LATENCY AND INTEGRATION ISSUES
DATABASE
DATA SOURCE
Query
Storage
Program
DATABASE
RESULTS
Polling
Export
Insert
File
DATABASE
DATA SOURCE
DATA MESH
Event Update
Real-time and Low Latency
Provision and Caching

17. 17
B
Batch
DATABASE
RESULTS
USE CASE 2: Solve the inefficiency and performance impact of
various batch processing SOLVE PROBLEM THAT INTEGRATION ISSUES
DATABASE
DATA SOURCE
Query
A
Program
DATABASE
RESULTS
Insert
DATABASE
DATA SOURCE
DATA MESH NODE
Provision
Supply to Multiple Applications without Performance Impact
A
Service
DATA MESH PIPELINE Push
DATA MESH PIPELINE B
Service
Push
Parallel
Caching
A
Batch
Query and Extract
B
Program
A
B
Subscribe
Subscribe
Query and Extract
Insert

18. 18
Program
SNAPSHOT
USE CASE 3: High-speed parallel processing of the aggregation
and correlation of multiple data sources USE CASE: INTEGRATION WITH MULTIPLE SOURCES
IT
DATA SOURCES
OT
DATA SOURCES
Program Output
Aggregate and Transform
Round-Trip Problem
Round-Trip Problem
IT
DATA SOURCES
OT
DATA SOURCES
DATA MESH NODE
Event
DATA MESH NODE
Event
DATA MESH PIPELINE Present
Aggregate
Snapshot
Snapshot
Output
Transform
Aggregation in Parallel for High Performance
Parallel
Caching
Caching

19. 19
Program
SNAPSHOT
USE CASE 4: Integration of hybrid cloud, cross-site and cross-
environment USE CASE: INTEGRATION FOR HYBRID CLOUD
IT
DATA SOURCES
OT
DATA SOURCES
Program Output
Aggregate and Transform
Round-Trip Problem
Round-Trip Problem
IT
DATA SOURCES
OT
DATA SOURCES
DATA MESH NODE
Event
DATA MESH NODE
Event
DATA MESH NODE Present
Aggregate
Snapshot
Snapshot
Output
Transform
Aggregation Across Clouds With
Parallel
Caching
Caching
On-Premise
Public Cloud
External Site
On-Premise
Public Cloud
External Site

20. ＤＥＭＯ
0３

21. About Brobridge
Brobridge refers to Broad Bridge Co., Ltd. (a limited company formed under the laws of Taiwan). Please refer to www.brobridge.com for a detailed description.
Brobridge Co., Ltd. provides information technology consulting, application development, IT infrastructure design, consulting services and system
maintenance services to listed and unlisted clients in various industries. Brobridge Co., Ltd. recruits the most professionals in the industry, is committed to the
pursuit of excellence and setting an example.
Brobridge Co., Ltd.
Brobridge refers to Brobridge Co., Ltd.. Brobridge Co., Ltd. enjoys a good reputation in the industry for its excellent customer service, excellent talents, perfect
training and rigorous development technology. Through the resources of Brobridge Co., Ltd., we provide a full range of services for customer information
systems, including virtual desktop systems, multi-cloud integration, IT infrastructure planning and design, and advanced cutting-edge program development.
This publication is compiled based on general information and is for readers' reference only. Brobridge Co., Ltd. is not deemed to provide professional
advice or services to anyone because of this publication. Any individual of Brobridge Co., Ltd. shall not be held responsible for any loss caused by reliance
on this publication.
© 2016 Broad Bridge Co., Ltd. All rights reserved.
寬橋
THANK YOU!