Presentation given at QCon SP. What does DevOps and Continuous Delivery mean for data? How does polyglot persistence fit into a microservices architecture? What are Data Microservices?

1. A NEW PLATFORM FOR A NEW ERA

2. © Copyright 2014 Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved.
DevOps, CD & [Data] Microservices
2
Breaking the data monolith
Fred Melo
@fredmelo_br

3. © Copyright 2014 Pivotal. All rights reserved.
$
Business
DevelopmentQA
Operations
Customer
WHY?

4. © Copyright 2014 Pivotal. All rights reserved.
$
WHY?
Concept Cash

5. © Copyright 2014 Pivotal. All rights reserved.
How?

6. © Copyright 2014 Pivotal. All rights reserved.

7. © Copyright 2014 Pivotal. All rights reserved.
User Mgmt
Order Mgmt
Shipping
Inventory
Payment
Payment
Order
Mgmt
Inventory
User
Mgmt
Shipping
Contract
Contract
Contract
Contract
Contract
Conway’s Law and Microservices
Architecture x Development teams

8. © Copyright 2014 Pivotal. All rights reserved.
Deploy
>cf push
Cloud
Controller
BBS
(message bus)
Router
App Deployed
Order
Mgmt
Contract

9. © Copyright 2014 Pivotal. All rights reserved.
App App App
App App
App
Scale
AppApp
App
Auto-heal
Migrate
Pivotal Cloud Foundry

10. © Copyright 2014 Pivotal. All rights reserved.
App App App
App App
App
Scale
AppApp
App
Auto-heal
Migrate
Logging
Metrics
Platform
Services
Pivotal Cloud Foundry

11. © Copyright 2014 Pivotal. All rights reserved.
App
Stateless
Ephemeral disk
Short-lived context
“12 Factor"
App App App
App App
AppAppApp
App
Pivotal Cloud Foundry

12. © Copyright 2014 Pivotal. All rights reserved.
Statefull
Disk-based storage
Long-lived context
Data Service
???????
App App App
App App
AppAppApp
App
Pivotal Cloud Foundry

13. © Copyright 2014 Pivotal. All rights reserved.
Scale
Re-deploy
Migrate
Distribute
Upgrade Update
Pivotal Cloud Foundry

14. © Copyright 2014 Pivotal. All rights reserved.
User Mgmt
Order Mgmt
Shipping
Inventory
Payment
Payment
Order
Mgmt
Inventory
User
Mgmt
Shipping
Contract
Contract
Contract
Contract
Contract
Polyglot Persistence
"RDBMs is the king" “The right tool for each job"
One Size Fits All

15. © Copyright 2014 Pivotal. All rights reserved.
Polyglot Persistence
RDBMs only IMDGs, K-V Stores, Object stores, …
One Size Fits All
Scale-up only
Grid-based HADisk-based HA
Scale-out
Specific hardware Containerized

16. © Copyright 2014 Pivotal. All rights reserved.
Scale
Re-deploy
Migrate
Distribute
Upgrade Update
Order
Mgmt
Contract
User
Mgmt
Contract
Pivotal Cloud Foundry

17. © Copyright 2014 Pivotal. All rights reserved.
Order
Mgmt
Contract
User
Mgmt
Contract
Inventory
Contract
Payments
Contract
Shipping
Contract
SQL?
Analytics?
BI?
Reporting?

18. © Copyright 2014 Pivotal. All rights reserved.
Order
Mgmt
Contract
User
Mgmt
Contract
Inventory
Contract
Payments
Contract
Shipping
Contract
Analytical System
DeliveryFrequency
Degreeofautomation
Manual &
Tooling
Transactions
Analytics
Data Scientist
MDM
Analytical Team

19. © Copyright 2014 Pivotal. All rights reserved.
How about Data Pipelines?
Anatomy of a Data Pipeline
Source Destination
Processing
Step
Processing
Step
Processing
Step
Processing
Step
Data Data
Monolith

20. © Copyright 2014 Pivotal. All rights reserved.
Data Pipeline
Scale
Re-deploy
Upgrade
Migrate
Distribute
Update
Source Destination
Processing
Step
Processing
Step
Processing
Step
Processing
Step

21. © Copyright 2014 Pivotal. All rights reserved.
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
Binding Abstraction Layer
Transport Options
Data Pipeline
Spring Cloud Stream
Breaking the Data Pipeline Monolith

22. © Copyright 2014 Pivotal. All rights reserved.
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
Transport Options
Spring Cloud Stream
Apache
YARN
Breaking the Data Pipeline Monolith

23. © Copyright 2014 Pivotal. All rights reserved.
From Microservices to Data Microservices
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
Cloud Services
Transport Choices
Spring Cloud Stream
Pivotal Cloud Foundry
Scale
Auto-heal
Logging
Metrics
Registry and
Discovery
Circuit
Breaker
Distributed
Config

24. © Copyright 2014 Pivotal. All rights reserved.
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
Pivotal Cloud Foundry
Services
Data Microservices
Spring Cloud
Data Flow
Data Microservices
Orchestration
Pivotal Cloud Foundry

25. © Copyright 2014 Pivotal. All rights reserved.
SpringBoot
app
Contract
SpringBoot
app
Contract
Spring Cloud
Data Flow
JSON
Filter
Transform Aggregate
Custom
HTTP
Sample pipeline
Pivotal Cloud Foundry
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
SpringBoot
app
Contract
Deploy

26. © Copyright 2014 Pivotal. All rights reserved.
Your own
infrastructure
Pivotal Cloud Foundry
Microservices
Spring Cloud
Services
GemFire
Data Microservices
Spring Cloud
Data Flow
Redis
MySQL
Greenplum
Polyglot persistence
PostgreSQL
HTAP

27. A NEW PLATFORM FOR A NEW ERA