1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0
Running Data Platforms
Like Products
June 14, 2018
Mike Koleno, CTO, Solstice, @_koleno
Dormain Drewitz, Sr Director, Pivotal Software, @DormainDrewitz

2. Cover w/ Image
Industry Context
From centralized DBA teams to
diffused database responsibilities…
From monolithic RDBMS to
distributed, NoSQL datastores...
From request/response paradigms
to event-driven architectures…
“Whatever the nascent
reconsideration and
reevaluation of database
technologies, processes
and procedures comes
to be called [...] change
is coming.”
- Steve O’Grady, RedMonk
http://redmonk.com/sogrady/2018/01/09/whither-the-dba/

3. What You Should Take Away from Today
Understanding the
product mindset for
your internal teams
(particularly data folks)
Get some ideas about
what your “read” data
products might be and
how to build them
Get some ideas about
what your “write” data
products might be and
how to build them

4. Part of This Complete Source of Long Lead Times
PMO
Analyst
Architect
Lead Developer
Development Team
Operations Team
QA
DBA
Task 1
Task 2
Task 3
Task 4
Task 6
Task 5
Task 7
Task 9
Task 8
Adapted from: https://content.pivotal.io/white-papers/crossing-the-value-stream-improving-development-with-pivotal-and-cloud-foundry

5. Part of This Complete Source of Long Lead Times
PMO
Analyst
Architect
Lead Developer
Development Team
Operations Team
QA
DBA
Task 1
Task 2
Task 3
Task 4
Task 6
Task 5
Task 7
Task 9
Task 8
DBA

6. Not one, but TWO ways to stop/delay delivery to prod

7. Not one, but TWO ways to stop/delay delivery to prod

8. Data in SOAP-land
App1:
“I need
some
data”
App2:
“I need
some
data”
Mainframe
ESB

9. Data in SOAP-land
App1:
“I need
some
data”
App2:
“I need
some
data”
Mainframe
ESB
Problem A: The values don’t match because
App1 interpreted the data differently from App2

10. DBAs: Stewards of Extracts, Satellite Reports & Dashboards

11. Master Data Management Issues
App1
App2 DB from some
acquisition
DB
Contains user
email
addresses
Also contains
user email
addresses
User has
accounts in
both apps with
different email
addresses

12. Master Data Management Issues
App1
App2 DB from some
acquisition
DB
Contains user
email
addresses
Also contains
user email
addresses
User has
accounts in
both apps with
different email
addresses
Problem B: App3 comes along and doesn’t
know which email address to use

13. What if “data” was an internal
product?
Hypothesis

14. Cover w/ Image
Explain “internal product”?
● Clearly identified customer(s)
○ E.g. development teams,
infrastructure teams,
leadership
● Product pride
○ E. g. Branding, showcasing
customer success
https://content.pivotal.io/blog/product-all-the-things

15. Two Types of Data “Products”
Reads
As a developer, I need to use data.
To read-data product team: Make it
simple, easy, and consistent to get
the right data into my application.
Writes
As a developer, I need to write
data somewhere.
To write-data product team: Make
it easy for to do the right thing,
while keeping developers
productive

16. Yes, this aligns with CQRS
Source: http://www.kennybastani.com/2017/01/building-event-driven-microservices.html
“... use different models
to update and read
domain data”

17. Getting Data
Data Products

18. Examples of data “products”: Reads

19. Massive data engineering abstracted for developers
Location data for over 100mn
places
(Black box)
Places API
Geocoding
API
Geolocation
API
Time Zone
API
Product team domain

20. HCSC: Rebuilding SOAP services
https://content.pivotal.io/geode-at-springone-platform-2017/high-performance-cloud-native-apis-using-apache-geode-anna-jung-paul-vermeulen

21. HCSC: Rebuilding SOAP services
https://content.pivotal.io/geode-at-springone-platform-2017/high-performance-cloud-native-apis-using-apache-geode-anna-jung-paul-vermeulen
Customer A
Product team domain
Customer B
Customer N

22. Example: Challenges with Stored Procedures
Today: A Data Free-For-All
● n applications = n ways of
getting data
● Everything requires a DBA
● No centralized data access
strategy, leading to duplicated
efforts, inconsistencies and
conflicts
● Business logic entombed in
monolithic applications’ stored
procedures
Future: Data Products
● n applications = 1 way of
getting data
● DBA rarely required
● Provides centralized point of
access to common sets of
enterprise data
● Business logic stored in API
endpoints in server-side code,
which is much easier to debug

23. Example: Challenges with Stored Procedures
Today: A Data Free-For-All
● n applications = n ways of
getting data
● Everything requires a DBA
● No centralized data access
strategy, leading to duplicated
efforts, inconsistencies and
conflicts
● Business logic entombed in
monolithic applications’ stored
procedures
Future: Data Products
● n applications = 1 way of
getting data
● DBA rarely required
● Provides centralized point of
access to common sets of
enterprise data
● Business logic stored in API
endpoints in server-side code,
which is much easier to debug
● Hard to debug
● Hard to add new consumers
● Heavy DBA burden
● Centralization/Rationalizatio
n of process
● CI/CD
● Run server-side apps on a
cloud-native platform
● Autonomy

24. Building the Data API Supermarket

25. Building the Data API Supermarket

26. Building the Data API Supermarket

27. Writing Data
Data Products

29. Anti-pattern: Sharing a database
● Autonomy to iterate
● Autonomy of tech stack
● Performance
● Scaling

30. Data APIs (Services)
● Microservices do not access data
layer directly
○ Except for those that implement
the data API
● A surface area to:
○ Implement access control
○ Implement throttling
○ Perform logging
○ Other policies…

31. Adding a Data API/Service layer
● Performance?
● Resilience?
● Polyglot persistence?
● Schema versioning?

32. Adding a Data API/Service layer - with cache
● Performance
○ In-memory, horizontally
scalable
● Resilience
○ Provides bulkhead
● Polyglot persistence?
● Schema versioning?

33. Cover w/ Image
Every Microservice Needs a
Cache
https://www.youtube.com/watch?v=Rzdxgx3RC0Q&t=205s
Not just for performance
● Scaling
● Autonomy
● Resilience

34. Shift the system of record safely
● Performance
○ In-memory, horizontally
scalable
● Resilience
○ Provides bulkhead
● Polyglot persistence
○ Strangler pattern to transfer
data ownership
● Schema versioning?
Recommended reading:
http://www.kennybastani.com/2016/08/strangling-legacy-microservices-
spring-cloud.html#more

35. Data API evolve in support of the evolution of the
microservice(s)
● Performance
○ In-memory, horizontally
scalable
● Resilience
○ Provides bulkhead
● Polyglot persistence
○ Strangler pattern to transfer
data ownership
● Schema versioning
○ Parallel deployments
V1
V2

36. ACH payments example: Multiple ways to move money

37. ACH payments example: Unifying payments via API product

38. CI/CD for data

39. Cover w/ Image
The Consultative DBA
● Easy access
○ NOT tickets
○ Email alias? Slack channel?
● Tooling
○ Own the path to production
● Workshoping
○ Start with the “why”
○ Listen
● Goals and Metrics
○ Metrics of effort
○ Extrinsic validation metrics

40. What You Should Take Away from Today
Understanding the
product mindset for
your internal teams
(particularly data folks)
Get some ideas about
what your “read” data
products might be and
how to build them
Get some ideas about
what your “write” data
products might be and
how to build them

41. Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.
Mike Koleno
@_koleno
Dormain Drewitz
@DormainDrewitz