Modern Data Driven Cloud Native Architecture for Digital Transformation

1. © Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
Mukund Babbar
Sr. Platform Architect
Pivotal Inc.
Data Analytics in
Digital Transformation

2. IMPLEMENTING
new methodologies to influence
the software development culture
of Silicon Valley’s most influential
Internet companies
DISCOVERING ACCELERATING
an agile, rapid iteration, test-
driven approach to software
development
the digital transformation of the
world’s largest companies with a
modern software development
methodology, advanced analytics
and modern cloud platform
the world’s largest companies
into cloud native software
companies including one-third
of the Fortune 100
TRANSFORMING
Pivotal
2000s1990s 2013 Now
Founded

3. Anatomy of a Modern Digital Business
BUSINESS DRIVERS • New systems of engagement
• New business models
• Internet of Things
“Events, people or things
happening now and affect the
outcome”
ENABLED BY The Private / Public Cloud
“Infinite, inexpensive compute storage”
CAPABILITIES NEEDED
Compelling, Unique User Experience/Model
Agile Product Development Culture
Data Analytics
Platform
Existing Systems

4. Millions of “trip”
events each day
globally
400+ billion viewing-
related events per
day
Five billion
training data
points for Price
Tip feature
Disrupters Use a LOT of Data

5. Data manifests as new app features
“We’ve found that when a
host selects a price that’s
within 5% of their tip,
they’re nearly 4 times
more likely to get booked”
“The importance of
accuracy and efficiency
[…], will continue to rise
as we expand and
improve products like
uberPOOL and beyond.”
“Over 75% of what
people watch come from
our recommendations”

6. © 2017 Pivotal Software, Inc. All rights reserved.
Open Source Innovation
Agile Methodologies
Cultural Transformation
Massively Parallel
Processing
Data Science and
Machine Learning
Loosely-coupled
Data Microservices
Cloud Native
Continuous
Delivery
So How are they accomplishing this?

7. “Companies need to learn how to influence
people or things in the act of doing
something and affect the outcome”
PAUL MARITZ
EXECUTIVE CHAIRMAN, PIVOTAL

8. Analytical Workflows in Smart Apps
Be Predictive
REQUIRES:
Machine Learning at Scale
• Demand Forecasting
• Preventive Maintenance
• Customs Delays
REQUIRES:
Large Scale Optimization
• Routing
• Maintenance Scheduling
• Aircraft Load Plans
Optimize
Decisions
REQUIRES:
Simple interface (applications)
for analysts to interact with the
optimization algorithms
• Querying Solution
• Disruption Management
• User Preference Capture
Enable What-if
Scenarios

9. Becoming a Data-Driven Company Is Hard
Legacy data infrastructure
can’t scale or cope
Affecting outcomes requires
precision and speed
Companies in the top third of their industry in the use of data-driven
decision making are, on average, 5% more productive and 6% more
profitable than their competitors 1
1http://hbr.org/2012/10/big-data-the-management-revolution/ar/2
Turning insight into
impact is mysterious

10. Analytics within Digital Transformation
Cannot Be
Accomplished
In A Silo

11. Large Enterprises look like this...
● Silo’d and aging database systems
● Spaghetti data pipelines
● Expensive, proprietary data
management systems
● Lack of structured platforms for
continuous software delivery
● Monolithic application architectures
● Batch-oriented data integration
● Limited operationalization of analytics
● Proprietary systems

12. Stream + Batch Processing
Programming + Operating Model
Cloud-Native Platform
Microservices FrameworkPlatform Runtime
Cheap
HW
MPP
OSS
Microservices and Polyglot Persistence
IMDG
K/V Store
Relational DB
Data Science &
Machine Learning
Cloud Infrastructure
Modern Cloud Native Data Architecture

13. Modern Cloud Native Data Architecture - Technology
NeedsOSS
Fast Ingest
• Pipelines to consume
streaming and batch data from
various endpoints
• Ability to do real time scoring
Speed/Serving Layer
• In-memory data grids for
real-time and aggregated
data sets
• ms latency access
Analytics Platform
• Commodity hardware
• Ability to support scale out compute
• Parallel execution of SQL, R, Python, and other languages
• Ability to run machine learning at scale
Platform for Scalable
Web/Mobile Apps
• Infrastructure Automation
• Container Orchestration
• 12 factor apps
• Data Services that scale
• Develop, run and manage
Web/mobile applications
without the complexity of
building and maintaining the
infrastructure

14. Recipe for Smart Apps
Scale-out
analytic
database
Model as
API
Cloud Native
Application
Platform
Data
Sources
0 5
Recipe for
Smart Apps

15. A Real-World ExampleAPP
Data
Microservice

16. A Real-World Example
App
Development
Data
analytics
Cloud-native
App platform
Data
Science
& Model
building
APP
Data
Microservice

17. A Real-World Example
App
Development
Data
analytics
Cloud-native
App platform
Data
Science
& Model
building
Data
Microservice
APP
Must support scale-out
query processing
Must deliver as an API
Must embrace agile development,
focus on outcomes
Must support
microservices, agile dev, and
connect to big data analytics

18. Reference Architecture
Data Sources
ms-Latency Data API
Analytical Platform
Update with
findings
Client application
Machine learning
model
Evaluate Train
Real-time
data streams

19. Can you...
Ask any of question of your data?
Modify data pipelines and add processing steps frequently?
Consume a wide range of data sources and protocols?
Release new features in minutes, multiple times a day?
Support a microservices architecture?
Update algorithms and models daily?

20. Case Studies

21. MOTIVE
Maximize Value from Industrial Assets by Creating a
Strong Software Foundation for the Industrial Internet
BENEFITS
● Intelligent Machines
● Real-time safety
● Transmitting Valuable data
● Optimizing Operations
● Empowering Technicians
● Efficient Maintenance
● Huge Savings
Industrial Internet

22. I’m not seeing any alarms…
Why are our customers
having poor service?
Network Outage
CHALLENGES
● Poor network quality and frequent outages
● Little visibility on what’s happening on the
network
● Takes long to detect root cause of network
failure
BENEFITS
● Real-time data insights on cell phone network
quality KPIs
● Data Science models for improving the network
quality by predicting network failures

23. You’re the third person I’ve
been handed off to!
Can’t anyone help me?
Smart IVR
● Real time ingestion of customer data from
multiple source systems
● Intent Prediction Model deployed and updated
every couple of hours
BENEFITS
● Better routing of calls based on past
engagements and customer intent
● Reduced calls to care agents
● Better customer experience
Australian Telco

24. You’ve charged me extra.
I need to know details or give
me refund
Bill Shock
● Real time ingestion of network probes data
6 Billion events a day.
● Hourly aggregate of customer data usage
pattern based on application and location
BENEFITS
● Visibility of customer data usage pattern to care
agents
● Huge reduction in credits given to customers
● Better customer experience
Australian Telco

25. Predictive Car Maintenance
MOTIVE
● Manage business more efficiently
● Ability to detect failure at the earliest stage
● Handle sensitive data from sensors and other
applications
● Get car and IT development teams in sync
BENEFITS
● Data Science models to more accurately predict
part failure
● Optimize part repair and replacement schedule
● Better customer experience

26. “We chose pivotal because we believe it provides a 360-
degree view of the process. From a data science and data
technology perspective, it means delivering best-in-class data
technologies and enabling them on their platform.”

27. BECOMING A DATA DRIVEN COMPANY…
IS NOT Just about deploying Hadoop
OR How many Data Scientists you have

28. IT’S ALL ABOUT
HOW YOU
OPERATIONALIZE
YOUR INSIGHTS

29. Transforming How The World Builds Software
© Copyright 2017 Pivotal Software, Inc. All rights Reserved.