The Modern Data Platform - How to Conquer a New World with Old Problems

1. © Copyright 2014 BMC Software, Inc. 1 — The Modern Data Platform How to Conquer a New World with Old Problems
2. © Copyright 2014 BMC Software, Inc. 2 BMC & Control-M Leader in Digital Enterprise Management 98 of Forbes Global 100 & 80%+ of Fortune 500 Workload Automation – #1 Market share – #1 Analyst Recognition – Big Data leadership – Customer satisfaction
3. © Copyright 2014 BMC Software, Inc. 3
4. © Copyright 2014 BMC Software, Inc. 4 DIGITAL WORLD WE LIVE IN A
5. © Copyright 2014 BMC Software, Inc. 5 of S&P 500 will be replaced by 2027 75%25%of businesses will lose competitive ranking due to digital incompetence by 2017 The future is here. Go digital or go extinct. © Copyright 2016 BMC Software, Inc.
6. © Copyright 2014 BMC Software, Inc. 6 Progress means building upon … not leaving behind
7. © Copyright 2014 BMC Software, Inc. 7 How will Enterprise IT manage Big Data?
8. © Copyright 2014 BMC Software, Inc. 8 © 2015 Cloudwick. All rights reserved. Confidential and Proprietary. Build, Operate & Manage Focused on Three Key Phases Assess & Plan Architect & Engineer Optimise & Scale Size, Environment & Use Cases Secure, Govern & Data Management Applications, Analytics & Visualisation
9. © Copyright 2014 BMC Software, Inc. 9 © 2015 Cloudwick. All rights reserved. Confidential and Proprietary. 7 Founded in 2010 200+ Data Engineers, Developers & Data Scientist 50k Hadoop Clusters Engineered & Operated On-Prem & Cloud 3 Global Regions Americas | EMEA | APAC 300+ Big Data & Security Enterprise Engagements 6 Strategic Partnerships Cloudwick Open Source, Big Data, Advanced Analytics & Cloud
10. © Copyright 2014 BMC Software, Inc. 10 Data Accessibility Typical Enterprise Challenges Unaffordable storage at Scale Data Ingestion / ETL Security Data Governance & Management Data Quality & Performance
12. © Copyright 2014 BMC Software, Inc. 12 Building on each stage of the Foundation Data Science ML, IoT, AI BI/Analytics Predictive & Prescriptive Reporting Transactional, Operational, Ad-hoc & Self Service Management Strategy, Govern, Quality, Platform, Multi-tenancy & Operations Consolidation Ingest, Cleanse/ETL, Store, Secure
13. © Copyright 2014 BMC Software, Inc. 13 Who holds the Key?
14. © Copyright 2014 BMC Software, Inc. 14 IT • Enabling the Answers • Reporting: Batch to Streaming • Operational foundation & management • People: Resourcing and developing • Optimise: Savings Focus • Design: Govern, Access & Secure Business • Driving the Questions • Demanding: Mining & Exploring • Immediate & Scalable • People: training & accessing • Generating: Revenue Focus • Enable: Govern, Access and secure Aligning Gaps
15. © Copyright 2014 BMC Software, Inc. 15 Prioritising Use Cases
16. © Copyright 2014 BMC Software, Inc. 16 Batch to Streaming & Beyond
17. © Copyright 2014 BMC Software, Inc. 17 STABILITY COMPLIANCE COST EFFICIENCY
18. © Copyright 2014 BMC Software, Inc. 18 Workflow Construction • Drag and Drop makes building workflows simple • Native out-of-the-box integration for traditional and big data applications and technologies • Same Look & Feel HDFS Java MapReduce Pig, Hive, Sqoop Spark Distcp Tajo, Streaming File Transfer Informatica DataStage Business Objects Cognos Oracle Sybase SQL Server SSIS PostgreSQL z/OS Linux/Unix/Windows Amazon EC2 / VMware NetBackup / TSM SAP / OEBS / Peoplesoft
19. © Copyright 2014 BMC Software, Inc. 19 AUTOMATION AS A COMPETITIVE WEAPON
20. © Copyright 2014 BMC Software, Inc. 20 Sky Q and Control-M © Copyright 2016 BMC Software, Inc.
21. © Copyright 2014 BMC Software, Inc. 21 21 Navistar’s OnCommand™ service reduces vehicle downtime by 30%  Reduced unplanned repairs by up to 30%  20M+ data records/day collected and transformed from more than 160,000 trucks  20,000 Control-M jobs daily  20% improved time savings for Hadoop engineers  Actionable data sets delivered 5x faster “We’ve just begun to tap the power of Control-M to help us use big data to enhance remote diagnostics, improve vehicle quality, and protect critical resources from unauthorized access, among other initiatives.”

Notas do Editor

Introduce myself and a few words about BMC
Big Data project is a journey
This is not a simple process - right tools, the right skills and even the right culture
Enterprise-grade implementations = complex (Data ingestion, Processing, Analytics
In this session : Cloudwick and BMC sharing experience
WE have been in this business of Digital Enterprise Management for over 30 years now
Workload Automation or Scheduling – providing end to end automation of business process across the enterprise
MF, Unix, Linux, ERP, File Transfers, Java, Informatica, Cognos, Hadoop and so on and so on
Analysts recognition
A personal story – Vietnam
Digital Transformation
Technology adoption
Customer expectations
It may be cliche, we are truly living in a digital world
Every speaker nowadays talk about the use of the mobile (flights, check-in, hotel, etc)
Online doctor appointment via video on a mobile app
Oh, and yes occasionally I will use it also for phone calls
Going digital is not an option – it’s a must to maintain and grow the business
You can see the numbers
Davos World Economic Forum – The chairman of BBVA Talking about the Digital disruption …
20,000 banks in the world and many will disappear – those that will not adopt new technologies as he called it – the Uber moment
So, let’s drop all the technologies we have been using for decades and go digital? Yeah? Really? Hmm… not quite
Anyone recognizes the city in the photo? New York, 1904 , what happened in 1904? The first subway was operated ..
But the original subway cars have been replaced, other things like plumbing, lighting, air conditioning, security have been improved as well
Today, the subway is still the heartbeat of the city, moving over 4 million people every day, and tying buildings together
New York City is a great example of building upon layers, rather than leaving them behind
Modernizing the legacy subway infrastructure and at the same time adding the newest modern architectures - like an enterprise IT environment
When going digital or specifically implementing Big Data and IOT initiatives, companies cannot drop existing technology and replace with new
Like the subway in New York, the strong foundation, is what really allows the modernization to success and flourish
So, when we talked to companies implementing Big Data we realized that the challenges they are facing …
How do I schedule my Big Data workflow? how can I identify something went wrong and fix it on time?
How do I assure my SLAs are met? ready for changes? Integrate? secure? scale?
I will talk more about it and about our scheduling solution that helps you address all of these challenges, in a couple of minutes but first, let me turn over to our partner Tina Diamond from Cloudwick who will share from her experience and of how they have been helping customers in these areas
How will Enterprises manage big data -

It is about Building a Data Platform for efficiently, optimised for scale
Enterprises Require: Security, Access and Control with optimum Performance
Scalable architecture in the platform and software
Built on a flexible infrastructure
Advanced analytics and Apps to grow the business

*I would like to introduce C
I am the MD, I joined Cloudwick, a US based OS PS in Jan 2016

Focused on OS, Big Data& Advanced Analytics, delivering On prem, hybrid and cloud environments to Enterprises

*Share old world challenges with new world solutions:

What is our Differentiator: Cloudwick was founded on BD, it is our DNA
Core/Foundation: Network & Consultants
These are not new challenges
New world challenges with Old World Problems still includes:
Building a platform in order to access their data
Security
Governance: with access and control
Predictive / Prescriptive Analytics

Example of a UK customer that Transformed these challenges : UKDA
UKDA – They used old existing way to base the new security… what happened, completely locked down, couldn’t get data in or out

UKDA – Largest Government Research Agency for Social Sciences
Security – 0 Data Loss – DR strategy took time
Hybrid environment accessing data – security –
Synergy between platform and analytics
Strategy: share their architecture and data with other research compn
Going a bit deeper, sharing where The most time and focus was spent understanding and building this solution
Just like in Maslow’s model, each lower stage is a necessary foundation for the stages above it.
If any foundation is weak, the higher stages may crumble and leak even if they initially work. Resources for building each stage involve a combination of human talent and the right set of tools for your needs.

Collect: For a robust data pipeline, you want access to clean, reliable data then manage for - Govern, Quality, Platform, Multi-tenancy & Operations
Manage: Once you have a consistent stream of clean data, you need to have a clear understanding of the data’s composition. Descriptive statistics is where you look into the past and ask questions that have concrete answers to them.

TR: The other question I’m always interested in ,… who holds the key to building?
Both IT & Business hold the key to unlock the value – the goal is the same, to secure, govern allow the individuals that have signing authority to access – once that is there, the individual can access as often as they like –rely on their valuable assets being in the safest and most secure location for them, with back up strategies and insurance in case of a robbery

Business understands what data they want to get to
IT unlocks and makes sure it’s secure and accessible to the business users

Transition: Once the keys are established, how do you build alignment?
IT builds from the bottom up to enable business to drive from the top down
IT Builds for growth, Business demands growth
Automated technology is allowing the Gap to close
Though still about understand the old problems with new solutions
Business is becoming more savvy with Tech
IT are becoming aware of the business

AZ…. Pharma that has both
Once the synergy and conversations unfold, use cases start to emerge
How to you…. Determine the Business value
I see it by prioritsing Use Cases, and calculating 4 areas:
Metrics to improve: application, activation, retention, revenue, referral, operation
Expected improvement: 10x or more, 2x or more, 1-2x, 1x, <1x
% of users affected: 100%, 80-99%, 50-79%, 20-49%, 10-19%, <10%
Frequency of Use Case: Daily, few times/week, weekly, monthly, few times a year, once a year or less
Calculate:
As the previous slide showed, it’s a staged process, building upon each other, going from….
Once you implement batch use cases, streaming will follow and accelerate the value for your compeletive edge

Global Copyright infringment: Use Case / Batch – got them down from 4 hours / match to 1 second
Next implementing Kafka use case

Disrupting revenue streaming –
Streaming is 50 days?
TR: At this time, I’d like to hand over to Alon to share the BMC offering he helps his customers with
We talked the challenges organizations implementing Big Data have and the many of these challenges have been around for a long time
business keeps throwing things at us and it boils down to the needs of the business day
With Control-M we are uniquely positioned to help our customers in reconciling these challenges
Being consistent across all the technologies
EDW offload job or Hadoop Pig or Hive or Scoop or Impala or any other tool to load and manipulate data
Beyond Hadoop
Self Service
Cloud
The process for building the workflows - Simple drag and drop
For example - transfer data, running a Hadoop process like scoop or MapReduce, a few in parallel, creating data and then perhaps another file transfer + apply SLA
That’s it - simple as that to put a workflow together
No need of scripting or the headache to integrate all apps
Automation is a must in order to support today’s modern application delivery
Scripting is just not good enough
The more I talk to orgs implementing Big Data, the more I hear about time wasted on scripting and failures - it doesn’t need 2 b this way
What we offer - automate each in every step of your big data journey, from the data ingestion process, through data processing and analytics
Sky, a British telecommunications – using Big Data to cost-effectively deliver their media services and content
Converted from Jenkins to Control-M (management, speed, reliability)
Uses Control-M for Hadoop to run customer analytics (make realistic estimations of what people are watching)
Estimates a 40% reduction in time from Dev to Prod by replacing Jenkins with Control-M in Development phase
Navistar, a $10B company who is on the latest frontier with big data and the Internet of Things (IoT)
OnCommand Connection system is used to monitor the performance of more than 160,000 trucks
It captures millions of data points from vehicles, including fuel economy, geo location, idle time, and potential failures
They can diagnose issues as they develop and provide guidance to fleet owners—helping them route trucks to dealer service centers
This means less time in the garage and more time on the road, and that translates into a higher return on the investment in the vehicle

The Modern Data Platform - How to Conquer a New World with Old Problems

Esté a ler uma amostra.

Confira estes a seguir

The Modern Data Platform - How to Conquer a New World with Old Problems

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a The Modern Data Platform - How to Conquer a New World with Old Problems (20)

Mais de DataWorks Summit/Hadoop Summit (20)

Mais recentes (20)

The Modern Data Platform - How to Conquer a New World with Old Problems

Notas do Editor

The Modern Data Platform - How to Conquer a New World with Old Problems

Esté a ler uma amostra.

The Modern Data Platform - How to Conquer a New World with Old Problems

Recomendadas

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a The Modern Data Platform - How to Conquer a New World with Old Problems (20)

Mais de DataWorks Summit/Hadoop Summit (20)

Mais recentes (20)

The Modern Data Platform - How to Conquer a New World with Old Problems

Notas do Editor