Hadoop World 2011: Hadoop Trends & Predictions - Vanessa Alverez, Forrester

Forrester’s View On Big Data-Trends in the Enterprise
Big data will disrupt the data management and analytics landscape

Vanessa Alvarez, Analyst
November 8, 2011

2 2 © 2010 Forrester Research, Inc. Reproduction Prohibited
2009
© 2011 Forrester Research, Inc. Reproduction Prohibited
2

Forrester’s View of Big Data

3

Big Data makes extreme scale economical

Our View of Big Data:
The techniques and technologies that make capturing
value from data at extreme scale economical

Big Data technology:
Handles data at extreme scale.
Can be characterized by:
• Massive parallel computing to divide and conquer huge workloads.
• Extremely flexible to allow unlimited data manipulation and
transformation.
• Massively scalable in terms of both technology and cost.


Dimensions of Extreme Scale


The Big Data has not cross the chasm

• Social Sensitives • Big Buckers
• Web 2.0s • Survivors • One footers
• Data crunchers • The Awakened

• The Hadoop Crowd
• Proprietary Plays • The “I do tooers” • The “I guess we betters”
• The NoSQL Crowd

• The biggest & fastest

Source: Forrester Research and http://en.wikipedia.org/wiki/Crossing_the_Chasm_(book)


What Does the Customer Look Like?

7

The Big Data customer has multiple personality
disorder
Firms and excited, confused and skeptical all at the same time…

 Firms want more but don’t now what to do with what they have
 We sent our survey to thousands of clients – only got 60 completes

 The high number of partial completes indicates confusion
 Firms are experimenting with data they already have
 …but planning for new types of data

 They plan on keeping their data a long time


They want more data, but aren’t sure how to
manage what they have

61% believe data will 56% are concerned with
change how business the ability to manage
operates. current data.

330 303

Source: “Global Survey: The Business Impact of Big Data,” Avanade, November 2010


Buyers really want this future…
Current Future

“No data is discarded anymore! U.S.
xPress leverages a large scale of
transaction data and a diversity of
interaction data, now extended to
perform big data processing like
“Our oil rigs generate about 25,000
Hadoop …We assess driver
data points per second and we only
performance with image files and pick
use about 5% of that information.”
up customer behaviors from texts by
customer service reps. U.S. xPress
— super major energy saved millions of dollars per year by
company executive …augmenting our enterprise data with
sensor, meter, RFID tags, and
geospatial data.”

— CTO, energy company


Forrester’s initial assessment verticals

ent
nce
om

le
ure ta inm

e sa
ura
c
ele

g
Ins

hol

or
urin
r
& L , Ent e
&T

ect
&W
e&

act
es

S
s

eis
anc
litie

v ic

blic
nuf
dia

tail
Ser
Fin

Me

Ma
Ut i
Data

Pu
Re
Marketing
Operations

Sales
Risk
Management
IT Analytics
Finance
Product
Development
Customer
Service
Logistics

Innovators Early Adopters Early Majority Late Majority & Laggards

Source: Forrester research – this data represents our preliminary qualitative assessment based interviews and inquiries


Various types of data are addressed with marketing
and operations being top
“What enterprise areas does your Big Data initiative address?”

Marketing 45%

Operations 43%

Sales 38%

Risk management 35%

IT analytics 33%

Product development 32%

Finance 32%

Customer service 30%

Logistics 22%

Other 12%

HR 12%

Brand management 8%

Base: 60 IT Professionals who are Forrester clients
Source: How Forrester Clients Are Using Big Data, September 2011 (Upcoming)

A ¾ are actually doing something with big data or at
least evaluating it.

“What is the status of your Big Data initiative?”

Other Don't know
Testing 2% 2%
2%

Piloting
18%

Evaluating
In production 53%
23%




What Does the Data Look Like?

1

Firms Are Storing Many Types Of Data
“Overall, including the data/information on networked and direct-attached disk storage that you know about,
how much data/information does your firm currently maintain? “

0 to 100GB 100GB to 999GB 1TB to 9 TB 10TB to 99TB 100TB to 9PB 10PB or more
1%
Overall — company wide 5% 22% 33% 29% 6%

Database systems 13% 23% 27% 18% 12% 2%

General file storage, including file servers 5% 20% 34% 21% 13% 2%

Server backups for disaster recovery and continuity 10% 19% 31% 21% 12% 2%

Archiving, all forms including compliance, discovery, and
14% 19% 29% 18% 11% 2%
infrequently accessed data
Digital and Web content repositories, such as video, audio,
29% 22% 25% 10% 6% 1%
images, and Web pages

PC backups for disaster recovery and continuity 41% 20% 17% 10%4%

Enterprise content management 31% 20% 21% 11% 5% 1%

Other data 35% 18% 17% 8%4% 1%

Base: 1,252 North American and European IT executives and technology decision-makers
Source: Forrester’s Forrsights Hardware Survey, Q3 2010

Data Is Growing Faster Than Firms Can Deal With
“For each data/information type, what do you expect will be the growth at your company over the
next 12 months?”

More than 100% 50% to 100% 26% to 50% 11% to 25% Grow 1% to 10% No
growth
2%
Overall — company wide 7% 16% 42% 31% 2%
1%
Database systems 4% 11% 34% 44% 4%
1%
General file storage, including file servers 3% 10% 33% 46% 4%
2%
Archiving, all forms including compliance, discovery, and
4% 11% 28% 44% 9%
infrequently accessed data
1%
Server backups for disaster recovery and continuity 3% 11% 32% 40% 10%
1%
Digital and Web content repositories, such as video, audio,
3% 10% 24% 44% 15%
images, and Web pages
1%
Enterprise content management 3% 9% 19% 50% 15%
1%
Other data 4% 14% 51% 23%
1%
PC backups for disaster recovery and continuity 5% 18% 44% 26%

Base: 1,252 North American and European IT executives and technology decision-makers

Buyers are interested in big data for a number of
reasons
“What are the main business requirements or inadequacies of earlier-generation BI/DW/ET
technologies, applications, and architecture that are causing you to consider or implement
Big Data?”
In traditional BI and DW applications, requirements come first, applications come later. In other worlds requirements
drive applications. Big Data turns this model upside down, where free form exploration using Big Data technology to
prove a certain hypothesis or to find a pattern, often results in specifications for a more traditional BI/DW application

Data volume 75%
Analysis-driven requirements (Big Data) vs.
58%
requirements driven analysis (traditional …

Data diversity, variety 52%
Velocity of change and scope/requirements
38%
unpredictability
Cost. Big Data solutions are less expensive
30%
than traditional ETL/DW/BI solutions

Other 10%

Don't know 3%

Cost is also a factor, in many cases, dealing with data using big data technologies is simpliy cheaper and faster than
other methods.

Base: 60 IT Professionals


Most Current Production Implementations Use Existing
Transactional Data Sources For Big Data Analysis

“What types of data/records are you planning to analyze using Big Data technologies?”
Most big data use cases hype its application for analysis of new, raw data from social media, sensors and web traffic,
but we found that firms are being very practical, with early adopters using it for operating on enterprise data they
already have.

Transactional data from enterprise applications 72%

Sensor/machine/device data 42%

Unstructured content from email, office… 35%

Social media (Facebook, Twitter, etc) 35%

Locational/geospatial data 27%

Clickstream 27%

Image (large video/photographic) data 13%

Scientific/genomic data 12%

Other 7%

Don't know 5%


BI, ERP and CRM applications are the biggest
recipients of big data insights

“Do your Big Data applications stand on their own?”

Business intelligence (BI), analytics 55%

Enterprise applications (ERP, CRM) 28%

Don't know 17%
Lower responses for
BPM and BRE
Business rules (BRE) 17% indicate that
automation of
processes using
Business processes (BPM) 17% insights from big data
is not as common as it
might eventually be.
Other 12%

(multiple responses accepted)

A significant number of Forrester clients are still using
commercial technology for Big Data

Commercial source Big Data tools 47%

Open source Big Data technology (Hadoop,
MapReduce, Cassandra, and the other Apache 37%
open source specs)

Don't know 23%

Other 17%

(multiple responses accepted)


What are the Technology Concerns with
Big Data?

2

Customer challenges for securing Big Data

• Customers are not actively talking about
Awareness and security concerns.

understanding are • Customers need help understanding threats
lacking in a Big Data environment

• Main considerations: Synchronizing retention
Country policies and disposition policies across jurisdictions,
moving data across countries.
and laws add
complexity • Customers need help navigating frameworks
and changes.

Key takeaway
Customers are in need of
education and guidance.

Storage Efficiency Challenges for Big Data

• Challenge: In most instances, data is
random and inconsistent, not duplicated

• Opportunity: There is a need for more
De-duplication intelligent identification of data

• Challenge: Compression normally
happens instead of de-duplication, yet, will
compress duplicate data regardless

Compression • Opportunity: There is a need for an
automated manner in doing both de-
duplicating, and then compressing


A Conservative Perspective of Storage $ Increase
How do you expect your firm’s spending on the following IT infrastructure expenses to
change over the next 12 months?

Stay about the same Increase 5%-10% Increase more than 10%

Storage technologies 36% 40% 15%

IT infrastructure/operations staff salaries 52% 34% 4%

Systems management 59% 25% 4%

Data center and IT facilities 49% 30% 9%

Servers and server operating systems 43% 35% 9%

Infrastructure systems integration and consulting services 57% 21% 6%

PCs and PC operating systems 48% 29% 6%

Infrastructure outsourcing and managed services (based
57% 18% 5%
on fixed-price, long-term contracts)

Pay-per-use online hosted infrastructure services, such as
54% 16% 5%
cloud

Base: 1725 North American and European IT executives and technology decision-makers

Commodity vs Specialized Hardware: Pros and Cons

• Opportunity: Cost efficient, scalable, open
Commodity • Challenge: reliability, availability

• Opportunity: performance, scalable
Specialized
• Challenge: Cost, proprietary

Source: Forrester Research

Potential future state architecture: ANY DATA

Analytics Tools

Hadoop
•Compression
•De-duplication
•Encryption
Intelligent Management •Provisioning
•Virtualization

DAS SAN NAS
Network Network Network


Moving Ahead

2

On-Premise vs Cloud: Pros and Cons

• Challenge: Capex, space
On-Premise • Opportunity: Security, ownership

• Challenge: Security, manageability
Cloud
• Opportunity: Cost efficient, scalable, opex


The market is rapidly evolving to fill gaps

 Other file storage systems that capture some Hadoop
market share.
 Other computational frameworks, beyond MapReduce
that improve operational analytics.
 Advanced streaming and transactional processing that
further support real-time.
 Business-friendly features that empower your
workforce.
 Storage efficiency and security issues will be solved by
somebody…(?)

Gaps in the Big Data market mean opportunities

 Intelligent Management
– Storage efficiency
– Security

 Interactive analytics
– – Hadoop is batch only at present
– More real time analytics

 Analytics, Big Data as a Service
– Leveraging cloud


How Does This Role Evolve?

 Data ownership is everyone’s responsibility
 Important to bring together a team within IT as well as a business analyst

 Understand your peers in applications and database
 Train or hire a data analyst that can handle data lifecycle management


Questions, comments, discussion?


Thank you

Vanessa Alvarez
+1 617.613.6259
valvarez@forrester.com
Twitter: @vanessaalvarez1
Blog: blog.forrester.com
www.forrester.com


Big Data Is Disruptive Not Incremental

Big data is not incremental solutions to old problems
where data has grown bigger
 Big data is
– New techniques and technologies
– To handle that we do not deal with today
– Using scalable and parallel technologies
– That trade off consistency for high availability and node failures.

 Forrester definition excludes some solutions that other might call “big
data”. Examples:
– Massively parallel data warehouses
– In-memory databases


…but the “why” is about speed to value


Security considerations for Big Data are nascent
 Current customer focus:
– What is Big Data, what can we do with it?
– Concerns over stability, scale, cost, availability,
unified management
– Not security (not yet)

 Current storage and security vendor focus:
– What do we need to know about Big Data?
– What can be done from a security perspective?
– Where do we fit?

 Another type of discussion emerging:
– Big Data for security
– Example – Zettaset Security Data Warehouse,
using Big Data to identify security threats


Key Pieces to the Big Data Puzzle

 The role of SSD
– Prices coming down
– Commoditization of hardware happens more at the SAN, NAS than SSD
– Analytics require speed and performance, in order to achieve near real-time

 Storage Virtualization
– Just like servers were virtualized, so will storage be, and a “storage hypervisor”
will determine where storage capacity will come from, whether it be SAN, NAS
or SSD (DAS)
– Enterprises are looking to still leverage their existing infrastructure, instead
of ripping and replacing with “commodity infrastructure”


Hadoop World 2011: Hadoop Trends & Predictions - Vanessa Alverez, Forrester

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Hadoop World 2011: Hadoop Trends & Predictions - Vanessa Alverez, Forrester

Semelhante a Hadoop World 2011: Hadoop Trends & Predictions - Vanessa Alverez, Forrester (20)

Mais de Cloudera, Inc.

Mais de Cloudera, Inc. (20)

Último

Último (20)

Hadoop World 2011: Hadoop Trends & Predictions - Vanessa Alverez, Forrester