A Dozen Things to Remember on your Big Data Journey. A presentation by John Bostick of LUCRUM, July 2013 to the JAX Chamber IT Council.

1. If companies are not careful,
“Big Data” will become “Big Dilbert”
-- John Bostick, LUCRUM, July 2013
A Dozen Things to Remember on your Big
Data Journey

2. A Dozen Things to Remember on your Big Data Journey
1. Data is Growing
2. Decision-Making is Accelerating
3. Data is Changing
4. Questions are Maturing
5. Methods are Changing
6. Architectures are Expanding
7. Technologies are Evolving
8. Terminology is Expanding
9. Business Cultures Are Transforming
10. Errors occur in milliseconds
11. Resources Demands are Escalating
12. IT Departments are Falling Behind
2

3. 1. Data is Growing Exponentially
• Industry experts estimated that
75% of that growth comes from
individuals.
• Additionally, they estimate that
80% of data is used by
commercial organizations.
• The number of mobile-
connected devices will exceed
the world's population in 2013.
• Your digital footprint extends
across Facebook, Google,
Twitter, Netflix, etc.
3
Anonymity is becoming algorithmically impossible.
– MIT Technology Review
http://mashable.com/2013/05/07/big-data-anonymity/

4. 1. Data is Growing Exponentially -- continued--
• Wal-Mart handles more than 1 million customer transactions
every hour, which is imported into databases estimated to contain
more than 2.5 petabytes of data.
• Akamai analyzes 75 million events per day to better target
advertisements.
• 20B credit card transactions per year in the US.
• Kroger processed ~2B transaction logs per week (2004 Stat).
• Individuals create 70 percent of all data. Enterprises store 80
percent.
• In other words, the amount of data in the world today is equal to:
– Every person in the US tweeting three tweets per minute for 26,976 years.
– Every person in the world having more than 215M high-resolution MRI
scans a day.
– More than 200B HD movies – which would take a person 47MM years to
watch.
4
Sources: http://wikibon.org/blog/big-data-statistics, www.indexcreditcards.com, http://www.csc.com/

5. 2. Decision-Making is Accelerating
5
• Is my brand
profitable?
• Is my customer
profitable?
Data Velocity --------- >
DataVolume--------->
• How do I influence my
customer?
• How do others influence my
customer?
• Do I need to act now to keep
my customer happy?
Answers are needed at more granular level and a
faster pace!

6. “We welcome change and openness; for we believe that
freedom and security go together, that the advance of
human liberty can only strengthen the cause of world
peace. There is one sign the Soviets can make that would be
unmistakable, that would advance dramatically the cause
of freedom and peace. General Secretary Gorbachev, if you
seek peace , if you seek prosperity for the Soviet Union and
eastern Europe, if you seek liberalization, come here to this
gate. Mr. Gorbachev, open this gate. Mr. Gorbachev, Mr.
Gorbachev, tear down this wall!”
3. Data is Changing
6
Internal
Structured
Data
External
Structured
Data
Internal
Unstructured
Data
External
Unstructured
Data
Sales Analysis, Financial Reports, Key
Performance Indicators, Inventory
Analysis, ………..
Sentiment Analysis, Customer Churn,
License Plate Tracking, Reputation
Analysis, Brand Monitoring, ……..
Big
Data
Market Share, Share of Wallet, Credit
Worthiness, Background Check,
Competitive Pricing, ……..
Call Center Training, Medical Text
Mining , Product Research Mining, Legal
Records Research, Fraud Detection, ……

7. Improved
Decision
Making
4. Questions are Maturing
Increased
Business
Value
Query
Drilldown
Alerts
Adhoc
Reports
Standard
Reports
Predictive
Modeling
Optimization
Forecasting
Statistical
Analysis
What
happened?
How many, how often?
Where exactly is the problem?
What actions are needed?
Why is this happening?
What if these trends continue?
What will happen next?
What is the best that can
happen?

8. 5. Methods are Changing
Agility to grow and change is key
• Facebook
– There is no information on the exact count, but estimates from power
user put their server counts at approximately:
– 180,000 (Aug 2012) from…
– 60,000 (June 2010) from…
– 30,000 (Oct 2009)
8
Facebook
• Google is estimated to be over a million
servers.
• Facebook now updates its code twice
every day
• Flickr – 10 releases per day
• Instagram – 100M users, 5B images, 3
engineer and Amazon’s Elastic Cloud
http://news.cnet.com/8301-1023_3-57486696-93/facebook-now-updates-its-code-twice-every-day
Sources: CNET, Instagram Engineering, Flickr, Facebook, Mvdirona

9. 6. Architectures are Expanding
Marketing Management
Marketing Operations
Customer
Experience
E-commerce
Social Networks
Mobile, SMS, ..
Surveys
Mail, Billing
Phone
email
In Store
Media
KIOSK
PURL, QR
POS
3rd Party
Website
Mobile
Website
Voice of
Customer
Email
engine
Social
Monitoring
Couponing
Local
Marketing
Lead Gen
Call
Center RoboCall
More..
Print
Media
CRM
Marketing
Analytics
Segmentation
Trends
Behavior
Event
Pattern
Context
Content
Language
Sentiment
Customer Value
Customer Churn
Marketing Information
People,
Places, &
Things
Activity,
Transactions,
Etc.
Customer
Master
Content
Repository
Product
Catalog
Promotion
Catalog
Sales
Web logs
Social
Activity
Emails,
Calls,
Texts, …
Multi –channel
Campaign Mgmt
Performance &
Financial Mgmt
Optimization &
Modeling
Integrated
Marketing Mgmt
Big Data in Customer Communications

10. 7. Technologies are Evolving
10
Technology continues to reinvent itself.

11. 8. Terminology is Expanding
HFT
ACID
V3
Infomediary
Metadata
Situational Awareness
Quant
CDO
Authoritative Source
Data Lifecycle
NLP
Latency
YottaByte
HiPPO
11

12. 9. Business Cultures are Transforming
12
Becoming “data aware” is a journey. Incubated in a series of
projects and ending with a cultural transformation.
Gartner BI Maturity Model
Is the HiPPO going
the way of the
dinosaur?

13. 10. Errors occur in milliseconds
• Which is better: faster or slower?
– A “Twitter hoax” briefly erased $200 billion of value
from the US Stock Market in April.
• False reports of explosions in the White House
triggered a set of algorithms monitoring news feeds
into a two minute selling spree.
• DOW drops 145 points.
• Why? New technology can ‘read’ social media
messages and place bets accordingly
13
What losses were incurred by algorithms
reacting to a news feed and potentially other
algorithms reacting to those algorithms???

14. 10. Errors occur in milliseconds -- continued --
• The cost of bad data exceeds $600B dollars for US
businesses annually.
• Almost, 50% of respondents cite data quality as the
greatest barrier to adopting Business Intelligence.
• Poor data quality will cost the UK’s 4 largest
supermarkets $1B dollars over the next 5 yrs.
• Poor data is cited as the number one reason for project
overruns.
• For a median Fortune 1000 company, a 10% increase in
data usability would increase revenue by $2B.
14
http://www-new.insightsquared.com

15. 10. Errors Re-Occur over Days – continued--
15
Amazon Sale Price
$23.7M + 3.99 for Shipping
Two sellers with two different
pricing algorithms that
automatically set prices based
on competing prices
Price of book rises to $23.7M
over 10 days!

16. 11. Resource Demands are Escalating
Sales, Payments,
Orders, Transactions,
…
Email, SMS,
Twitter, …..
YouTube,
Instagram, Netflix,
flickr, twitpic,
Dailymotion, ….
Skype, lingo,
phonepower, ITP,
phone.com,….
16
A 2011 research report by
Mckinsey Global Institute
predicted that by 2018 , the US
job market would experience a
shortage of around 1.5M
managers & analysts with the
know-how to use analysis on big
data.
Volume!
Velocity!
Big
Data

17. 12. IT Departments are Falling Behind
Available Resources
Run & Maintain
Staff
Time
Questions from the CIO……..
• How do I meet the demands of the business for innovation?
• How do I develop business subject matter experts that are adept at
applying technology to business problems?
• How do I train my employees on new technologies? By the way, which
new technology(s)?
• How do I reduce my support time?
• How do I find, hire and retain top resources?
The Time for Innovation is
shrinking!
17

18. Summary
• Big Data is Here
– and has been for awhile
• Big Data is not a “Technology Project”
– Although there are many technology choices
• Big Data does not solve every Problem
– People do! (i.e., Data Quality)
• Big Data is a Journey
• Big Data is a Cultural Change
18

19. 19
John Bostick
513-702-3810
jbostick@lucruminc.com
www.lucruminc.com

20. Mobile Usage is Growing
• Global mobile data traffic grew
70 percent in 2012
• Mobile video traffic was 51
percent of traffic by the end of
2012
• Globally, 33%of total mobile
data traffic was offloaded onto
the fixed network in 2012.
20
• Mobile Data Traffic is expected to grow at a 66% CAGR from
2012 to 2017.
• The number of mobile-connected devices will exceed the
world's population in 2013.
Source: Cisco Global Mobile Data
Traffic Forecast Update, 2012–2017

21. Cloud Computing Growth
Workloads per traditional
server:
– 2011 = 1.5
– 2016 = 2.0
Workloads per cloud server:
– 2011 = 4.2
– 2016 = 8.5
21
Source: Cisco Global Cloud Index:
Forecast and Methodology, 2011–2016
By 2016, nearly two-thirds of all workloads will be
processed in the cloud.