Hadoop 101 for Chattanooga Hadoop Meetup

2. Josh Patterson
Email:
josh@pattersonconsultingtn.com
Twitter:
@jpatanooga
Github:
https://github.com/jpatanooga
Slideshare
http://www.slideshare.net/jpatanooga/
Past
Published in IAAI-09:
“TinyTermite: A Secure Routing Algorithm”
Grad work in Meta-heuristics, Ant-algorithms
Tennessee Valley Authority (TVA)
Hadoop and the Smartgrid
Cloudera
Principal Solution Architect
Today: Patterson Consulting

3. Overview
• What is Hadoop?
• Hadoop and Industry
• Is Hadoop for Me?

5. Hadoop Distributed File
System (HDFS)
MapReduce
Apache Hadoop
• Consolidates Mixed Data
• Move complex and relational
data into a single repository
• Stores Inexpensively
• Keep raw data always available
• Use industry standard hardware
• Processes at the Source
• Eliminate ETL bottlenecks
• Mine data first, govern later
5

6. Why is it Called Hadoop?
Doug’s son had a toy elephant that he
called “Ha-Doop”
Doug Cutting Invented Hadoop.

7. What Hadoop Does
Uses commodity hardware / servers
Scales into Petabytes without hardware changes
Manages fault tolerance and replication with its distributed
file system
Scalable processing engine handles all types of data
Text, logs, documents
Binary, images, video

8. Hadoop Distributed File System (HDFS)
Based on design of Google’s GFS
Meant for high levels of throughput which sustain
Map Reduce parallel processing jobs
Data stored in large files
Large blocks, (64MB, 128MB, 256MB, etc) per block

9. MapReduce: Distributed Processing

10. Hadoop Analysis Tools
Java Map Reduce
Hive and Impala
SQL-like language for Hadoop
Declarative higher level language
Pig
Procedural higher level language
Filters, joins, udfs
10

12. Starting Out: 2008
Got going at Facebook and Yahoo
Became the backbone of many CA startups
In 2009 we did a POC with Hadoop @ TVA
http://openpdc.codeplex.com/

13. Source: IDC White Paper - sponsored by EMC.
As the Economy Contracts, the Digital Universe Expands. May 2009.
.
Unstructured Data Explosion
• 2,500 exabytes of new information in 2012 with Internet as primary driver
Relational
Complex, Unstructured
(You)

14. Financial Services Got Interested in Hadoop
Banks saw the potential to look at a lot of transactions for
things like
Fraud
Money Laundering Detection
Comprehensive Credit Reports
Teradata had become very expensive
Started using Hadoop to augment mass data transforms

15. Genomics
A genome is 2.8GB
There are lots of genomes
Genomics (ISB, Novartis, etc) became very
interesting on Hadoop around 2010 and 2011
There are many CPU bound processes in genomics
But a lot of it is also disk bound – great for Hadoop

16. Other Verticals Jumped In
Telecoms
Lot of call histories to look at, Billing, etc
Media
Help recommend folks stuff to watch based on what they watch
Manufacturing
Sensor data on devices works well as timeseries in Hadoop
Insurance
Lots of data can build better models of how folks live
Can give insurance co’s better ways to model policies

18. Data is Not Always Big
But the world is becoming progressively more interested
in data
Big and small
Data analysis is driving how we build new products,
manage our lives, and consume content
Focus on producing a result that is relevant to the industry
And not on how much data you have or if it qualifies as “big”

19. ETL Pipelines
Many early Hadoop use cases involve porting a data
transform pipeline into Hive or MapReduce
Allows for linearly scalability on throughput
Processing web logs has always been the
MapReduce base case
Many times the result data is sent back to an
RDBMS store

20. Recommend Products
Ever used Facebook’s “People you may Know”?
Ever used Amazon’s “People Also Bought”?
These are recommendation systems
Hadoop powers both of these
Deep Dive into Recommenders on Hadoop
http://www.slideshare.net/jpatanooga/la-hug-dec-2011-
recommendation-talk

21. Hadoop as Next-Gen Data Warehouse
Easier to work with data where it lives
Less dependent on DBAs and schemas
Hadoop is Community driven
Is spiritually very similar to Linux
Open source core
With Distro model (think RedHat and Ubuntu)

22. Is Hadoop For My Use Case?
Am I doing a lot of table / disk based transforms?
The process is disk bound and batch-oriented, so yes
Do I need to ad-hoc query large amounts of data?
Hive or Impala make sense here
Am I dealing with a lot of incoming transactional data that
I’d like to analyze (logs, typically)?
Hadoop is great for cheap storage and scalable processing

23. Questions?
Thanks for coming out to hear about Hadoop!