deck from my 5 part series of YouTube (SoCalDevGal channel) on Hadoop MapReduce

1. Hadoop MapReduce Fundamentals
@LynnLangit
a five part series – Part 1 of 5

2. Course Outline

3. What is Hadoop?
 Open-source data storage and processing API
 Massively scalable, automatically parallelizable

Based on work from Google

GFS + MapReduce + BigTable

Current Distributions based on Open Source and Vendor Work

Apache Hadoop

Cloudera – CH4 w/ Impala

Hortonworks

MapR

AWS

Windows Azure HDInsight

4. Why Use Hadoop?
 Cheaper

Scales to Petabytes or
more
 Faster

Parallel data processing
 Better

Suited for particular types
of BigData problems

5. What types of business problems for Hadoop?
Source: Cloudera “Ten Common Hadoopable Problems”

6. Companies Using
Hadoop
 Facebook
 Yahoo
 Amazon
 eBay
 American Airlines
 The New York Times
 Federal Reserve Board
 IBM
 Orbitz

7. Forecast growth of Hadoop Job Market
Source: Indeed -- http://www.indeed.com/jobtrends/Hadoop.html

8. Hadoop is a set of Apache Frameworks and more…
 Data storage (HDFS)

Runs on commodity hardware (usually Linux)

Horizontally scalable
 Processing (MapReduce)

Parallelized (scalable) processing

Fault Tolerant
 Other Tools / Frameworks

Data Access

HBase, Hive, Pig, Mahout

Tools

Hue, Sqoop

Monitoring

Greenplum, Cloudera
Hadoop Core - HDFS
MapReduce API
Data Access
Tools & Libraries
Monitoring & Alerting

9. What are the core parts of a Hadoop distribution?

10. Hadoop Cluster HDFS (Physical) Storage

11. MapReduce Job – Logical View
Image from - http://mm-tom.s3.amazonaws.com/blog/MapReduce.png

12. Hadoop Ecosystem

14. Common Hadoop Distributions
 Open Source

Apache
 Commercial

Cloudera

Hortonworks

MapR

AWS MapReduce

Microsoft HDInsight (Beta)

15. A View of Hadoop (from Hortonworks)
Source: “Intro to Map Reduce” -- http://www.youtube.com/watch?v=ht3dNvdNDzI

16. Setting up Hadoop Development

17. Demo – Setting up Cloudera Hadoop
Note: Demo VMs can be downloaded from - https://ccp.cloudera.com/display/SUPPORT/Demo+VMs

18. Hadoop MapReduce Fundamentals
@LynnLangit
a five part series – Part 2 of 5

19. So, what’s the problem?
 “I can just use some ‘SQL-like’ language to query Hadoop, right?
 “Yeah, SQL-on-Hadoop…that’s what I want
 “I don’t want learn a new query language and….
 “I want massive scale for my shiny, new BigData

20. Ways to MapReduce
Libraries Languages
Note: Java is most common, but other languages can be used

21. Demo – Using Hive QL on CDH4

22. What is Hive?
 a data warehouse system for Hadoop that

facilitates easy data summarization

supports ad-hoc queries (still batch though…)

created by Facebook
 a mechanism to project structure onto this data and query the data using a
SQL-like language – HiveQL

Interactive-console –or-

Execute scripts

Kicks off one or more MapReduce jobs in the background
 an ability to use indexes, built-in user-defined functions

23. Is HQL == ANSI SQL? – NO!
--non-equality joins ARE allowed on ANSI SQL
--but are NOT allowed on Hive (HQL)
SELECT a.*
FROM a
JOIN b ON (a.id <> b.id)
Note: Joins are quite different in MapReduce, more on that coming up…

24. Preparing for MapReduce

25. Common Hadoop Shell Commands
hadoop fs –cat file:///file2
hadoop fs –mkdir /user/hadoop/dir1 /user/hadoop/dir2
hadoop fs –copyFromLocal <fromDir> <toDir>
hadoop fs –put <localfile>
hdfs://nn.example.com/hadoop/hadoopfile
sudo hadoop jar <jarFileName> <method> <fromDir> <toDir>
hadoop fs –ls /user/hadoop/dir1
hadoop fs –cat hdfs://nn1.example.com/file1
hadoop fs –get /user/hadoop/file <localfile>
Tips
-- ‘sudo’ means ‘run as administrator’ (super user)
--some hadoop configurations use ‘hadoop dfs’ rather than ‘hadoop fs’ – file paths to hadoop differ for the former, see the link
included for more detail

26. Demo – Working with Files and HDFS

27. Thinking in MapReduce
 Hint: “It’s Functional”

28. Understanding MapReduce – P1/3
 Map>>

(K1, V1) 

Info in

Input Split

list (K2, V2)

Key / Value out
(intermediate values)

One list per local
node

Can implement local
Reducer (or
Combiner)

29. Understanding MapReduce – P2/3
 Map>>

(K1, V1) 

Info in

Input Split

list (K2, V2)

Key / Value out
(intermediate values)

One list per local
node

Can implement local
Reducer (or
Combiner)
 Shuffle/Sort>>

30. Understanding MapReduce – P3/3
 Map>>

(K1, V1) 

Info in

Input Split

list (K2, V2)

Key / Value out
(intermediate values)

One list per local
node

Can implement local
Reducer (or
Combiner)
 Reduce

(K2, list(V2) 

Shuffle / Sort phase
precedes Reduce phase

Combines Map output
into a list

list (K3, V3)

Usually aggregates
intermediate values
(input) <k1, v1>  map  <k2, v2>  combine  <k2, v2>  reduce  <k3, v3> (output)
 Shuffle/Sort>>

31. Image from: http://blog.jteam.nl/wp-content/uploads/2009/08/MapReduceWordCountOverview1.png
MapReduce Example - WordCount

32. MapReduce Objects
Each daemon spawns a new JVM

33. Ways to MapReduce
Libraries Languages
Note: Java is most common, but other languages can be used

34. Demo – Running MapReduce WordCount

35. Hadoop MapReduce Fundamentals
@LynnLangit
a five part series – Part 3 of 5

36. Ways to run MapReduce Jobs
 Configure JobConf options
 From Development Environment (IDE)
 From a GUI utility

Cloudera – Hue

Microsoft Azure – HDInsight console
 From the command line

hadoop jar <filename.jar> input output

37. Ways to MapReduce
Libraries Languages
Note: Java is most common, but other languages can be used

38. Setting up Hadoop On Windows Azure
 About HDInsight

39. Demo – MapReduce in the Cloud
 WordCount MapReduce using HDInsight

40. MapReduce (WordCount) with Java Script
Note: JavaScript is
part of the Azure
Hadoop distribution

41. Common Data Sources for MapReduce Jobs

42. Where is your Data coming from?
 On premises

Local file system

Local HDFS instance
 Private Cloud

Cloud storage
 Public Cloud

Input Storage buckets

Script / Code buckets

Output buckets

43. Common Data Jobs for MapReduce

44. Demo – Other Types of MapReduce
Tip: Review the Java MapReduce code in these samples as well.

45. Methods to write MapReduce Jobs
 Typical – usually written in Java

MapReduce 2.0 API

MapReduce 1.0 API
 Streaming

Uses stdin and stdout

Can use any language to write Map and Reduce Functions

C#, Python, JavaScript, etc…
 Pipes

Often used with C++
 Abstraction libraries

Hive, Pig, etc… write in a higher level language, generate one or more
MapReduce jobs

46. Ways to MapReduce
Libraries Languages
Note: Java is most common, but other languages can be used

47. Demo – MapReduce via C# & PowerShell

48. Ways to MapReduce
Libraries Languages
Note: Java is most common, but other languages can be used

49. Using AWS MapReduce
Note: You can select Apache or MapR Hadoop Distributions to run your MapReduce job on the
AWS Cloud

50. What is Pig?
 ETL Library for HDFS developed at Yahoo

Pig Runtime

Pig Language

Generates MapReduce Jobs
 ETL steps

LOAD <file>

FILTER, JOIN, GROUP BY, FOREACH, GENERATE, COUNT…

DUMP {to screen for testing}  STORE <newFile>

51. MapReduce Python Sample
Remember that white space matters in Python!

52. Demo – Using AWS MapReduce with
Pig
Note: You can select Apache or MapR Hadoop Distributions to run your MapReduce job on the
AWS Cloud

53. AWS Data Pipeline with HIVE

54. Hadoop MapReduce Fundamentals
@LynnLangit
a five part series – Part 4 of 5

55. Better MapReduce - Optimizations

56. Optimization BEFORE running a MapReduce Job

57. More about Input File Compression
 From Cloudera…
 Their version of LZO ‘splittable’
Type File Size GB Compress Decompress
None Log 8.0 - -
Gzip Log.gz 1.3 241 72
LZO Log.lzo 2.0 55 35

58. Optimization WITHIN a MapReduce Job

59. 59

60. Mapper Task Optimization

61. Data Types
 Writable

Text (String)

IntWritable

LongWritable

FloatWritable

BooleanWritable
 WritableComparable for keys
 Custom Types supported – write RawComparator

62. Reducer Task Optimization

63. MapReduce Job Optimization

64. Demo – Unit Testing MapReduce
 Using MRUnit + Asserts
 Optionally using ApprovalTests
Image from http://c0de-x.com/wp-content/uploads/2012/10/staredad_english.png

65. A note about MapReduce 2.0
 Splits the existing JobTracker’s roles

resource management

job lifecycle management
 MapReduce 2.0 provides many benefits over the existing MapReduce
framework, such as better scalability

through distributed job lifecycle management

support for multiple Hadoop MapReduce API versions in a single cluster

66. What is Mahout?
 Library with common machine learning algorithms
 Over 20 algorithms

Recommendation (likelihood – Pandora)

Classification (known data and new data – spam id)

Clustering (new groups of similar data – Google news)
 Can non-statisticians find value using this library?

67. Mahout Algorithms

68. Setting up Hadoop on Windows
 For local development
 Install from binaries from Web Platform Installer
 Install .NET Azure SDK (for Azure BLOB storage)
 Install other tools

Neudesic Azure Storage Viewer

69. Demo – Mahout
 Using HDInsight

70. What about the output?

71. Clients (Visualizations) for HDFS
 Many clients use Hive

Often included in GUI console tools for Hadoop distributions as well
 Microsoft includes clients in Office (Excel 2013)

Direct Hive client

Connect using ODBC

PowerPivot – data mashups and presentation

Data Explorer – connect, transform, mashup and filter

Hadoop SDK on Codeplex
 Other popular clients

Qlikview

Tableau

Karmasphere

72. Demo – Executing Hive Queries

73. Demo – Using HDFS output in Excel 2013
To download Data Explorer:
http://www.microsoft.com/en-
us/download/details.aspx?id=36803

74. AboutVisualization

75. Demo – New Visualizations – D3

76. Hadoop MapReduce Fundamentals
@LynnLangit
a five part series – Part 5 of 5

77. Limitations of MapReduce

78. Comparing: RDBMS vs. Hadoop
Traditional RDBMS Hadoop / MapReduce
Data Size Gigabytes (Terabytes) Petabytes (Hexabytes)
Access Interactive and Batch Batch – NOT Interactive
Updates Read / Write many times Write once, Read many times
Structure Static Schema Dynamic Schema
Integrity High (ACID) Low
Scaling Nonlinear Linear
Query Response
Time
Can be near immediate Has latency (due to batch processing)

79. Microsoft alternatives to MapReduce
 Use existing relational system

Scale via cloud or edition (i.e. Enterprise or PDW)
 Use in memory OLAP

SQL Server Analysis Services Tabular Models
 Use “productized” Dremel

Microsoft Polybase – status = beta?

80. Looking Forward - Dremel or Apache Drill
 Based on original research from Google

81. Apache Drill Architecture

82. In-market MapReduce Alternatives
Cloudera
 Impala
Google
 Big Query

83. Demo – Google’s BigQuery
 Dremel for the rest of us

84. Hadoop MapReduce Call to Action

85. More MapReduce Developer Resources
 Based on the distribution – on premises

Apache

MapReduce tutorial - http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.htmlCloudera

Cloudera

Cloudera University - http://university.cloudera.com/

Cloudera Developer Course (4 day) - *RECOMMENDED* -
http://university.cloudera.com/training/apache_hadoop/developer.html

Hortonworks

MapR
 Based on the distribution – cloud

AWS MapReduce

Tutorial - http://aws.amazon.com/elasticmapreduce/training/#gs

Windows Azure HDInsight

Tutorial -
http://www.windowsazure.com/en-us/manage/services/hdinsight/using-mapreduce-with-hdinsight/

More resources - http://www.windowsazure.com/en-us/develop/net/tutorials/intro-to-hadoop/

86. The Changing Data Landscape