Zh Tw Introduction To Map Reduce

MapReduce
• Google
• Google
•
• Map Reduce
• Google map reduce

• MapReduce
• Map
– [1,2,3,4] – (*2)  [2,3,6,8]
• Reduce
– [1,2,3,4] – (sum)  10

– (Divide and Conquer)

Copyright 2009 - Trend Micro Inc.

MapReduce
• MapReduce Google
Map Reduce
MapReduce
•
– Map ” ” key/value ” ”
intermediate key/value
– Reduce intermediate key
intermediate values key/value
• MapReduce


•
–
–
–

•

http://www.dbms2.com/2008/08/26/known-applications-of-mapreduce/


MapReduce
•
– map (K1, V1)  list(K2, V2)
– reduce (K2, list(V2))  list(K3, V3)

• grep
– Map: (offset, line)  [(match, 1)]
– Reduce: (match, [1, 1, ...])  [(match, n)]

• MapReduce :


6
Classification Copyright 2009 - Trend Micro Inc.

‧ ➝
‧ ➝


Word Count


MapReduce

• (Distributed Grep)
– (pattern)

• (Distributed Sort)
–

• URL (Count of URL Access Frequency)
– Web URL


MapReduce


Hadoop MapReduce

• Apache Hadoop Google MapReduce
– MapReduce
– Java
– Hadoop (HDFS)
• Yahoo!
• Google, Yahoo!, IBM, Amazon Hadoop
• (Trend Micro) Hadoop MapReduce


Hadoop MapReduce

• Map/Reduce framework
– JobTracker
– TaskTracker
• JobTracker
– Job
– Job JobTracker Job.
• TaskTrackers
• Job

Classification

Hadoop MapReduce
class MyJob {

class Map { // Map
}
class Reduce { // Reduce
}

}
main() {
// job
JobConf conf = new JobConf(“MyJob.class”);
conf.setInputPath(…);
conf.setOutputPath(…);
conf.setMapperClass(Map.class);
conf.setReduceClass(Reduce.class)
// Job
JobClient.runJob(conf);
}
}

•
–
–
–

–
HDFS MapReduce
•
–
–
•
•

, GUID, , ,
1, 123, 131231231, VSAPI, open file
2, 456, 123123123, VSAPI, connect internet


Map
• Mapper map()
• Map : (K1, V1)  list(K2, V2)

map( WritableComparable, Writable,
OutputCollector, Reporter)

• input map()
• OutputCollector collect() method

OutputCollector.collect( WritableComparable,Writable )


Map

class MapClass extends MapReduceBase

implements Mapper<LongWritable, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);

private Text hour = new Text();

public void map( LongWritable key, Text value,
OutputCollector<Text,IntWritable> output, Reporter reporter) throws
IOException {

String line = ((Text) value).toString();

String[] token = line.split(quot;,quot;);

String timestamp = token[1];

Calendar c = Calendar.getInstance();

c.setTimeInMillis(Long.parseLong(timestamp));

Integer h = c.get(Calendar.HOUR);

hour.set(h.toString());

output.collect(hour, one)

}}} Copyright 2007 - Trend Micro Inc.

Reduce
• Reducer reduce() method
• Reduce : (K2, list(V2))  list(K3, V3)

reduce (WritableComparable, Iterator,
OutputCollector, Reporter)

• OutputCollector collect() method

OutputCollector.collect( WritableComparable,Writable )


Reduce

class ReduceClass extends MapReduceBase implements Reducer< Text,
IntWritable, Text, IntWritable> {

IntWritable SumValue = new IntWritable();

public void reduce( Text key, Iterator<IntWritable> values,

OutputCollector<Text, IntWritable> output, Reporter reporter)

throws IOException {

int sum = 0;

while (values.hasNext())

sum += values.next().get();

SumValue.set(sum);

output.collect(key, SumValue);

}}


• JobConf
– Mapper Reducer Inputformat OutputFormat Combiler Petitioner

–
–
–
• map reduce
•

• JobClient JobConf

JobClient.submitJob(conf);
JobClient.setJobEndNotificationURI(URI);


Main Function
Class MyJob{
public static void main(String[] args) {
JobConf conf = new JobConf(MyJob.class);
conf.setJobName(”Caculate feedback log time distributionquot;);
// set path
conf.setInputPath(new Path(args[0]));
conf.setOutputPath(new Path(args[1]));
// set map reduce
conf.setOutputKeyClass(Text.class); // set every word as key
conf.setOutputValueClass(IntWritable.class); // set 1 as value
conf.setMapperClass(MapClass.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(ReduceClass.class);
onf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
// run
}}


1.
– javac -classpath hadoop-*-core.jar -d MyJava
MyJob.java
2.
– jar –cvf MyJob.jar -C MyJava .
3.
– bin/hadoop jar MyJob.jar MyJob input/ output/


• bin/hadoop jar MyJob.jar MyJob input/ output/

Classification

Web Console
http://172.16.203.132:50030/

Classification

Hadoop MapReduce
• Mapper ?
– Mapper Input Input Hadoop
Mapper
– JobConf setNumMapTasks(int) Hadoop
Mapper Hadoop

• Reducer ?
– JobConf JobConf.setNumReduceTasks(int)
Reducer
– Reducer Reducer
MapReduce Map Reduce

Classification

Non-Java Interface
• Hadoop Pipes
– MapReduce C++ API
– C++ java
• Hadoop Streaming
– MapReduce


• Google MapReduce
– http://labs.google.com/papers/mapreduce.html
– http://code.google.com/edu/submissions/mapreduce/listing.html
– http://code.google.com/edu/submissions/mapreduce-minilecture/
listing.html
• Hadoop
– http://hadoop.apache.org/core/


• Eclipse MapReduce (IBM
)
– Eclipse Hadoop
–
• http://code.google.com/edu/parallel/tools/hadoopvm/hadoop-
eclipse-plugin.jar
• Hadoop (Google )
– VMware Hadoop

VMware Google
–
• http://code.google.com/edu/parallel/tools/hadoopvm/
index.html


Zh Tw Introduction To Map Reduce

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Zh Tw Introduction To Map Reduce

Semelhante a Zh Tw Introduction To Map Reduce (20)

Último

Último (20)

Zh Tw Introduction To Map Reduce