This document provides an overview and lessons learned from Hadoop. It discusses why Hadoop is used, how MapReduce and HDFS work, tips for integration and operations, and the outlook for the Hadoop community moving forward with real-time capabilities and refined APIs. Key takeaways include only using Hadoop if necessary, fully understanding your data pipeline, and "unboxing the black box" of Hadoop.

1. Hadoop
lessons learned

2. @tcurdt
github.com/tcurdt
yourdailygeekery.com

3. Data

5. hiring

6. Agenda
· hadoop? really? cloud?
· integration
· mapreduce
· operations
· community and outlook

7. Why Hadoop?

8. “It is a new and improved
version of enterprise tape
drive”

9. 20 machines
Map Reduce
20 files, 1.5 GB each
hadoop job grep.jar
grep “needle” file
ir
0 17.5 35.0 52.5 70.0
f a
u n

12. Run your own?
http://bit.ly/elastic-mr-pig

13. Integration

14. black box

15. Engineers
· hadoop-cat
· hadoop-grep
· hadoop-range
--prefix /logs
--from 2012-05-15 --until 2012-05-22
--postfix /*play*.seq | xargs hadoop jar
· streaming jobs

16. Non-Engineering Folks
· mount hdfs
· pig / hive
· data dumps

17. Map Reduce
HDFS files
InputFormat
Split Split Split Split
Map Map Map Map
Combiner Combiner Combiner Combiner
Sort Sort Sort Sort
Partitioner
Copy and Merge
Combiner Combiner
Reducer Reducer
OutputFormat

18. Job Counters
12/05/25 01:27:38 INFO mapred.JobClient: Reduce input records=106
..
12/05/25 01:27:38 INFO mapred.JobClient: Combine output records=409
12/05/25 01:27:38 INFO mapred.JobClient: Map input records=112705844
12/05/25 01:27:38 INFO mapred.JobClient: Reduce output records=4
12/05/25 01:27:38 INFO mapred.JobClient: Combine input records=64842079
..
12/05/25 01:27:38 INFO mapred.JobClient: Map output records=64841776
map in : 112705844 *********************************
map out : 64841776 *****************
combine in : 64842079 *****************
combine out : 409 |
reduce in : 106 |
reduce out : 4 |
MAPREDUCE-346 (since 2009)

19. Job Counters
map in : 20000 **************
map out : 40000 ******************************
combine in : 40000 ******************************
combine out : 10001 ********
reduce in : 10001 ********
reduce out : 10001 ********

20. Map-only
mapred.reduce.tasks = 0

21. EOF on append
public class EofSafeSequenceFileInputFormat<K,V>
extends SequenceFileInputFormat<K,V> {
...
}
public class EofSafeRecordReader<K,V>
extends RecordReader<K,V> {
...
public boolean nextKeyValue()
throws IOException, InterruptedException {
try {
return this.delegate.nextKeyValue();
} catch(EOFException e) {
return false;
}
}
...
}

22. Serialization
before
ASN1, custom java serialization, Thrift
now
protobuf

23. Custom Writables
public static class Play extends CustomWritable {
public final LongWritable time
= new LongWritable();
public final LongWritable owner_id
= new LongWritable();
public final LongWritable track_id
= new LongWritable();
public Play() {
fields = new WritableComparable[] {
owner_id, track_id, time };
}
}

24. Fear the State
BytesWritable bytes = new BytesWritable();
...
byte[] buffer = bytes.getBytes();

25. Re-Iterate
public void reduce(
LongTriple key,
Iterable<LongWritable> values,
Context ctx) {
for(LongWritable v : values) { }
for(LongWritable v : values) { }
}
public void reduce(
LongTriple key,
Iterable<LongWritable> values,
Context ctx) {
buffer.clear();
for(LongWritable v : values) { buffer.add(v); }
for(LongWritable v : buffer.values()) { }
}
HADOOP-5266 (applied to 0.21.0)

26. BitSets
long min = 1;
long max = 10000000;
FastBitSet set = new FastBitSet(min, max);
for(long i = min; i<max; i++) {
set.set(i);
}
org.apache.lucene.util.*BitSet

27. Data Structures
http://bit.ly/data-structures
http://bit.ly/bloom-filters
http://bit.ly/stream-lib

28. General Tips
· test on small datasets, test on your machine
· many reducers
· always consider a combiner and partitioner
· pig / streaming for one-time jobs,
java/scala for recurring
http://bit.ly/map-reduce-book

29. Operations
use chef / puppet
runit / init.d
pdsh / dsh
pdsh -w "hdd[001-019]"
"sudo sv restart /etc/sv/hadoop-tasktracker"

30. Hardware
· 2x name nodes raid 1
· 12 cores, 48GB RAM, xfs, 2x1TB
· n x data nodes no raid
· 12 cores, 16GB RAM, xfs, 4x2TB

31. Monitoring
dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
dfs.period=10
dfs.servers=...
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
mapred.period=10
mapred.servers=...
jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
jvm.period=10
jvm.servers=...
rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
rpc.period=10
rpc.servers=...
# ignore
ugi.class=org.apache.hadoop.metrics.spi.NullContext

32. Monitoring
total capacity capacity used

33. Compression
# of 64MB blocks
# of bytes needed
# of bytes used
# bytes reclaimed
bzip2 / gzip / lzo / snappy
io.seqfile.compression.type = BLOCK
io.seqfile.compression.blocksize = 512000

34. Janitor
hadoop-expire
-url namenode.here
-path /tmp
-mtime 7d
-delete

35. The last block of an HDFS block only
occupies the required space. So a 4k
file only consumes 4k on disk.
-- Owen
E D
S T
B U

36. Logfiles
find
-wholename "/var/log/hadoop/hadoop-*"
-wholename "/var/log/hadoop/job_*.xml"
-wholename "/var/log/hadoop/history/*"
-wholename "/var/log/hadoop/history/.*.crc"
-wholename "/var/log/hadoop/history/done/*"
-wholename "/var/log/hadoop/history/done/.*.crc"
-wholename "/var/log/hadoop/userlogs/attempt_*"
-mtime +7
-daystart
-delete

37. Limits
sysctl.conf
fs.file-max = 128000
limits.conf
hdfs hard nofile 128000
hdfs soft nofile 64000
mapred hard nofile 128000
mapred soft nofile 64000

38. Localhost
before
127.0.0.1 localhost localhost.localdomain
127.0.1.1 hdd01.some.net hdd01
hadoop
127.0.0.1 localhost localhost.localdomain
127.0.1.1 hdd01

39. Rackaware
site config
<property>
<name>topology.script.file.name</name>
<value>/path/to/script/location-from-ip</value>
<final>true</final>
</property>
topology script
#!/usr/bin/ruby
location = {
'hdd001.some.net' => '/ams/1',
'10.20.2.1' => '/ams/1',
'hdd002.some.net' => '/ams/2',
'10.20.2.2' => '/ams/2',
}
puts ARGV.map { |ip| location[ARGV.first] || '/default-rack' }.join(' ')

40. Fix the Policy
for f in `hdfs hadoop fsck / | grep "Replica
placement policy is violated" | awk -F: '{print $1}'
| sort | uniq | head -n1000`; do
hadoop fs -setrep -w 4 $f
hadoop fs -setrep 3 $f
done

41. Fsck
hadoop fsck / -openforwrite -files | grep -i
"OPENFORWRITE: MISSING 1 blocks of total size" | awk
'{print $1}' | xargs -L 1 -i hadoop dfs -mv {} /lost
+notfound

42. Community
hadoop
* from markmail.org

43. Community
The Enterprise Effect
“The Community Effect” (in 2011)

46. Community
core
mapreduce
* from markmail.org

47. The Future
incremental
real time
refined API
flexible pipelines
refined implementation

48. Real Time Datamining and Aggregation at Scale (Ted Dunning)
Eventually Consistent Data Structures (Sean Cribbs)
Real-time Analytics with HBase (Alex Baranau)
Profiling and performance-tuning your Hadoop pipelines (Aaron Beppu)
From Batch to Realtime with Hadoop (Lars George)
Event-Stream Processing with Kafka (Tim Lossen)
Real-/Neartime analysis with Hadoop & VoltDB (Ralf Neeb)

49. Take Aways
· use hadoop only if you must
· really understand the pipeline
· unbox the black box

50. That’s it
folks!
@tcurdt
github.com/tcurdt
yourdailygeekery.com