This document discusses Hadoop 2.0 and its key components HDFS and MapReduce. It provides an overview of HDFS, including its goals of handling hardware failures, streaming data access for batch processing of large datasets, and moving computation instead of data. HDFS architecture is also summarized, describing how it is a very large distributed file system that works with commodity hardware using file replication and recovery from failures. The document then summarizes MapReduce and how it allows parallel processing of data by splitting files into tasks processed across multiple nodes to read data locally instead of over the network.

1. Big Data: Hadoop 2.0
Map Reduce / HDFS 2.0
@diego_pacheco
Software Architect | Agile Coach

2. Big Data

11. Hadoop - Cases

12. Hadoop

13. Hadoop
Ddistributed
F ile
S ystem
4000 nodes: 14PB storage

14. HDFS – Assumptions and Goals
• Hardware Failure: Houndred or thousands machines, expect to fail.
• Streaming Data Access: Batch processing, high throughtput not low latency.
• Large Data Sets: Terrabytes, works on cluster, scale, milions of files single instance.
• Simple Coherency Model: Write-once-read-many(create, read, close, no
changes) maximize coherency and high throughtput, perfect for Map/Reduce.
• Moving Computation instead of Moving Data: Is way
more cheaper, huge data, minimize network. HDFS moves the computation close to the data.
• Sofware and hardware Portability: Easily Portable.

15. HDFS
• Very large distributed FS
• 10k nodes, 100M files, 10PB
• Works with comodity hardware
• File replication
• Detect and recover from failures
• Optimized for batch processing
• Files break by blocks 128mb
• blocks: replicated in N dataNodes
• Data Coherency
• Write Once, Read Many
• Only Append to existent files

16. HDFS - Architecture

17. HDFS 2.0 - Federation

18. Hadoop

19. Map Reduce

20. Today: Parallelism per file
Single LARGE File
Single Thread
No
Parallelism

21. Map/Reduce: Unit of data
Task 0
0..64 mb
Task 1
64..128mb
Task 2
128..192mb
Each task process a unit of data
Task 3
192..256mb

22. Today: Network issue

23. Map/Reduce: Local Read
Task 0
Task 1
Task 2
Task 3
0..64 mb
64..128mb
128..192mb
192..256mb
Node 0 Node 1 Node 2 Node 3
• Local Read, no need for network copy
• Data is read from many disks in parallel

24. Map/Reduce: The Magic!
Single Hard Drive: Reads 75mb/second
12 hard drive
Per machine
12 * 75mb/second * 4k =
3.4 TB/ second

25. Map

26. Reduce

27. Big Data: Hadoop 2.0
Map Reduce / HDFS 2.0
Obrigado!
Thank You!
@diego_pacheco
Software Architect | Agile Coach