bigdata

1. Programming with Hadoop
Presented By
Dr. Jyoti Parsola

2. Contents
• Introduction
• Characteristics
• Hadoop Components
• Word Count Example

3. Introduction
Hadoop is Apache’s open source software framework for
distributed processing and distributed storage of large datasets
across large clusters of computers build on commodity
hardware.

4. Hadoop
supports
Hadoop Running applications
on big data

5. Big Data
Large and growing files
Unstructured data
(does not fit on relational
model)
Coming from users
,application,systems,
sensors

6. Big Data Challenges
Velocity Volume Variety

7. Traditional Approach
Big data Powerful
computer
Processing

8. Hadoop Approach
Big data
Broken into piece

9. Hadoop Approach
Big data
Computation
Computation
Computation
Computation
Combined
Result

10. Hadoop Components
Map Reduce
HDFS

11. Distributed Model
Linux
Low cost
computer
Works on
linux based
machine
Linux Linux Linux

12. Distributed Model
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Task
Tracker
Data
Node
Slaves

13. Slaves
Master
Map Reduce
HDFS

14. Hadoop Framework

15. Slaves
Master
Fault Tolerant

16. Batch Processing
Slaves
Master
Application
Queue

17. Slaves
Master
Scalable
1000s

18. Hadoop Computing Model
• Two main phases: Map and Reduce
• Any job is converted into map and reduce tasks
• Developers need ONLY to implement the Map and Reduce classes
Data Flow
Shuffling and
Sorting
Blocks of
the input
file in HDFS
Map tasks (one
for each block)
Reduce tasks
Output is written
to HDFS

19. K’, list (V’)

20. Word Count

21. Hadoop tools
• Hive
• Hbase
• Pig
• Oozie
• Flume
• Scoop

22. Thank you