hadoop architecture,hadoop,advantages of hadoop,disadvantages of hadoop,uses of hadoop,why hadoop,doug cutting,big data,hdfs,

1. BIRLA VISHVAKARMA
MAHAVIDHYALAYA
TOPIC :SUBMITTED TO : MISSES BIJAL DALWADI
SUBMITTED BY : NIKITA SURE(140080116025)
JIMMY CHOPDA(140080116013)
VRUTI TANKARIA(140080116057)
MESHWA PATEL(140080116035)
DATABASE MANAGEMENT SYSTEM

2. PART 1
What is hadoop?
History of hadoop.
Why hadoop?
Where hadoop.

3. What is Hadoop?
Apache hadoop is an open-source
software framework written in java for
distributed storage and distributed
processing of very large data set on
computer clusters built from commodity
hardware.

4. Hadoop was created by doug
cutting and mike cafarella in 2005.
Cutting who was working at yahoo !
At that time, named it after his
son’s toy elephant.
It was originally developed to
support distribution for the nutch
search engine project.
It’s latest release version is 2.7.1
on july 6,2015.
Doug Cutting

5. Hadoop - Why ?
The complexity of modern analytics needs is
outstripping the available computing power
of legacy systems. With its distributed
processing, Hadoop can handle large
volumes of structured and unstructured data
more efficiently than the traditional
enterprise data warehouse. Because Hadoop
is open source and can run on commodity
hardware, the initial cost savings are
dramatic and continue to grow as your
organizational data grows.

6.  Smart meters are deployed in homes worldwide to help consumers and utility
companies manage the use of water, electricity, and gas better. Historically,
meter readers would walk from house to house recording meter read outs and
reporting them to the utility company for billing purposes. Because of the labor
costs, many utilities switched from monthly readings to quarterly. This delayed
revenue and made it impossible to analyze residential usage in any detail.
Consider a fictional company called CostCutter Utilities that serves 10 million
households. Once a quarter, they gathered 10 million readings to produce
utility bills. With government regulation and the price of oil skyrocketing,
CostCutter started deploying smart meters so they could get hourly readings of
electricity usage. They now collect 21.6 billion sensor readings per quarter
from the smart meters. Analysis of the meter data over months and years can
be correlated with energy saving campaigns, weather patterns, and local
events, providing savings insights both for consumers and CostCutter Utilities.
When consumers are offered a billing plan that has cheaper electricity from 8
p.m. to 5 a.m., they demand five minute intervals in their smart meter reports
so they can identify high-use activity in their homes. At five minute intervals,
the smart meters are collecting more than 100 billion meter readings every 90
days, and CostCutter Utilities now has a big data problem. Their data volume
exceeds their ability to process it with existing software and hardware. So
CostCutter Utilities turns to Hadoop to handle the incoming meter readings.

7. The base Apache Hadoop framework is composed
of the following modules:
1. Hadoop Common – contains libraries and
utilities needed by other Hadoop modules
2. Hadoop Distributed File System (HDFS) – a
distributed file-system that stores data on
commodity machines
3. Hadoop YARN – a resource-management
platform responsible for managing computing
resources in clusters and using them for
scheduling of users' applications
4. Hadoop MapReduce – a programming model for
large scale data processing

8. PART 2
 Map reduce.
 HDFS.
 Introduction to hadoop architecture.

9. What is map reduce?
 Mapreduce is a processing technique and
a program model for distributed
computing based on java.
 The mapreduce algorithm contain two
important tasks, namely map and reduce.
 Map takes a set of data and converts it
into another set of data, where individual
elements are broken down into tuples.

10. What is HDFS?
 HDFS is a Java-based file system that provides
scalable and reliable data storage, and it was
designed to span large clusters of commodity
servers. HDFS has demonstrated production
scalability of up to 200 PB of storage and a single
cluster of 4500 servers, supporting close to a billion
files and blocks.
 Hadoop can work directly with any mountable
distributed file system, but the most common file
system used by Hadoop is the HDFS. It is a fault-
tolerant distributed file system that is designed for
commonly available hardware. It is well-suited for
large data sets due to its high throughput access to
application data.

11. HADOOP ARCHITECTURE
 "Hadoop employs a master/slave
architecture for both distributed storage
and distributed computation". In the
distributed storage, the NameNode is the
master and the DataNodes are the slaves.
In the distributed computation, the
Jobtracker is the master and the
Tasktrackers are the slaves

12. MASTERs
1 . NAMENODE
 The NameNode is the HEART of an HDFS file
system. It keeps the directory tree of all files in
the file system, and tracks where across the
cluster the file data is kept. It does not store the
data of these files itself.
 When the NameNode goes down, the file system
goes offline.
2 . JOBTRACKER
 The JobTracker is the service within Hadoop that
farms out MapReduce tasks to specific nodes in
the cluster, ideally the nodes that have the data,
or at least are in the same rack.
 The JobTracker is a point of failure for the
hadoop MapReduce service. If it goes down, all
running jobs are halted.

13. SLAVES
1 . DATANODE
 A DataNode stores data in the
HadoopFileSystem. A functional filesystem
has more than one DataNode, with data
replicated across them.
2 . TASKTRACKER
A TaskTracker is a node in the cluster that
accepts tasks - Map, Reduce and Shuffle
operations - from a JobTracker.

14. PART 3
 How hadoop architecture works.
 Reading files from HDFS.
 Writing files from HDFS.

18. PART 4
 Advantages of hadoop.
 Disadvantages of hadoop
 Where it is used?
 Subprojects of hadoop

19. Hadoop RelatedSubprojects
 Pig
◦ High-level language for data analysis
 HBase
◦ Table storage for semi-structured data
 Zookeeper
◦ Coordinating distributed applications
 Hive
◦ SQL-like Query language and Metastore
 Mahout
◦ Machine learning
Etc….

20. Hadoop advantages
1. Scalable
Hadoop is a highly scalable storage platform, because it can
store and distribute very large data sets across hundreds of
inexpensive servers that operate in parallel.
2. Cost effective
Hadoop is designed as a scale-out architecture that can
affordably store all of a company’s data for later use. The cost
savings are staggering: instead of costing thousands to tens of
thousands of pounds per terabyte, Hadoop offers computing
and storage capabilities for hundreds of pounds per terabyte.

21. 3. Flexible
 Hadoop can be used for a wide variety of purposes, such as
log processing, recommendation systems, data
warehousing, market campaign analysis and fraud
detection.
 4. Fast
 Hadoop’s unique storage method is based on a distributed
file system that basically ‘maps’ data wherever it is located
on a cluster. The tools for data processing are often on the
same servers where the data is located, resulting in much
faster data processing. If you’re dealing with large volumes
of unstructured data, Hadoop is able to efficiently process
terabytes of data in just minutes, and petabytes in hours.
 5. Resilient to failure
 A key advantage of using Hadoop is its fault tolerance.
When data is sent to an individual node, that data is also
replicated to other nodes in the cluster, which means that
in the event of failure, there is another copy available for
use.

22. Problems
 Coding is tedious
 Want to change that data? SQL UPDATE
and the change is in. Hadoop does not do
this.
 Hadoop stores data in files, and does not
index them. If you want to find something,
you have to run a MapReduce job going
through all the data.
 Where Hadoop works is where the data is
too big for a database!

23. What we want ?
 Guaranteed data processing
 Fault-tolerance
 No intermediate message brokers!
 Higher level abstraction than message
passing
 “Just works” !!

24. Who uses hadoop?
Hadoop is in use at most organizations that
handle big data:
 Amazon/A9
 Facebook
 Google
 IBM
 New York Times
 PowerSet
 Yahoo!

25. ANY QUESTIONS????