My Hadoop Ecosystem presentation at the 2011 BreizhCamp. See the talk video (in french): http://mediaserver.univ-rennes1.fr/videos/?video=MEDIA110628093346744

1. The
ecosystem

2. First of all...

3. Hadoop is not an acronym

4. Highly Available Data Object Oriented Program

5. Hadoop is an elephant

6. http://www.fickr.com/photos/18555810@N00/2145683109/in/photostream/

9. How it all began...

10. 1997

12. 2002

15. 2003

17. Nutch Distributed FileSystem

18. 2004

20. 2005

21. Nutch Algorithms ported to M/R

23. 2006

24. as subprojects of Lucene

25. Doug Cutting joins

26. One month later they adopt Hadoop
and set up a dedicated research team

29. 2007

33. 2008

34. Top Level Project

35. Yahoo! announces its Web index is built using Hadoop
(2008-02)

36. Hadoop sorts 1TB in 209 seconds using 910 nodes
(2008-04)

40. 2009

41. Doug Cutting joins

44. Today

45. Big Data is here to stay

46. http://salsahpc.indiana.edu/CloudCom2010/slides/PDF/tutorials/Yahoo_business_seminar.pdf

47. More and more commercial solutions

48. Hadoop Distributed FileSystem

49. Fault tolerant, very large fle support

50. Write once, read many access model

51. Hierarchical namespace

52. File permissions

53. Files are divided into blocks
Typical block size
64 or 128 MB

54. Blocks are replicated
multiple machines, multiple racks

55. Blocks are managed by DataNodes

56. DataNodes report to a NameNode

57. Write Data Flow

58. Read Data Flow

60. HDFS Thrift Server

61. Map Reduce

62. Origins in FP and λ-calculus

63. Map
Map(k,v) ⇒ list(k',v')

64. Sort
{ list(k',v') } ⇒ { (k', list(v')) }

65. Reduce
Reduce(k',list(v')) ⇒ list(k”, v”)

66. The Hello, World of MR, word count

67. MR word count
Map: <∅,line> ⇒ <word0,1> <word1,1> ... <wordk,1>
Reduce: <wordi, 10, 11, ..., 1N-1> ⇒ <wordi, N>

68. MapReduce in practice

69. Data moves slowly on the network

70. So we bring computation to the data

71. Data is read according to an InputFormat

72. An InputFormat splits the data

73. Each InputSplit is fed to a Mapper

74. HDFS blocks make ideal splits

75. Run Mappers on DataNodes

76. Each DataNode is also a TaskTracker

77. A JobTracker dispatches Mappers & Reducers

80. Mappers' output data are written locally

81. A Partitioner splits them per Reducer

82. The Reducers retrieve those partitions

83. The Shuffe

84. This step can be costly depending on volume

85. Compress Mappers' output

86. Run Combiners on Mappers' output

87. Combiners: Reducers executing at the Mappers'

88. Reducers sort data retrieved during Shuffe

89. MapReduce job ends with one fle per reducer

91. Hadoop Streaming

92. http://www.fickr.com/photos/19melissa68/2476168474/sizes/l/in/photostream/

93. Pig / Hive

95. Pig Latin is easy to learn

96. I bet you can understand the following
A = LOAD 'mydata' USING PigStorage() AS (url, time, size);
B = GROUP A BY url;
C = FOREACH B GENERATE group,
COUNT(A),
SUM(size),
SUM(time)/COUNT(A);
DUMP C;

97. Pig Latin is converted to MR jobs

98. Pig supports Streaming and UDFs

100. Offers HiveQL, close to SQL
CREATE TABLE page_view(viewTime INT, userid BIGINT,
page_url STRING, referrer_url STRING,
friends ARRAY<BIGINT>, properties MAP<STRING, STRING>
ip STRING COMMENT 'IP Address of the User')
COMMENT 'This is the page view table'
PARTITIONED BY(dt STRING, country STRING)
CLUSTERED BY(userid) SORTED BY(viewTime) INTO 32 BUCKETS
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '1'
COLLECTION ITEMS TERMINATED BY '2'
MAP KEYS TERMINATED BY '3'
STORED AS SEQUENCEFILE;
INSERT OVERWRITE TABLE xyz_com_page_views
SELECT page_views.*
FROM page_views
WHERE page_views.date >= '2008-03-01' AND page_views.date <= '2008-03-31'
AND page_views.referrer_url like '%xyz.com';

101. HiveQL converted to MR jobs

102. Hive also supports UDFs

103. MapReduce is powerful

104. But MapReduce works in batch mode

105. Need something similar for real time streams

108. ZooKeeper

109. ZooKeeper provides a highly available flesystem abstraction

110. ZooKeeper has no fles or directories but znodes

111. Znodes form a hierarchical namespace

112. A znode can have data AND children
2f 85 1e 4a 73 47 c5 e4 39 ff 0f b6 46 79 ac c5
48 c1 99 85 48 16 df 04 6a 2c cc ce 9e 4f ae cb
20 a5 9d 62 57 96 35 c3 eb 3d cb c3 1c cb 91 f8
a2 4d 90 57 0a 62 24 f9 5e a4 50 00 6a bd 3c ea
68 61 3f bf 7a 48 8f 26 63 24 e9 d4 3b b4 55 c2

113. A znode has Access Control Lists
CREATE (children)
READ (list children and get data)
WRITE (set data)
DELETE (children)
ADMIN (setACL)

114. Znodes can be persistent or ephemeral
Ephemeral znodes vanish with their creator's session
Persistent znodes outlive their creator's session
Ephemeral znodes cannot have children

115. Znodes can be sequential
A monotonically increasing counter is appended to the znode name
/zk/foo-1
/zk/foo-3
/zk/foo-4
...
This can be used to impose a global order among direct children

116. Znodes can have watches
Watches allow clients to be notifed when znodes change
NodeCreated
NodeDeleted
NodeDataChanged
NodeChildrenChanged

117. ZooKeeper has a simple API
exists
getData
getChildren } set watches
create
delete
setData
setACL
getACL
sync

118. ZooKeeper consistency
Sequential Consistency
Updates from a client are applied in the order sent
Atomicity
Updates either succeed or fail
Single System Image
Unique view, regardless of the server we connect to
Durability
Updates once succeeded will not be undone
Timeliness
Lag is bounded

119. ZooKeeper use cases
Confguration service
Get latest confg and get notifed when it changes
Lock service
Provide mutual exclusion
Leader election
There can be only one...
Group membership
Dynamically determine members of a group
Queue
Producer/Consumer paradigm

122. http://www.fickr.com/photos/7-how-7/2308008668/sizes/o/in/photostream/

123. HBase

125. Data Model

126. “a sparse, distributed multi-dimensional sorted map”

127. Row keys, column qualifers, values
byte[]

128. Regions

129. A region is a slice of row keys

131. A region is served by a Region Server

132. HBase Master

133. Coordinates the slaves (Region Servers)

134. Assigns Regions to Region Servers

135. Does so using ZooKeeper

136. .META.

137. Special table, stores region assignments

138. -ROOT-

139. Special single region table

140. Stores .META. region assignments

141. Region Server

143. MemStore fushed when full

144. New immutable fles written

145. When too many fles per region

146. Perform a minor compaction

147. Merge most recent N fles

148. Periodically merge all fles in a region

149. That's a major compaction

150. When aggregate size of a region is too big

151. Split region into two new regions

152. Parent region fles eventually discared

153. Region Server Crashes

154. Master will split WAL into Region oldlogs

155. Regions will be reassigned

156. HBase API

157. Put, Get, Delete

158. CheckAndPut, Lock

159. Counters

160. Scan

161. Filters, row/column, in Region Server

162. Coprocessors

163. REST / Thrift gateways for non Java access

164. Map Reduce
NextGen

165. NodeManager

166. Resource Containers

167. Scheduler

168. Application Manager

169. More generic and more scalable

171. Monitor Everything

172. Automate as much as possible

173. Test Drive on

174. We're hiring
www.arkea.com

175. @herberts