SQL Saturday #313 Rheinland - MapReduce in der Praxis

MapReduce in der Praxis
MapReduce Entwurfsmuster
Sascha Dittmann
Blog: http://www.sascha-dittmann.de
Twitter: @SaschaDittmann

Organizer
SQLSaturday Rheinland 201428.06.2014

You Rock! Sponsor

Gold Sponsor

Silver Sponsor

Bronze Sponsor and Media Partner

WAS IST BIG DATA?
Wo liegt das Problem?
28.06.2014 SQLSaturday Rheinland 2014

Wo liegt das Problem?
"eins, zwei, zwei, drei, drei, drei, vier, vier, vier, vier"
.Split (',')
.GroupBy (x => x)
.ToList ()
.ForEach (g =>
Console.WriteLine (
“{0} wurde {1} mal gefunden.",
g.Key.Trim(),
g.Count()
));

Die 3 V’s
Variety
Velocity
• Variety (Vielfalt)
• Relational, XML, Video, Text, ...
• Velocity (Geschwindigkeit)
• Batch, Intervall, Echtzeit, ...
• Volume (Menge)
• KB, MB, GB, TB, EB, PB, ZB,
YB, ...
Volume

Skalierung
Vertikale Skalierung Horizontale Skalierung

WAS IST HADOOP / HDINSIGHT
Auf der Suche nach Lösungen

Apache Hadoop / Microsoft HDInsight
+

Apache Hadoop 1.x Ecosystem
MapReduce (Job Scheduling/Execution System)
HDFS
(Hadoop Distributed File System)
HBase (Column DB)
Pig (Data
Flow)
Hive
(Warehouse
and Data
Access)
Oozie
(Workflow)
Sqoop
Traditional BI Tools
HBase / Cassandra
(Columnar NoSQL Databases)
Avro(Serialization)
Zookeeper(Coordination)
Apache
Mahout
Cascading
(programming
model)
Hadoop = MapReduce + HDFS
Flume

Microsoft HDInsight
MapReduce (Job Scheduling/Execution System)
HDFS
(Hadoop Distributed File System)
HBase (Column DB)
Pig
(Data
Flow)
Hive
(Warehous
e and Data
Access)
Oozie
(Workflow)
Sqoop
Traditional BI Tools
HBase / Cassandra
(Columnar NoSQL Databases)
Avro(Serialization)
Zookeeper(Coordination)
Apache
Mahout
Cascading
(programming
model)
Hadoop = MapReduce + HDFS
Flume
Windows
SystemCenter
ActiveDirectory
Visual Studio

THE HADOOP CORE
HDFS + MapReduce

Hadoop Distributed File System (HDFS)
 Portable Operating System Interface (POSIX)
 Replikation auf mehrere Datenknoten
js> #ls /user/Sascha/input/ncdc
Found 9 items
drwxr-xr-x - Sascha supergroup 0 2013-04-24 13:09 /user/Sascha/input/ncdc/all
drwxr-xr-x - Sascha supergroup 0 2013-04-24 13:01 /user/Sascha/input/ncdc/all2
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/metadata
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/micro
drwxr-xr-x - Sascha supergroup 0 2013-04-23 13:06 /user/Sascha/input/ncdc/micro-tab
-rw-r--r-- 3 Sascha supergroup 529 2013-04-23 13:06 /user/Sascha/input/ncdc/sample.txt
-rw-r--r-- 3 Sascha supergroup 168 2013-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz

Map/Reduce am Beispiel von Messdaten
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
Jahr Lufttemperatur

Map/Reduce am Beispiel von Messdaten
0067011990999991950051507004+68750+023550FM-12+038299999V0203301N00671220001CN9999999N9+00001+99999999999
0043011990999991950051512004+68750+023550FM-12+038299999V0203201N00671220001CN9999999N9+00221+99999999999
0043011990999991950051518004+68750+023550FM-12+038299999V0203201N00261220001CN9999999N9-00111+99999999999
0043012650999991949032412004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+01111+99999999999
0043012650999991949032418004+62300+010750FM-12+048599999V0202701N00461220001CN0500001N9+00781+99999999999
Messqualität

Map/Reduce
Map
Sort
Shuffle
DataNode
Map
Sort
Shuffle
DataNode
Map
Sort
Shuffle
DataNode
Reduce
0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1950,22
1950,55
1952,-11
1950,33
1949,0
1950,[22,33,55]
1952,-11
1949,0
1950,55
1952,-11

Map/Reduce mit Combiner-Funktion
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Reduce
0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1950,22
1950,55
1952,-11
1950,33
1949,0
1950,55
1952,-11
1950,33
1949,0
1950,[33,55]
1952,-11
1949,0
1950,55
1952,-11

Map/Reduce mit Combiner-Funktion
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Map
Combine
Sort
Shuffle
DataNode
Reduce
0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1950,22
1950,55
1952,-11
1950,33
1949,0
1950,55
1952,-11
1950,33
1949,0
1950,[33,55]
1952,-11
1949,0
1950,55
1952,-11
Partitioner (Key % Anzahl der Reducer)

NUMERISCHE AGGREGATION

Numerische Aggregation (T-SQL)
SELECT MIN(CreationDate)
, MAX(CreationDate)
, COUNT(UserId)
FROM dbo.Comments
GROUP BY UserId

Min. / Max. / Count (MapReduce)

Min. / Max. / Count (Datenfluß)
UserId Min Max Count
12345 14.06.2001 14.06.2001 1
12345 11.01.1995 11.01.1995 1
12345 23.05.2009 23.05.2009 1
56789 23.06.1996 23.06.1996 1
56789 05.07.1997 05.07.1997 1
52737 06.12.2000 06.12.2000 1
UserId Min Max Count
12345 11.01.1995 23.05.2009 3
52737 06.12.2000 06.12.2000 1
56789 23.06.1996 05.07.1997 2
Mapper
Combiner / Reducer

Standardabweichung / Median

Standardabweichung / Median (Datenfluß)
UserId Length
4 18
4 14
4 18
3 4
3 4
9 19
Hour Length / Count Pairs
3 4:2,
4 18:2, 14:1
9 19:1
Mapper
Combiner
Reducer

Schnelles Zählen mit Countern

FILTERN

Filtern (T-SQL)
SELECT *
FROM dbo.Table
WHERE Value < 42

TOP / DISTINCT

DATEN ORGANISIEREN

Struktur ändern / Partitionen / Sortierung

JOINS

Filtern (T-SQL)
SELECT *
FROM dbo.Table1 T1
JOIN dbo.Table2 T2
ON T1.ID = T2.ID

JOINS

Code Beispiele
 C#
https://github.com/SaschaDittmann/MapReduceSamples
 Java
https://github.com/SaschaDittmann/MapReduceSamplesJava
 Blog Posts
http://www.sascha-dittmann.de/?tag=/MapReduce

Thank you!
for sponsorship
for volunteering
for participation
for a great
SQLSaturday #313

SQL Saturday #313 Rheinland - MapReduce in der Praxis

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a SQL Saturday #313 Rheinland - MapReduce in der Praxis

Semelhante a SQL Saturday #313 Rheinland - MapReduce in der Praxis (20)

Mais de Sascha Dittmann

Mais de Sascha Dittmann (17)

Último

Último (20)

SQL Saturday #313 Rheinland - MapReduce in der Praxis