Hive Apachecon 2008

HIVE Data Warehousing & Analytics on Hadoop Ashish Thusoo, Facebook Data Team

Why Another Data Warehousing System? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What is HIVE? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive/Hadoop Usage @ Facebook ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Warehousing at Facebook Today Web Servers Scribe Servers Filers Hive on Hadoop Cluster Oracle RAC Federated MySQL

HIVE: Components HDFS Hive CLI DDL Queries Browsing Map Reduce MetaStore Thrift API SerDe Thrift Jute JSON.. Execution Hive QL Parser Planner Web UI

Data Model Logical Partitioning Hash Partitioning Schema Library clicks HDFS MetaStore / hive/clicks /hive/clicks/ds=2008-03-25 /hive/clicks/ds=2008-03-25/0 … Tables #Buckets=32 Bucketing Info Partitioning Cols

MetaStore ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive Query Language ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

(Simplified) Map Reduce Review Machine 2 Machine 1 <k1, v1> <k2, v2> <k3, v3> <k4, v4> <k5, v5> <k6, v6> <nk1, nv1> <nk2, nv2> <nk3, nv3> <nk2, nv4> <nk2, nv5> <nk1, nv6> Local Map <nk2, nv4> <nk2, nv5> <nk2, nv2> <nk1, nv1> <nk3, nv3> <nk1, nv6> Global Shuffle <nk1, nv1> <nk1, nv6> <nk3, nv3> <nk2, nv4> <nk2, nv5> <nk2, nv2> Local Sort <nk2, 3> <nk1, 2> <nk3, 1> Local Reduce

Hive QL – Join ,[object Object],[object Object],[object Object],[object Object],X = page_view user pv_users pageid userid time 1 111 9:08:01 2 111 9:08:13 1 222 9:08:14 userid age gender 111 25 female 222 32 male pageid age 1 25 2 25 1 32

Hive QL – Join in Map Reduce page_view user pv_users Map Shuffle Sort Reduce key value 111 < 1, 1> 111 < 1, 2> 222 < 1, 1> pageid userid time 1 111 9:08:01 2 111 9:08:13 1 222 9:08:14 userid age gender 111 25 female 222 32 male key value 111 < 2, 25> 222 < 2, 32> key value 111 < 1, 1> 111 < 1, 2> 111 < 2, 25> key value 222 < 1, 1> 222 < 2, 32> Pageid age 1 25 2 25 pageid age 1 32

Joins ,[object Object],[object Object],[object Object],[object Object],[object Object]

Join To Map Reduce ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Joining multiple tables in a single Map Reduce ,[object Object],[object Object],[object Object],A Map Reduce B C AB Map Reduce ABC key av bv 1 111 222 key av 1 111 key bv 1 222 key cv 1 333 key av bv cv 1 111 222 333

Hive QL – Group By ,[object Object],[object Object],[object Object],pv_users pageid age 1 25 2 25 1 32 2 25 pageid age count 1 25 1 2 25 2 1 32 1

Hive QL – Group By in Map Reduce pv_users Map Shuffle Sort Reduce pageid age 1 25 2 25 pageid age count 1 25 1 1 32 1 pageid age 1 32 2 25 key value <1,25> 1 <2,25> 1 key value <1,32> 1 <2,25> 1 key value <1,25> 1 <1,32> 1 key value <2,25> 1 <2,25> 1 pageid age count 2 25 2

Hive QL – Group By with Distinct ,[object Object],[object Object],page_view pageid userid time 1 111 9:08:01 2 111 9:08:13 1 222 9:08:14 2 111 9:08:20 pageid count_distinct_userid 1 2 2 1

Hive QL – Group By with Distinct in Map Reduce page_view Shuffle and Sort Reduce Map Reduce pageid count 1 1 2 1 pageid count 1 1 pageid userid time 1 111 9:08:01 2 111 9:08:13 pageid userid time 1 222 9:08:14 2 111 9:08:20 key v <1,111> <2,111> <2,111> key v <1,222> pageid count 1 2 pageid count 2 1

Group by Optimizations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Inserts into Files, Tables and Local Files ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extensibility - Custom Map/Reduce Scripts ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Custom ‘map_script’ in map side TRANSFORM Custom ‘reduce_script’ in reduce side TRANSFORM Key dt between map and reduce using CLUSTER BY

Extensibility – User Defined Functions interface UDF class com.example.myUDF implements UDF myUDF.jar hive.aux.jars CREATE TEMPORARY FUNCTION myudf AS ‘com.example.myUDF’; SELECT myudf(t.c1) FROM t; drop in the jar register udf use udf use udf

Extensibility- User Defined Data Formats ,[object Object],101010…. interface serde2.DeSerialize class com.example.myDeSerializer extends serde2.DeSerialize class com.example.mySerializer extends serde2.Serialize RowObject hive.aux.jars plug & play myExtensions.jar ObjectInspector

Extensibility – User Defined Types ,[object Object],struct { map<string, smallint> f11; struct { list<double> f21; int f22; } f12; } new StandardStructObjectInspector f11: f12: new StandardStructObjectInspector f21: f22: PrimitiveObjectInspector(int) new StandardMapObjectInspector key: PrimitiveObjectInspector(string) val: PrimitiveObjectInspector(smallint) new StandardListObjectInspector elem: PrimitiveObjectInspector(double) myExtensions.jar drop in the jar

Extensibility- Reading Rich Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extensibility- Reading Rich Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Interoperability – External Tables ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Interoperability – JDBC driver ,[object Object],[object Object],External Application (e.g. BI) Hive JDBC Driver MetaStore NameNode JobTracker Hadoop Cluster

Evolving Open Source Community ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Future Work – Longer Term ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Information ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Most Importantly – People @fb ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive Apachecon 2008

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Hive Apachecon 2008

Semelhante a Hive Apachecon 2008 (20)

Último

Último (20)

Hive Apachecon 2008