Replacing Telco DB/DW to Hadoop and Hive

Replacing Telco DB/DW to
Hadoop and Hive

JunHo Cho

Data Analysis Platform Team

Friday, July 1, 2011

• Cloud Computing Platform - Xen

• Cloud Storage Platform - hadoop

• Massive Email Archiving Solution - hadoop, lucene

• HIVE : social network analysis using email

• Log Archiving Solution - hadoop

• Data Analysis
data mining, machine learning, data statistic

• Data Platform - hadoop, lucene, hive

• Cloud Architecture - KT Cloud


Telco Data


OpenSource


OpenSource

Storage & Computing


OpenSource

Collection


OpenSource

Search


OpenSource

Analysis


OpenSource

Coordination


Hive Internal


Hive Architecture

UI Driver

DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result


Hive Architecture

UI Driver select col1 from tab1 where ...

DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result


Hive Architecture
a 123344
b 121211
c 342434

UI Driver

DDL HQL
Execution
Works
Engine
MetaStore Compiler
ORM Hadoop
Result


Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF/UDAF
SELOperator
substr
sum
MetaStore Hive QL FSOperator average

Thrift API Parser ExecMapper/ExecReducer
Plan SerDe

Optimizer Input/OutputFormat

Task
HDFS StorageHandler
RCFile
DB ... HBase


Parser
Parser
Select col1,col2 From tab1 Where col3 > 5

TOK_QUERY

TOK_FROM TOK_INSERT

TOK_DESTINATION TOK_SELECT TOK_WHERE

TOK_TABNAME
TOK_SELEXPR TOK_SELEXPR

TOK_DIR
>

TOK_TABLE_OR_COL TOK_TABLE_OR_COL

TOK_TMP_FILE
TOK_TABLE_OR_COL 5


Parser
Parser

QB
TOK_QUERY

TOK_FROM TOK_INSERT


TOK_TABNAME

TOK_DIR
>


TOK_TMP_FILE
TOK_TABLE_OR_COL 5


Parser
Parser

TOK_QUERY

TOK_FROM TOK_INSERT


TOK_TABNAME
QB tab1
TOK_DIR
>


TOK_TMP_FILE
TOK_TABLE_OR_COL 5


Parser
Parser

TOK_QUERY

TOK_FROM TOK_INSERT


TOK_TABNAME
tab1
TOK_DIR
>


TOK_TMP_FILE
TOK_TABLE_OR_COL 5

QB insclause-0


Parser
Parser

TOK_QUERY

TOK_FROM TOK_INSERT


TOK_TABNAME
tab1
TOK_DIR
>


TOK_TMP_FILE
col1 QB
TOK_TABLE_OR_COL 5

insclause-0


Parser
Parser

TOK_QUERY

TOK_FROM TOK_INSERT


TOK_TABNAME
tab1
TOK_DIR
>


col1 col2 QB
TOK_TMP_FILE
TOK_TABLE_OR_COL 5

insclause-0


Parser
Parser

TOK_QUERY

TOK_FROM TOK_INSERT

TOK_DESTINATION TOK_SELECT TOK_WHERE QB
TOK_TABNAME
tab1
TOK_DIR
>


col1 col2
TOK_TMP_FILE
TOK_TABLE_OR_COL 5

insclause-0


Plan
Plan

QB


Plan
Plan

QB

TOK_FROM

TOK_WHERE

TOK_SELECT

TOK_DESTINATION


Plan
Plan

QB

TOK_FROM TableScanOperator

TOK_WHERE

TOK_SELECT

TOK_DESTINATION


Plan
Plan

QB


TOK_WHERE FilterOperator

TOK_SELECT

TOK_DESTINATION


Plan
Plan

QB



TOK_SELECT SelectOperator

TOK_DESTINATION


Plan
Plan

QB



TOK_SELECT SelectOperator

TOK_DESTINATION FileSinkOperator


Optimizer
Optimizer Select col1,col2 From tab1 Where col3 > 5

TableScanOperator

FilterOperator

SelectOperator

FileSinkOperator


Optimizer

tab1 {col1, col2, col3, col4,col5,col6,col7}

TableScanOperator

FilterOperator

SelectOperator

FileSinkOperator


Optimizer

Context

TableScanOperator

FilterOperator
ColumnPruner

SelectOperator

FileSinkOperator


Optimizer

Context

TableScanOperator

FilterOperator FIL
ColumnPruner TS
SEL
SelectOperator

FileSinkOperator


Optimizer


TableScanOperator

FilterOperator FIL
ColumnPruner TS
SEL
SelectOperator

FileSinkOperator Context


Optimizer


TableScanOperator

FilterOperator
ColumnPruner

SelectOperator
FIL
FileSinkOperator Context TS
SEL


Optimizer


TableScanOperator

FilterOperator
ColumnPruner

FIL
SelectOperator Context TS
SEL

FileSinkOperator


Optimizer


TableScanOperator

FilterOperator
ColumnPruner

FIL
SelectOperator Context TS
SEL col1, col2

FileSinkOperator


Optimizer


TableScanOperator

FIL col1, col2, col3
FilterOperator Context TS
ColumnPruner
SEL

SelectOperator

FileSinkOperator


Optimizer


TableScanOperator

FIL
FilterOperator Context TS
ColumnPruner
SEL

SelectOperator

FileSinkOperator


Optimizer


FIL
TableScanOperator Context TS col1, col2, col3
SEL

FilterOperator
ColumnPruner

FilterOperator

SelectOperator

FileSinkOperator


Task
Task Select col1,col2 From tab1 Where col3 > 5

TS - GenMRTableScan1
TaskFactory
FS - GenMRFileSink1
QB


Task

TaskFactory
FS - GenMRFileSink1
QB

FetchTask


Task

TaskFactory
FS - GenMRFileSink1
QB

TableScanOperator

FilterOperator FetchTask

FilterOperator

SelectOperator

FileSinkOperator


Task

TaskFactory
FS - GenMRFileSink1
QB
MapRedTask

TableScanOperator


FilterOperator

SelectOperator

FileSinkOperator


Task

TaskFactory

QB
MapRedTask

TableScanOperator


FilterOperator

SelectOperator

FileSinkOperator


Task

TaskFactory

QB
MapRedTask
MapRedTask
TableScanOperator


FilterOperator

SelectOperator

FileSinkOperator


Hive Internal
Map Reduce
Web UI Hive CLI JDBC
TSOperator User Script
Browse, Query, DDL
UDF
FILOperator SELOperator

MetaStore Hive QL FILOperator FSOperator

Thrift API Parser ExecMapper/ExecReducer
Plan SerDe

Optimizer Input/OutputFormat

Task
HDFS StorageHandler
RCFile
DB ... HBase


Oracle Migration
to Hive


l
l

l

l


l l
l l
l l
l l


Understand Oracle SQL

• more than 3000 ETL SQL
• understand Data-Flow
• Group similar SQL Pattern
• Investigate used Oracle Function


Oracle SQL


Data Model Convert


Data Model Convert

Table


Data Model Convert

Table Table


Data Model Convert

Table Table

Partition


Data Model Convert

Table Table

Partition Partition


Data Model Convert

Table Table

Partition Partition

Sampling


Data Model Convert

Table Table

Partition Partition

Sampling Bucket


DataType Convert


DataType Convert

NUMBER(n)


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT

NUMBER(n,m)


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT

NUMBER(n,m) FLOAT/DOUBLE


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT


VARCHAR2


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT


VARCHAR2 STRING


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT


VARCHAR2 STRING

DATE


DataType Convert

NUMBER(n) TINYINT
INT/BIGINT


VARCHAR2 STRING

DATE STRING
“yyyy-MM-dd HH:mm:ss” format


HIVE DML

• HIVE supports ANSI-SQL
• Only Support Sub-Queries in FROM clause
• Join query : equi-join/inner-join
outer-join
self-join


IN Clause


IN Clause
IN SubQuery


IN Clause
IN SubQuery
SELECT * from Employee e WHERE e.DeptNo

IN(SELECT d.DeptNo FROM Dept d)


IN Clause
IN SubQuery

IN(SELECT d.DeptNo FROM Dept d)

SELECT * from Employee e

LEFT SEMI JOIN Dept d ON (e.DeptNo=d.DeptNo)


NOT IN Clause


NOT IN Clause
NOT IN SubQuery


NOT IN Clause
NOT IN SubQuery

NOT IN(SELECT d.DeptNo FROM Dept d)


NOT IN Clause
NOT IN SubQuery

NOT IN(SELECT d.DeptNo FROM Dept d)

SELECT e.* from Employee e

LEFT OUTER JOIN Dept d ON (e.DeptNo=d.DeptNo)

WHERE d.DeptNo IS NULL


JOIN Operator


JOIN Operator
JOIN


JOIN Operator
JOIN
SELECT *

FROM Employee e1, Dept d1 WHERE e1.ID = d1.Id


JOIN Operator
JOIN
SELECT *

FROM Employee e1, Dept d1 WHERE e1.ID = d1.Id

SELECT *

FROM Employee e1 JOIN Dept d1 ON (e1.ID = d1.Id )


Oracle Function


Functions


Functions

Math Function
round,ceil,mod,
power,sqrt,sin/cos


Functions

Math Function Math Function
round,ceil,mod, round,ceil,pmod,
power,sqrt,sin/cos power,sqrt,sin/cos


Functions


Character Function
substr,trim,lpad/rpad
ltrim/rtrim,replace


Functions


Character Function Character Function
substr,trim,lpad/rpad substr,trim,lpad/rpad
ltrim/rtrim,replace ltrim/rtrim,regexp_replace


Functions



NULL Function
coalesce,nvl,nvl2


Functions



NULL Function NULL Function
coalesce,nvl,nvl2 coalesce


Functions



NULL Function NULL Function
coalesce,nvl,nvl2 coalesce

No NVL,NVL2

Custom UDF Function
• Condition Function

• DECODE, GREATEST

• Null Comparison Function

• NVL / NVL2

• Type Conversion

• TO_NUMBER

• TO_CHAR

• TO_DATE

• INSTR4

• DATE_FORMAT

• LAST_DAY


Oracle Analytic
Function


Analytic Function


Analytic Function
RANK


Analytic Function
RANK
SELECT name,dept,salary,RANK() OVER (PARTITION BY dept
ORDER BY salary DESC) FROM emp


Analytic Function
RANK

SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary)
FROM (SELECT name, dept, salary FROM empDISTRIBUTED
BY dept SORT BY dept, salary DESC) e


Analytic Function
RANK

RANK(arg1,arg2) - Custom UDF
SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary)
FROM (SELECT name, dept, salary FROM empDISTRIBUTED
BY dept SORT BY dept, salary DESC) e


Analytic Aggregation Function


MIN


MIN
SELECT dept, MIN(salary) OVER (PARTITION BY dept)
FROM emp


MIN
FROM emp

SELECT dept,tmp.m FROM emp JOIN (SELECT dept, MIN(salary) m
FROM emp GROUP BY dept) tmp ON emp.dept = tmp.dept


MIN
FROM emp

Aggregation + JOIN
SELECT dept,tmp.m FROM emp JOIN (SELECT dept, MIN(salary) m
FROM emp GROUP BY dept) tmp ON emp.dept = tmp.dept


Merge Join Tree Bug

• select * from a join b on a.v1 = b.v1
join c on a.v1 = c.v1
join d on a.v1 = d.v1
join e on a.v2 = e.v2

• select * from a join e on a.v2 = e.v2
join b on a.v1 = b.v1


Merge Join Tree Bug

join d on a.v1 = d.v1 educ e #3
M a pR



Merge Join Tree Bug

join d on a.v1 = d.v1 educ e #3
M a pR

join d on a.v1 = d.v1 duc e #2
Ma pRe


Merge Join Tree Bug Fix
• SemanticAnalyzer
private void mergeJoinTree(QB qb) {

QBJoinTree root = qb.getQbJoinTree();
QBJoinTree parent = null;
while (root != null) {
boolean merged = mergeJoinNodes(qb, parent, root, root.getJoinSrc());

if (parent == null) {
if (merged) {
root = qb.getQbJoinTree();
} else {
parent = root;
root = root.getJoinSrc();
}

} else {
parent = parent.getJoinSrc();
root = parent.getJoinSrc();
}


Merge Join Tree Bug Fix
• SemanticAnalyzer
private void mergeJoinTree(QB qb) {

QBJoinTree root = qb.getQbJoinTree();
QBJoinTree parent = null;
while (root != null) {
boolean merged = mergeJoinNodes(qb, parent, root, root.getJoinSrc());

if (parent == null) {
if (merged) {
} else {
parent = root;
root = root.getJoinSrc();
}

} else {
} else {
if parent = parent.getJoinSrc();
(merged) {
} } else {
parent = parent.getJoinSrc();
}
}


New HQL Syntax


New HQL Syntax
INSERT INTO


New HQL Syntax
INSERT INTO
INSERT INTO table VALUES(col1 ... coln)
SELECT ... FROM tmp ...


New HQL Syntax
INSERT INTO
INSERT INTO table VALUES(col1 ... coln)
SELECT ... FROM tmp ...

• INSERT [OVERWRITE] destination
• grammar
• modify FileSinkPlan
• New Feature - HIVE-306
• INSERT INTO destination

Tuning


Tuning
• Hadoop Tunning


Tuning
• Hadoop Tunning
• mapred.job.reuse.jvm.num.task


Tuning
• Hadoop Tunning

• mapred.child.java.opts


Tuning
• Hadoop Tunning


• mapred.min.split.size / mapred.max.split.size


Tuning
• Hadoop Tunning



• dfs.block.size


Tuning
• Hadoop Tunning



• dfs.block.size

• Hive Tunning


Tuning
• Hadoop Tunning



• dfs.block.size

• Hive Tunning
• hive.input.format = CombineHiveInputFormat


Tuning
• Hadoop Tunning



• dfs.block.size

• Hive Tunning
• hive.input.format = CombineHiveInputFormat

• query tuning - reduce # of MapReduce
using HQL Plan


Wrap-Up
Oracle 2 Hive


Wrap-Up
Oracle 2 Hive
Look insight data ﬂow & model


Wrap-Up
Oracle 2 Hive
Modify Oracle SQL to Hive Query Syntax


Wrap-Up
Oracle 2 Hive
Use Built-in function


Wrap-Up
Oracle 2 Hive
Develop custom UDF/UDAF/UDTF


Wrap-Up
Oracle 2 Hive
Support analytic function


Wrap-Up
Oracle 2 Hive
- distributed by + sort by + udf


Wrap-Up
Oracle 2 Hive
- join + udf (aggregation)


Wrap-Up
Oracle 2 Hive
Modify internal hive


Wrap-Up
Oracle 2 Hive
Modify internal hive
Hadoop + Hive Tunning


Question ?


Replacing Telco DB/DW to Hadoop and Hive

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a Replacing Telco DB/DW to Hadoop and Hive

Semelhante a Replacing Telco DB/DW to Hadoop and Hive (20)

Último

Último (20)

Replacing Telco DB/DW to Hadoop and Hive