Oracle in Database Hadoop

<Insert Picture Here>

Oracle In-Database Hadoop:
When MapReduce Meets RDBMS
Kuassi Mensah | db360.blogspot.com @kmensah
Director Product Management | kuassi.mensah@oracle.com

The following is intended to outline our general product direction. It
is intended for information purposes only, and may not be
incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in
making purchasing decisions. The development, release, and
timing of any features or functionality described for Oracle s
products remains at the sole discretion of Oracle.

Hadoop Summit 2012, June 13-14, San Jose, California, USA Oracle In-Database Hadoop: When MapReduce Meets RDBMS. Kuassi Mensah

2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8

Oracle In-Database Hadoop
Agenda

•  In-Database MapReduce
•  Why
•  Previous Initiatives and Limitations
•  Oracle In-Database Hadoop
•  Integration with Oracle’s Big Data solution
•  Summary


In-Database MapReduce


MapReduce Paradigm
You All Know This Stuff!

Map:
<K1,V1>
→

{<K2,V2>,…}

Shuffle:
{<K2,V2>,
…}
→

{<K2,{V2,…,V2}>,…}

Reduce:
<K2,{V2,…,V2}>

→
{<K3,V3>,…}


Why?

•  Avoid shipping data residing in RDBMS to a
separate infrastructure.
•  Many initiatives
•  Address top two issues preventing broader
adoption of Hadoop in the enterprise
•  Lack of development and/or administration skills
•  Lack of enterprise-class security


Previous Efforts and Limitations

•  SQL-MapReduce,HadoopDB (Hadapt), etc.
•  PL/SQL User-defined pipelined table functions
and aggregation objects
•  Limitations
•  Lack of compatibility with Hadoop
•  Loose integration with Hadoop
•  Dependency on Hadoop infrastructure


Oracle In-Database Hadoop is
a prototype (not a feature of
Oracle products), built on
current Oracle products.


Goals

•  Avoid shipping data residing in Oracle database to
Hadoop clusters.
•  Preserve Hadoop programming model
•  Reduce dependency on Hadoop infrastructure
•  Get enterprise developers up to speed with minimal
training
•  Get enterprise administrators (DBAs) up to speed
with minimal training
•  Reduce deployment time
•  Bring enterprise class security to MapReduce
•  Seamless integration with Oracle’s Big Data
solution


Compatibility & Minimal Dependency on Hadoop Infra
Node 1 Node 2 Node 3
Pipelelined Table
Function w Java
impl.
Mapping Process Mapping Process Mapping Process

PARTITION by
CLUSTER BY Clause

Node 1 Node 2 Node 3

Pipelined Table
Function w Java impl.
Reducing Process Reducing Process Reducing Process


Preserve Hadoop Programming Model
•  Source-compatibility
•  Job configuration
•  Invocation thru Java interface: job.run()
•  Direct table access: TableReader and TableWriter


SQL and MapReduce Integration

•  Mix SQL and MapReduce processing for flexibility and
efficiency.
•  MapReduce steps as pipelined table functions.

INSERT
INTO
OutTable

SELECT
*
FROM
TABLE

(Word_Count_Reduce(:ConfKey,

CURSOR(SELECT
*
FROM
TABLE

(Word_Count_Map(:ConfKey,

CURSOR(SELECT
*
FROM
InTable))))))


SQL and Java interfaces

SELECT
*
FROM
TABLE

public
class
WordCount
{

(Reduce_VARCHAR2_NUMBER(:ConfKey,

public
static
void
main()
throws
Exception
{

/*
Setup
the
parameters
and
run
the
job
*/

CURSOR(SELECT
*
FROM
TABLE

……

(Map_VARCHAR2_NUMBER(:ConfKey,

job.init();

CURSOR(SELECT
*
from
InTable))))))

job.run();

}


Leverage Enterprise Skills
•  Get database developers up to speed, with minimal
training, on developing MapReduce jobs by reusing
Hadoop Mappers and Reducers
•  Get DBAs up to speed on deploying and managing
MapReduce jobs with minimal training


Oracle Database Security
Bringing Enterprise Class Security to MapReduce

•  Auditing and Monitoring
•  Database Activity Auditing
•  Database Firewall Monitoring
•  Centralized Audit Data Warehouse
•  Encryption and Masking
•  Transparent Data Encryption
•  Network Encryption/Strong Auth
•  Data Masking for Non-Production
•  Privileged User Access Control and Contextual
Authorization
•  Separation of Duties for DBAs
•  Protection Realms & Rules
•  Label Based Access Control


Seamless integration with Oracle’s Big
Data Solution


Oracle’s Big Data solution
Endeca Information Discovery

Oracle
Big Data
Oracle
Appliance
Exadata
Oracle
Exalytics

InfiniBand InfiniBand

Oracle
Real-Time
Decisions

Acquire Organize & Discover Analyze Decide


Oracle Direct Connector for HDFS

Direct Access from
HDFS Oracle Database Oracle Database
SQL Query

SQL access to HDFS
External
Table External table view

Data query or import

DCH HDFS
Infini
Band
DCH
DCH
Client


Oracle In-Database Analytics

Oracle Advanced
Analytics
Statistical
Data Mining
Text
Graph
Spatial
Semantic

2 miles


What Have We Done?


Summary
A prototype:
•  Apply MapReduce processing to data in
Oracle RDBMS without the need of a
separate infrastructure.
•  Compatibility with Hadoop while
minimizing dependency on the Apache
Hadoop infrastructure.
•  Reduce training and deployment time.
•  Integration with Oracle SQL, allowing
mixing MapReduce steps with
sophisticated SQL queries.
•  Bring Enterprise Class Security to
Hadoop MapReduce
•  Seamless integration with Oracle’s Big
Data solution


Demo


Oracle in Database Hadoop

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (7)

Semelhante a Oracle in Database Hadoop

Semelhante a Oracle in Database Hadoop (20)

Mais de DataWorks Summit

Mais de DataWorks Summit (20)

Último

Último (20)

Oracle in Database Hadoop