ORC File Introduction

•Transferir como PPTX, PDF•

13 gostaram•11,815 visualizações

Owen O'Malley

I present the Optimized Row Columnar (ORC) file format for Apache Hive.

Tecnologia

ORC Files

Owen O’Malley
owen@hortonworks.com

December 2012

© Hortonworks Inc. 2012 Page 1

Top Level

Page 2
© Hortonworks Inc. 2012

File Structure

Page 3
© Hortonworks Inc. 2012

Stripe Structure

Page 4
© Hortonworks Inc. 2012

File Layout

Page 5
© Hortonworks Inc. 2012

Integer Column Serialization

Page 6
© Hortonworks Inc. 2012

String Column Serialization

Page 7
© Hortonworks Inc. 2012

Compression

Page 8
© Hortonworks Inc. 2012

Projection and Predicate Filtering

Page 9
© Hortonworks Inc. 2012

Example File Sizes

Page 10
© Hortonworks Inc. 2012

Final notes

Page 11
© Hortonworks Inc. 2012

Comparison

RC File Trevni ORC File
Hive Type Model N N Y
Separate complex columns N Y Y
Splits found quickly N Y Y
Default column group size 4MB 64MB* 250MB
Files per a bucket 1 >1 1
Store min, max, sum, count N N Y
Versioned metadata N Y Y
Run length data encoding N N Y
Store strings in dictionary N N Y
Store row count N Y Y
Skip compressed blocks N N Y
Store internal indexes N N Y

Page 12
© Hortonworks Inc. 2012

Mais conteúdo relacionado

Mais procurados

File Format Benchmarks - Avro, JSON, ORC, & ParquetOwen O'Malley

File Format Benchmark - Avro, JSON, ORC & ParquetDataWorks Summit/Hadoop Summit

HiveManas Nayak

Inside Parquet FormatYue Chen

Chicago Data Summit: Apache HBase: An IntroductionCloudera, Inc.

Migrating your clusters and workloads from Hadoop 2 to Hadoop 3DataWorks Summit

Strongly Consistent Global Indexes for Apache PhoenixYugabyteDB

ORC File and Vectorization - Hadoop Summit 2013Owen O'Malley

Hadoop HDFS.ppt6535ANURAGANURAG

ORC File - Optimizing Your Big DataDataWorks Summit

SeaweedFS introductionchrislusf

Unit 5-apache hivevishal choudhary

What is new in Apache Hive 3.0?DataWorks Summit

Apache Sqoop: A Data Transfer Tool for HadoopCloudera, Inc.

Hadoop Overview & Architecture EMC

How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit

Apache hivepradipbajpai68

Compression Options in Hadoop - A Tale of TradeoffsDataWorks Summit

HbaseMilton Bahia

Introduction to sqoopUday Vakalapudi

Mais procurados (20)

File Format Benchmarks - Avro, JSON, ORC, & Parquet

File Format Benchmark - Avro, JSON, ORC & Parquet

Hive

Inside Parquet Format

Chicago Data Summit: Apache HBase: An Introduction

Migrating your clusters and workloads from Hadoop 2 to Hadoop 3

Strongly Consistent Global Indexes for Apache Phoenix

ORC File and Vectorization - Hadoop Summit 2013

Hadoop HDFS.ppt

ORC File - Optimizing Your Big Data

SeaweedFS introduction

Unit 5-apache hive

What is new in Apache Hive 3.0?

Apache Sqoop: A Data Transfer Tool for Hadoop

Hadoop Overview & Architecture

How to understand and analyze Apache Hive query execution plan for performanc...

Apache hive

Compression Options in Hadoop - A Tale of Tradeoffs

Hbase

Introduction to sqoop

Destaque

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon

Structor - Automated Building of Virtual Hadoop ClustersOwen O'Malley

Adding ACID Updates to HiveOwen O'Malley

Protecting Enterprise Data in Apache HadoopOwen O'Malley

Data protection2015Owen O'Malley

Plugging the Holes: Security and Compatability in HadoopOwen O'Malley

Next Generation MapReduceOwen O'Malley

Bay Area HUG Feb 2011 IntroOwen O'Malley

Next Generation Hadoop OperationsOwen O'Malley

Optimizing Hive QueriesOwen O'Malley

Hadoop Security ArchitectureOwen O'Malley

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

Mapreduce total order sorting techniqueUday Vakalapudi

Hive integration: HBase and Rcfile__HadoopSummit2010Yahoo Developer Network

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

ORC File & Vectorization - Improving Hive Data Storage and Query PerformanceDataWorks Summit

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

Hive and Apache Tez: Benchmarked at Yahoo! ScaleDataWorks Summit

Optimizing Hive QueriesDataWorks Summit

Destaque (20)

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

Structor - Automated Building of Virtual Hadoop Clusters

Adding ACID Updates to Hive

Protecting Enterprise Data in Apache Hadoop

Data protection2015

Plugging the Holes: Security and Compatability in Hadoop

Next Generation MapReduce

Bay Area HUG Feb 2011 Intro

Next Generation Hadoop Operations

Optimizing Hive Queries

Hadoop Security Architecture

Strata London 2016: The future of column oriented data processing with Arrow ...

Sql on everything with drill

Mapreduce total order sorting technique

Hive integration: HBase and Rcfile__HadoopSummit2010

Strata NY 2016: The future of column-oriented data processing with Arrow and ...

ORC File & Vectorization - Improving Hive Data Storage and Query Performance

Data Eng Conf NY Nov 2016 Parquet Arrow

Hive and Apache Tez: Benchmarked at Yahoo! Scale

Optimizing Hive Queries

Semelhante a ORC File Introduction

Inside hadoop-devSteve Loughran

Availability and Integrity in hadoop (Strata EU Edition)Steve Loughran

HA Hadoop -ApacheCon talkSteve Loughran

Mobile Development Meets Semantic TechnologyBlue Slate Solutions

Orange County HUG - Agile Data on HDPHortonworks

An Introduction to Spring DataOliver Gierke

LA HUG - Agile Analytics Applications on HDPHortonworks

Hadoop: today and tomorrowSteve Loughran

Cloud Consolidation with Oracle (RAC) - How much is too much?Markus Michalewicz

Sentri SharePoint Performance webinarSentri

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?Laura Hood

Agile analytics applications on hadoopHortonworks

Hortonworks: Agile Analytics Applicationsrussell_jurney

Compaction and Splitting in Apache AccumuloHortonworks

Introduction to DDDRadosław Mejer

ORC: 2015 Faster, Better, SmallerDataWorks Summit

Ozone and HDFS’s evolutionDataWorks Summit

ORC 2015t3rmin4t0r

Storage Characteristics Of Call Data Records In Column Store DatabasesDavid Walker

Java and MongoMarcio Mangar

Semelhante a ORC File Introduction (20)

Inside hadoop-dev

Availability and Integrity in hadoop (Strata EU Edition)

HA Hadoop -ApacheCon talk

Mobile Development Meets Semantic Technology

Orange County HUG - Agile Data on HDP

An Introduction to Spring Data

LA HUG - Agile Analytics Applications on HDP

Hadoop: today and tomorrow

Cloud Consolidation with Oracle (RAC) - How much is too much?

Sentri SharePoint Performance webinar

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?

Agile analytics applications on hadoop

Hortonworks: Agile Analytics Applications

Compaction and Splitting in Apache Accumulo

Introduction to DDD

ORC: 2015 Faster, Better, Smaller

Ozone and HDFS’s evolution

ORC 2015

Storage Characteristics Of Call Data Records In Column Store Databases

Java and Mongo

Mais de Owen O'Malley

Running An Apache Project: 10 Traps and How to Avoid ThemOwen O'Malley

Big Data's Journey to ACIDOwen O'Malley

ORC Deep Dive 2020Owen O'Malley

Protect your private data with ORC column encryptionOwen O'Malley

Fine Grain Access Control for Big Data: ORC Column EncryptionOwen O'Malley

Fast Access to Your Data - Avro, JSON, ORC, and ParquetOwen O'Malley

Strata NYC 2018 IcebergOwen O'Malley

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and ParquetOwen O'Malley

ORC Column EncryptionOwen O'Malley

Mais de Owen O'Malley (9)

Running An Apache Project: 10 Traps and How to Avoid Them

Big Data's Journey to ACID

ORC Deep Dive 2020

Protect your private data with ORC column encryption

Fine Grain Access Control for Big Data: ORC Column Encryption

Fast Access to Your Data - Avro, JSON, ORC, and Parquet

Strata NYC 2018 Iceberg

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and Parquet

ORC Column Encryption

Último

Install Stable Diffusion in windows machinePadma Pradeep

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

Slack Application Development 101 Slidespraypatel2

Key Features Of Token Development (1).pptxLBM Solutions

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

AI as an Interface for Commercial BuildingsMemoori

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

A Domino Admins Adventures (Engage 2024)Gabriella Davis

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls