Adding ACID Updates to Hive

The next generation of Hadoop MapReduce Arun C. Murthy presented the plans for the next generation of Apache Hadoop MapReduce. The MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. More information and video available at: http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/

We are starting to add inserts, updates, and deletes to Apache Hive while adding ACID guarantees.

Tecnologia Espiritual

Adding ACID Updates to Hive
Owen O’Malley
owen@hortonworks.com
@owen_omalley
October 2013

© Hortonworks Inc. 2013

Page 1

Who Am I?

© Hortonworks Inc. 2013

Page 2

A Data Flood

© Hortonworks Inc. 2013

Page 3

Use Cases

© Hortonworks Inc. 2013

Page 4

Delete, Update, and Insert in Hive

© Hortonworks Inc. 2013

Page 5

Stitching Buckets Together

© Hortonworks Inc. 2013

Page 7

HDFS Layout

© Hortonworks Inc. 2013

Page 8

Compaction

© Hortonworks Inc. 2013

Page 9

Limitations

© Hortonworks Inc. 2013

Page 10

Phases of Development

© Hortonworks Inc. 2013

Page 11

Why not HBase?

© Hortonworks Inc. 2013

Page 12

Conclusion

© Hortonworks Inc. 2013

Page 13

Thank You!
Questions & Answers

© Hortonworks Inc. 2013

Page 14

Mais conteúdo relacionado

Destaque

Next Generation MapReduce

Next Generation Hadoop Operations

Optimizing Hive Queries

Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. The upcoming Hive 0.11 will add a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding -- resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query. Finally, ORC works together with the upcoming query vectorization work providing a high bandwidth reader/writer interface.

ORC Files

Hadoop Summit June 2016 The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: * reading all of the columns * reading a few of the columns * filtering using a filter predicate * writing the data Furthermore, it is important to benchmark on real data rather than synthetic data. We used the Github logs data available freely from http://githubarchive.org We will make all of the benchmark code open source so that our experiments can be replicated.

File Format Benchmarks - Avro, JSON, ORC, & Parquet

Eric Hanson and I gave this presentation at Hadoop Summit 2013: Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. Hive 0.11 added a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding — resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query. Columnar storage formats like ORC reduce I/O and storage use, but it’s just as important to reduce CPU usage. A technical breakthrough called vectorized query execution works nicely with column store formats to do this. Vectorized query execution has proven to give dramatic performance speedups, on the order of 10X to 100X, for structured data processing. We describe how we’re adding vectorized query execution to Hive, coupling it with ORC with a vectorized iterator.

ORC File and Vectorization - Hadoop Summit 2013

Hadoop Security Architecture

Apache Hive is a data warehousing system for large volumes of data stored in Hadoop. However, the data is useless unless you can use it to add value to your company. Hive provides a SQL-based query language that dramatically simplifies the process of querying your large data sets. That is especially important while your data scientists are developing and refining their queries to improve their understanding of the data. In many companies, such as Facebook, Hive accounts for a large percentage of the total MapReduce queries that are run on the system. Although Hive makes writing large data queries easier for the user, there are many performance traps for the unwary. Many of them are artifacts of the way Hive has evolved over the years and the requirement that the default behavior must be safe for all users. This talk will present examples of how Hive users have made mistakes that made their queries run much much longer than necessary. It will also present guidelines for how to get better performance for your queries and how to look at the query plan to understand what Hive is doing.

Optimizing Hive Queries

Hortonworks recently unveiled the Hortonworks Sandbox, a free, comprehensive, easy-to-use, hands-on learning environment that provides the fastest onramp for anyone interested in learning, evaluating or using Apache Hadoop™ in an enterprise. This interactive webinar will discuss and demo features of the Hortonworks Sandbox, including: -How to download and use the Sandbox tutorials. -How to upload your own datasets to test and validate the use of Apache Hadoop. -Demos of features and use cases for your very own Hortonworks Sandbox.

ORC 2015

t3rmin4t0r

Parquet Hadoop Summit 2013

Julien Le Dem

Hive tuning

Michael Zhang

Destaque (11)

Next Generation MapReduce

Next Generation Hadoop Operations

Optimizing Hive Queries

ORC Files

File Format Benchmarks - Avro, JSON, ORC, & Parquet

ORC File and Vectorization - Hadoop Summit 2013

Hadoop Security Architecture

Optimizing Hive Queries

ORC 2015

Parquet Hadoop Summit 2013

Hive tuning

Semelhante a Adding ACID Updates to Hive

Go Zero to Big Data in 15 Minutes with the Hortonworks Sandbox

Inside hadoop-dev

Steve Loughran

Bring your Service to YARN

Compaction and Splitting in Apache Accumulo

DataWorks Summit/Hadoop Summit

Protecting Enterprise Data in Apache Hadoop

TEZ-8 UI Walkthrough

t3rmin4t0r

Protecting enterprise Data in Hadoop

Why hadoop for data science?

Halosys Technologies (I) Pvt. Ltd.

Munich HUG 21.11.2013

Emil Andreas Siemes

Ironic summary @ LCA 2014

Devananda Van Der Veen

2013 march 26_thug_etl_cdc_talking_points

Adam Muise

Ben Marden - Making sense of Big Data

WeAreEsynergy

Storm Demo Talk - Colorado Springs May 2015

Mac Moore

Corporate Brochure - Halosys Technologies

Presenter: Ofer Mendelevitch of Hortonworks > Learn the benefits of big data for data scientists, and how Hadoop and HDInsight fit into the modern data architecture and enable data-driven products. You'll learn: * What data science actually means * The term "data products" * The benefits of using big data for data scientists * How Hadoop helps data scientists work with big data * About HDInsight, the big data platform from Microsoft and Hortonworks

Hortonworks Big Data & Hadoop

Mark Ginnebaugh

Data Science with Hadoop - A primer

Ofer Mendelevitch

Enterprise-Grade Rolling Upgrade for a Live Hadoop Cluster

Docker based Hadoop provisioning - anywhere

UK - Agile Data Applications on Hadoop

In recent years we see a major shift toward SaaS solutions. More and more HPSW customers prefer to consume products like Quality Center, Performance Center and Agile Project Management as a Service. Meeting this increased demand for SaaS triggered a major shift within HP SW development groups and HP SaaS operations group to not only modernize our products and offering but also to modernized the way we develop, test, deploy and operate our software in a SaaS model by moving to DevOps. In this session we will discuss how HPSW Dev and Ops joined forces to establish the right methodologies, processes and technologies to build a true DevOPs delivery model that is aligned across HP SW, starting with Agile Manager, our first true SaaS product and continuing with traditional products like Quality Center. Today in SaaS for Agile Manager we have 4 farms located over 3 locations (3 regions – AMS, EMEA, APJ). We have more than 120 customers and over 6000 of users login each day to our systems with over 1000 active tenants. We have bi-weekly pushes and Quarterly major releases, comprehensive monitoring processes and extensive implementation of HP monitoring tools. Over 4000 tickets handled by both Operations and R&D.

eDevOps in HPSW from buzzword to reality

AgileSparks

Semelhante a Adding ACID Updates to Hive (20)

Go Zero to Big Data in 15 Minutes with the Hortonworks Sandbox

Inside hadoop-dev

Bring your Service to YARN

Compaction and Splitting in Apache Accumulo

Protecting Enterprise Data in Apache Hadoop

TEZ-8 UI Walkthrough

Protecting enterprise Data in Hadoop

Why hadoop for data science?

Munich HUG 21.11.2013

Ironic summary @ LCA 2014

2013 march 26_thug_etl_cdc_talking_points

Ben Marden - Making sense of Big Data

Storm Demo Talk - Colorado Springs May 2015

Corporate Brochure - Halosys Technologies

Hortonworks Big Data & Hadoop

Data Science with Hadoop - A primer

Enterprise-Grade Rolling Upgrade for a Live Hadoop Cluster

Docker based Hadoop provisioning - anywhere

UK - Agile Data Applications on Hadoop

eDevOps in HPSW from buzzword to reality

Mais de Owen O'Malley

Running An Apache Project: 10 Traps and How to Avoid Them

Big Data's Journey to ACID

ORC Deep Dive 2020

Fine-grained data protection at a column level in data lake environments has become a mandatory requirement to demonstrate compliance with multiple local and international regulations across many industries today. ORC is a self-describing type-aware columnar file format designed for Hadoop workloads that provides optimized streaming reads but with integrated support for finding required rows quickly. Owen O’Malley dives into the progress the Apache community made for adding fine-grained column-level encryption natively into ORC format, which also provides capabilities to mask or redact data on write while protecting sensitive column metadata such as statistics to avoid information leakage. The column encryption capabilities will be fully compatible with Hadoop Key Management Server (KMS) and use the KMS to manage master keys, providing the additional flexibility to use and manage keys per column centrally.

Protect your private data with ORC column encryption

Fine-grained data protection at a column level in data lake environments has become a mandatory requirement to demonstrate compliance with multiple local and international regulations across many industries today. ORC is a self-describing type-aware columnar file format designed for Hadoop workloads that provides optimized streaming reads, but with integrated support for finding required rows quickly. In this talk, we will outline the progress made in Apache community for adding fine-grained column level encryption natively into ORC format that will also provide capabilities to mask or redact data on write while protecting sensitive column metadata such as statistics to avoid information leakage. The column encryption capabilities will be fully compatible with Hadoop Key Management Server (KMS) and use the KMS to manage master keys providing the additional flexibility to use and manage keys per column centrally.

Fine Grain Access Control for Big Data: ORC Column Encryption

The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: * reading all of the columns * reading a few of the columns * filtering using a filter predicate While previous work has compared the size and speed from Hive, this presentation will present benchmarks from Spark including the new work that radically improves the performance of Spark on ORC. This presentation will also include tips and suggestions to optimize the performance of your application while reading and writing the data. Finally, the value of having open source benchmarks that are available to all interested parties is hugely important and all of the code is available from Apache.

Fast Access to Your Data - Avro, JSON, ORC, and Parquet

Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic. Netflix uses tables backed by S3 that, like other object stores, don’t fit this directory-based model: listings are much slower, renames are not atomic, and results are eventually consistent. Even tables in HDFS are problematic at scale, and reliable query behavior requires readers to acquire locks and wait. Owen O’Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout addresses the challenges of current Hive tables, with properties specifically designed for cloud object stores, such as S3. Iceberg is an Apache-licensed open source project. It specifies the portable table format and standardizes many important features, including: * All reads use snapshot isolation without locking. * No directory listings are required for query planning. * Files can be added, removed, or replaced atomically. * Full schema evolution supports changes in the table over time. * Partitioning evolution enables changes to the physical layout without breaking existing queries. * Data files are stored as Avro, ORC, or Parquet. * Support for Spark, Hive, and Presto.

Strata NYC 2018 Iceberg

The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: reading all of the columns reading a few of the columns filtering using a filter predicate While previous work has compared the size and speed from Hive, this presentation will present benchmarks from Spark including the new work that radically improves the performance of Spark on ORC. This presentation will also include tips and suggestions to optimize the performance of your application while reading and writing the data.

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and Parquet