A TPC Benchmark of Hive LLAP and Comparison with Presto

We discuss the current state of LLAP (Live Long and Process) – the concurrent sub-second execution of analytical queries engine for Hive 2.0. LLAP is a hybrid execution model that enables performance improvement in and across queries, such as caching of columnar data with cache coherence and intelligent eviction for disaggregated storage models (like S3, Isilon, Azure), JIT-friendly operator pipelines, asynchronous I/O, data pre-fetching and multi-threaded processing. LLAP features robust machine and service failure tolerance achieved by building on top of the time-tested fault tolerant subsystems, as well as a concurrency-directed design that achieves high utilization with low latency via resource sharing, reducing overheads for multiple queries, and enabling the system to preempt tasks of lower priority without failing any query in-flight. The talk also aims to cover the novel deployment model required for hybrid execution. The elasticity demands of the system are served by a long-lived YARN service interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework for query execution. We discuss the current state of the project, performance numbers, deployment and usage strategy, as well as future work, including how LLAP fits into a unified secure DataFrame access layer.

LLAP: Sub-Second Analytical Queries in Hive

Apache Hive ACID Project

LLAP Nov Meetup

t3rmin4t0r

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

Apache Hive 2.0: SQL, Speed, Scale

China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business. In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows: 1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes. 2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included. 3. About Flume: We use the reformed Flume to collect data as much as 200TB per day. Speakers Yuxuan Pan, Software Engineer, China Mobile Software Technology Duan Yunfeng, Chief Designer of China Mobile's big data system, China Mobile Communications Corporation

Practice of large Hadoop cluster in China Mobile

Hive acid and_2.x new_features

Alberto Romero

Evolving HDFS to Generalized Storage Subsystem

Tune up Yarn and Hive

rxu

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

LLAP (Live Long and Process) is the newest query acceleration engine for Hive 2.0, which entered GA in 2017. LLAP brings into light a new set of trade-offs and optimizations that allows for efficient and secure multi-user BI systems on the cloud. In this talk, we discuss the specifics of building a modern BI engine within those boundaries, designed to be fast and cost-effective on the public cloud. The focus of the LLAP cache is to speed up common BI query patterns on the cloud, while avoiding most of the operational administration overheads of maintaining a caching layer, with an automatically coherent cache with intelligent eviction and support for custom file formats from text to ORC, and explore the possibilities of combining the cache with a transactional storage layer which supports online UPDATE and DELETES without full data reloads. LLAP by itself, as a relational data layer, extends the same caching and security advantages to any other data processing framework. We overview the structure of such a hybrid system, where both Hive and Spark use LLAP to provide SQL query acceleration on the cloud with new, improved concurrent query support and production-ready tools and UI. Speaker Sergey Shelukin, Member of Technical Staff, Hortonworks

LLAP: Building Cloud First BI

This presentation was given at the Strata + Hadoop World, 2015 in San Jose. Apache Hive is the most popular and most widely used SQL solution for Hadoop. To keep pace with Hadoop’s increasingly vital role in the Enterprise, Hive has transformed from a batch-only, high-latency system into a modern SQL engine capable of both batch and interactive queries over large datasets. Hive’s momentum is accelerating: With Spark integration and a shift to in-memory processing on the horizon, Hive continues to expand the boundaries of Big Data. In this talk the speakers examined Hive performance, past, present and future. In particular they looked at Hive’s origins as a petabyte scale SQL engine. Through some numbers and graphs, they showed how Hive became 100x faster by moving beyond MapReduce, by vectorizing execution and by introducing a cost-based optimizer. They detailed and discussed the challenges of scalable SQL on Hadoop. The looked into Hive’s sub-second future, powered by LLAP and Hive on Spark. And showed just how fast Hive on Spark really is.

Hive edw-dataworks summit-eu-april-2017

alanfgates

Hive on spark is blazing fast or is it final

Hortonworks

Raft protocol has been successfully used for consistent metadata replication; however, using it for data replication poses unique challenges. Apache Ratis is a RAFT implementation targeted at high throughput data replication problems. Apache Ratis is being successfully used as a consensus protocol for data stored in Ozone (object store) and Quadra(block device) to provide data throughput that saturates the network links and disk bandwidths. Pluggable nature of Ratis renders it useful for multiple use cases including high availability, data or metadata replication, and ensuring consistency semantics. This talk presents the design challenges to achieve high throughput and how Apache Ratis addresses them. We talk about specific optimizations that have been implemented to minimize overheads and scale up the throughput while maintaining correctness of the consistency protocol. The talk also explains how systems like Ozone take advantage of Ratis’s implementation choices to achieve scale. We will discuss the current performance numbers and also future optimizations. MUKUL KUMAR SINGH, Staff Software Engineer, Hortonworks and LOKESH JAIN, Software Engineer, Hortonworks

High throughput data replication over RAFT

Many organizations today have already migrated Hadoop workloads to cloud infrastructure or they are actively planning to do such a migration. A common question in this scenario is "Which instance types should I use for my Hadoop cluster?" There are nuances to cloud infrastructure that require careful consideration when deciding which instances types to use. This session will show the results of performance comparison of Amazon Web Services (AWS) Elastic Compute Cloud (EC2) instance types commonly used in Hadoop clusters. More importantly, we will discuss the relative cost comparison of these instance types to demonstrate the which AWS instances offer the best price to performance ratio using standard benchmarks. Attendees of this session with leave with a better understanding of the performance of AWS EC2 instance types when used for Hadoop workloads and be able to make more informed decisions about which instance types makes the most sense for their needs. Speakers Michael Young, Senior Solutions Engineer, Hortonworks Marcus Waineo, Principal Solutions Engineer, Hortonworks

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations

Apache Apex

Comparative Performance Analysis of AWS EC2 Instance Types Commonly Used for ...

Hive & HBase for Transaction Processing Hadoop Summit EU Apr 2015

alanfgates

Sub-second-sql-on-hadoop-at-scale

Yifeng Jiang

Migrating pipelines into Docker

With the increase of Hadoop offerings in the Cloud, users are faced with many decisions to make: which Cloud provider, VMs to choose, cluster sizing, storage type, or even if to go to fully managed Platform-as-a-Service (PaaS) Hadoop? As the answer is always "depends on your data and usage", this talk will guide participants over an overview of the different PaaS solutions for the leading Cloud providers. By highlighting the main results benchmarking their SQL-on-Hadoop (i.e., Hive) services using the ALOJA benchmarking project. To compare their current offerings in terms of readiness, architectural differences, and cost-effectiveness (performance-to-price), to entry-level Hadoop based deployments. As well as briefly presenting how to replicate results and create custom benchmarks from internal apps. So that users can make their own decisions about choosing the right provider to their particular data needs.

Mais procurados (20)

From Device to Data Center to Insights

LLAP: Sub-Second Analytical Queries in Hive

Apache Hive ACID Project

LLAP Nov Meetup

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

Apache Hive 2.0: SQL, Speed, Scale

Practice of large Hadoop cluster in China Mobile

Hive acid and_2.x new_features

Evolving HDFS to Generalized Storage Subsystem

Tune up Yarn and Hive

Apache Phoenix and HBase: Past, Present and Future of SQL over HBase

LLAP: Building Cloud First BI

Hive edw-dataworks summit-eu-april-2017

Hive on spark is blazing fast or is it final

High throughput data replication over RAFT

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations

Comparative Performance Analysis of AWS EC2 Instance Types Commonly Used for ...

Hive & HBase for Transaction Processing Hadoop Summit EU Apr 2015

Sub-second-sql-on-hadoop-at-scale

Migrating pipelines into Docker

Semelhante a A TPC Benchmark of Hive LLAP and Comparison with Presto

Sql server 2016 it just runs faster sql bits 2017 edition

Bob Ward

Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong

Ceph Community

Hadoop World 2011: Hadoop Network and Compute Architecture Considerations - J...

Cloudera, Inc.

SQL Server It Just Runs Faster

Bob Ward

The state of SQL-on-Hadoop in the Cloud

Nicolas Poggi

The state of SQL-on-Hadoop in the Cloud

Since 2006 the world of big data has moved from terabytes to hundreds of petabytes, from local clusters to remote cloud storage, yet the original Apache Hadoop POSIX-based file APIs have barely changed. It is wonderful that these APIs have worked so well, but we can do a lot better with remote object stores, by providing new operations which suit them better, targeted at columnar data libraries such as ORC and Spark. Only a few libraries need to migrate to these APIs for significant speedups of all big data applications. This talk introduces a new Hadoop Filesystem API called "vectored read", coming in Hadoop 3.4. An extension of the classic FSDataInputStream it is automatically offered by all filesystem clients. The S3A connector is the first object store to provide a custom implementation, reading different blocks of data in parallel. In Apache Hive benchmarks with a modified ORC library, we saw a 2x speedup compared to using the classic s3a connector through the Posix APIs. We will introduce the API spec, the S3A implementation, and the benchmarks, and show how to use it in your own applications. We will also cover our ongoing work on providing similar speedups with other object stores, and the use of the API in other applications.

Ceph Performance Profiling and Reporting

Ceph Community

[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf

MukundThakur22

20140120 presto meetup_en

Ogibayashi

Originally presented at the BDOOP and Spark Barcelona meetup groups: http://meetu.ps/3bwCTM Cloud providers currently offer convenient on-demand managed big data clusters (PaaS) with a pay-as-you-go model. In PaaS, analytical engines such as Spark and Hive come ready to use, with a general-purpose configuration and upgrade management. Over the last year, the Spark framework and APIs have been evolving very rapidly, with major improvements on performance and the release of v2, making it challenging to keep up-to-date production services both on-premises and in the cloud for compatibility and stability. The talk compares: • The performance of both v1 and v2 for Spark and Hive • PaaS cloud services: Azure HDinsight, Amazon Web Services EMR, Google Cloud Dataproc • Out-of-the-box support for Spark and Hive versions from providers • PaaS reliability, scalability, and price-performance of the solutions Using BigBench, the new Big Data benchmark standard. BigBench combines SQL queries, MapReduce, user code (UDF), and machine learning, which makes it ideal to stress Spark libraries (SparkSQL, DataFrames, MLlib, etc.).

The state of Hive and Spark in the Cloud (July 2017)

Nicolas Poggi

Apache Impala is a modern, open-source MPP SQL engine architected from the ground up for the Hadoop data processing environment. Impala provides low latency and high concurrency for BI/analytic read-mostly queries on Hadoop, not delivered by batch frameworks such as Hive or SPARK. Impala is written from the ground up in C++ and Java. It maintains Hadoop’s flexibility by utilizing standard components (HDFS, HBase, Metastore, Sentry) and is able to read the majority of the widely-used file formats (e.g. Parquet, Avro, RCFile). To reduce latency, such as that incurred from utilizing MapReduce or by reading data remotely, Impala implements a distributed architecture based on daemon processes that are responsible for all aspects of query execution and that run on the same machines as the rest of the Hadoop infrastructure. Impala employs runtime code generation using LLVM in order to improve execution times and uses static and dynamic partition pruning to significantly reduce the amount of data accessed. The result is performance that is on par or exceeds that of commercial MPP analytic DBMSs, depending on the particular workload. Although initially designed for running on-premises against HDFS-stored data, Impala can also run on public clouds and access data stored in various storage engines such as object stores (e.g. AWS S3), Apache Kudu and HBase. In this talk, we present Impala's architecture in detail and discuss the integration with different storage engines and the cloud.

Performance Optimizations in Apache Impala

Cloudera, Inc.

Super Computer

gueste3bbd0

We implemented MapReduce cluster benchmark TeraSort by derivative free optimization (DFO) method having runtime function object. In this, every iteration of DFO method uses new values for Hadoop parameter configuration. These parameters are specified within the framework, we used Chef server and client tool which assists in this cluster configuration to ensure proper implementation of TeraSort application.

Using Derivation-Free Optimization in the Hadoop Cluster with Terasort

Anhanguera Educacional S/A

How Many Slaves (Ukoug)

Doug Burns

PostgreSQL 10: What to Look For

Amit Langote

At Twitter we started out with a large monolithic cluster that served most of the use-cases. As the usage expanded and the cluster grew accordingly, we realized we needed to split the cluster by access pattern. This allows us to tune the access policy, SLA, and configuration for each cluster. We will explain our various use-cases, their performance requirements, and operational considerations and how those are served by the corresponding clusters. We will discuss what our baseline Hadoop node looks like. Various, sometimes competing, considerations such as storage size, disk IO, CPU throughput, fewer fast cores versus many slower cores, 1GE bonded network interfaces versus a single 10 GE card, 1T, 2T or 3T disk drives, and power draw all need to be considered in a trade-off where cost and performance are major factors. We will show how we have arrived at quite different hardware platforms at Twitter, not only saving money, but also increasing performance.

Hadoop Hardware @Twitter: Size does matter!

Apache Spark is a powerful, scalable real-time data analytics engine that is fast becoming the de facto hub for data science and big data. However, in parallel, GPU clusters are fast becoming the default way to quickly develop and train deep learning models. As data science teams and data savvy companies mature, they will need to invest in both platforms if they intend to leverage both big data and artificial intelligence for competitive advantage. This session will cover: – How to leverage Spark and TensorFlow for hyperparameter tuning and for deploying trained models – DeepLearning4J, CaffeOnSpark, IBM’s SystemML and Intel’s BigDL – Sidecar GPU cluster architecture and Spark-GPU data reading patterns – The pros, cons and performance characteristics of various approaches You’ll leave the session better informed about the available architectures for Spark and deep learning, and Spark with and without GPUs for deep learning. You’ll also learn about the pros and cons of deep learning software frameworks for various use cases, and discover a practical, applied methodology and technical examples for tackling big data deep learning.

Deep Learning with Apache Spark and GPUs with Pierce Spitler

Databricks

In this talk, we will present how we analyze, predict, and visualize network quality data, as a spark AI use case in a telecommunications company. SK Telecom is the largest wireless telecommunications provider in South Korea with 300,000 cells and 27 million subscribers. These 300,000 cells generate data every 10 seconds, the total size of which is 60TB, 120 billion records per day. In order to address previous problems of Spark based on HDFS, we have developed a new data store for SparkSQL consisting of Redis and RocksDB that allows us to distribute and store these data in real time and analyze it right away, We were not satisfied with being able to analyze network quality in real-time, we tried to predict network quality in near future in order to quickly detect and recover network device failures, by designing network signal pattern-aware DNN model and a new in-memory data pipeline from spark to tensorflow. In addition, by integrating Apache Livy and MapboxGL to SparkSQL and our new store, we have built a geospatial visualization system that shows the current population and signal strength of 300,000 cells on the map in real time.

Apache Spark AI Use Case in Telco: Network Quality Analysis and Prediction wi...

Databricks

In this session I will use a simple HTTP benchmark to compare the performance of the Linux kernel networking stack with userspace networking powered by DPDK (kernel-bypass). It is said that kernel-bypass technologies avoid the kernel because it is "slow", but in reality, a lot of the performance advantages that they bring just come from enforcing certain constraints. As it turns out, many of these constraints can be enforced without bypassing the kernel. If the system is tuned just right, one can achieve performance that approaches kernel-bypass speeds, while still benefiting from the kernel's battle-tested compatibility, and rich ecosystem of tools.

Linux Kernel vs DPDK: HTTP Performance Showdown

ScyllaDB

WMS Performance Shootout 2009

Jeff McKenna

Semelhante a A TPC Benchmark of Hive LLAP and Comparison with Presto (20)

Sql server 2016 it just runs faster sql bits 2017 edition

Unlock Bigdata Analytic Efficiency with Ceph Data Lake - Zhang Jian, Fu Yong

Hadoop World 2011: Hadoop Network and Compute Architecture Considerations - J...

SQL Server It Just Runs Faster

The state of SQL-on-Hadoop in the Cloud

Ceph Performance Profiling and Reporting

[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdf

20140120 presto meetup_en

The state of Hive and Spark in the Cloud (July 2017)

Performance Optimizations in Apache Impala

Super Computer

Using Derivation-Free Optimization in the Hadoop Cluster with Terasort

How Many Slaves (Ukoug)

PostgreSQL 10: What to Look For

Hadoop Hardware @Twitter: Size does matter!

Deep Learning with Apache Spark and GPUs with Pierce Spitler

Apache Spark AI Use Case in Telco: Network Quality Analysis and Prediction wi...

Linux Kernel vs DPDK: HTTP Performance Showdown

WMS Performance Shootout 2009

Mais de Yu Liu

Cloud Era Transactional Processing -- Problems, Strategies and Solutions

Introduction to NTCIR 2016 MedNLPDoc

高性能データ処理プラットフォーム (Talk on July Tech Festa 2015)

Survey on Parallel/Distributed Search Engines

Paper introduction to Combinatorial Optimization on Graphs of Bounded Treewidth

Paper Introduction: Combinatorial Model and Bounds for Target Set Selection

An accumulative computation framework on MapReduce ppl2013

An Enhanced MapReduce Model (on BSP)

A Homomorphism-based Framework for Systematic Parallel Programming with MapRe...

An Introduction of Recent Research on MapReduce (2011)

A Generate-Test-Aggregate Parallel Programming Library on Spark

Introduction of A Lightweight Stage-Programming Framework

Start From A MapReduce Graph Pattern-recognize Algorithm

Introduction of the Design of A High-level Language over MapReduce -- The Pig...

On Extending MapReduce - Survey and Experiments

Tree representation in map reduce world

Introduction to Ultra-succinct representation of ordered trees with applications

On Implementation of Neuron Network(Back-propagation)

ScrewDriver Rebirth: Generate-Test-and-Aggregate Framework on Hadoop

A Homomorphism-based MapReduce Framework for Systematic Parallel Programming