SQL on Accumulo

•Transferir como PPTX, PDF•

5 gostaram•4,030 visualizações

7:30 SQL-on-Accumulo - Don Miner, ClearEdge IT Running SQL queries over data in Accumulo is easier said than done and has several nuanced design challenges that don't have clear answers. This talk will give an outline of the current state of the art in SQL-on-Accumulo technologies, while giving a realistic view on what is doable and what is not doable today.

Tecnologia

SQL-on-Accumulo
A talk about how we may be able to do it one day
DONALD MINER
1
May 5th, 2014

A brief history of Hadoop and SQL
2
time
1980 2000

A brief history of Hadoop and SQL
3
time
1980 2000

A brief history of Hadoop and SQL
5
time
1980 2000
SQL-on-Hadoop

SQL-on-Accumulo would be nice
Problem: Accumulo is just a data store
 We’ll have to do query somewhere else
6

WWHBD? - Hive
• Hive
 Runs in MapReduce
 Map col family and col qualifiers to columns
 Maintained by Hive community
• Impala and Shark inherit functionality from Hive
8

WWHBD? - next level
Problem:
Hive, Impala, and Shark don’t know how HBase works
… and don’t care
• Apache Phoenix
 Specifically SQL-on-HBase
 Currently Apache incubator project
 Client-embedded JDBC driver
 Uses series of scans and coprocessors
• Pivotal’s HAWQ and PXF
 PXF is external table functionality in HAWQ
 Native support for HAWQ: uses push down filters, range
scans, etc. to efficiently slurp data into HAWQ
9

ACCUMULO-143
people. technology. integrity. 10

SQL-on-Accumulo Status
Hive (and somewhat Impala and Shark)
• Github project by Brian Femiano [1]
 Doesn’t work on new versions
 Hasn’t been touched in 9 months
 Wasn’t committed into trunk
• Some rumors that some orgs have done it
themselves (but no public information)
people. technology. integrity. 11
[1] https://github.com/bfemiano/accumulo-hive-storage-manager (google for “accumulo hive”)

SQL-on-Accumulo Status
Phoenix
• Discussion on mailing list last week
• Some differences between iterators and
coprocessors makes this interesting
Pivotal’s HAWQ and PXF
• In development
• Will support visibility labels
• Pushdown and optimizations with iterators
people. technology. integrity. 12

Visibility Design Problems
13
These problems are unique to Accumulo
• SELECT and visibility labels
 Assume two cells, only uniqueness is visibility…
Which do I pick in a SELECT?
 Timestamps have this problem, but have a logical
assumption (most recent)
• Authorizations in SQL
 How do you tell the execution engine which
authorizations to use?
 Table definition? (hard to change)
 SQL statement? (extend SQL language?)
 Based on login? (how do you downgrade?)

What are the next steps?
I guess that’s up to the community
14

QUIZ: What is this definition trying to say?
Big Data:
• Volume
• Variety
• Velocity
• Veracity
15
A warning about SQL-on-Accumulo

QUIZ: What is this definition trying to say?
Big Data:
• Volume
• Variety
• Velocity
• Veracity
Answer: RDBMS/SQL suck at all these things
16
A warning about SQL-on-Accumulo

QUIZ: What is this definition trying to say?
Big Data:
• Volume
• Variety
• Velocity
• Veracity
Answer: RDBMS/SQL suck at all these things
17
A warning about SQL-on-Accumulo
What does SQL-on-Accumulo still suck at?
*Added context for my internet viewers since this could be controversial if taken literally and I’m not talking to my
slides: I’m trying to say that SQL-on-X can’t solve all of the worlds problems, but it can solve a good number of
them very well. It also tees up the conversation that SQL is not the end-all-be-all… there are ways that it could be
made better to adapt to “the big data use case”. Don’t take this the wrong way, SQL-on-Hadoop and SQL-on-
Accumulo would be incredible useful, but it doesn’t solve 100% of the problems.

SQL-on-Accumulo
DONALD MINER
18
dminer@clearedgeit.com @donaldpminer
Questions?

Mais conteúdo relacionado

Mais procurados

Scaling etl with hadoop shapira 3

Gwen (Chen) Shapira

Architecting Applications with Hadoop

markgrover

Impala Architecture presentation

hadooparchbook

Hadoop Summit 2015: Hive at Yahoo: Letters from the Trenches

Mithun Radhakrishnan

Functional Programming and Big Data

DataWorks Summit

Data Pipelines in Hadoop - SAP Meetup in Tel Aviv

larsgeorge

Way too many enterprise decision makers have clouded and uninformed views of how Hadoop works and what it does. Donald Miner offers high-level observations about Hadoop technologies and explains how Hadoop can shift the paradigms inside of an organization, based on his report Hadoop: What You Need To Know—Hadoop Basics for the Enterprise Decision Maker, forthcoming from O’Reilly Media. After a basic introduction to Hadoop and the Hadoop ecosystem, Donald outlines 10 basic concepts you need to understand to master Hadoop: Hadoop masks being a distributed system: what it means for Hadoop to abstract away the details of distributed systems and why that’s a good thing Hadoop scales out linearly: why Hadoop’s linear scalability is a paradigm shift (but one with a few downsides) Hadoop runs on commodity hardware: an honest definition of commodity hardware and why this is a good thing for enterprises Hadoop handles unstructured data: why Hadoop is better for unstructured data than other data systems from a storage and computation perspective In Hadoop, you load data first and ask questions later: the differences between schema-on-read and schema-on-write and the drawbacks this represents Hadoop is open source: what it really means for Hadoop to be open source from a practical perspective, not just a “feel good” perspective HDFS stores the data but has some major limitations: an overview of HDFS (replication, not being able to edit files, and the NameNode) YARN controls everything going on and is mostly behind the scenes: an overview of YARN and the pitfalls of sharing resources in a distributed environment and the capacity scheduler MapReduce may be getting a bad rap, but it’s still really important: an overview of MapReduce (what it’s good at and bad at and why, while it isn’t used as much these days, it still plays an important role) The Hadoop ecosystem is constantly growing and evolving: an overview of current tools such as Spark and Kafka and a glimpse of some things on the horizon

10 concepts the enterprise decision maker needs to understand about Hadoop

Donald Miner

Marcel Kornacker is a tech lead at Cloudera In this talk from Impala architect Marcel Kornacker, you will explore: How Impala's architecture supports query speed over Hadoop data that not only convincingly exceeds that of Hive, but also that of a proprietary analytic DBMS over its own native columnar format. The current state of, and roadmap for, Impala's analytic SQL functionality. An example configuration and benchmark suite that demonstrate how Impala offers a high level of performance, functionality, and ability to handle a multi-user workload, while retaining Hadoop’s traditional strengths of flexibility and ease of scaling.

Building a Hadoop Data Warehouse with Impala

huguk

Cloudera Impala: A Modern SQL Engine for Apache Hadoop

Cloudera, Inc.

Hadoop databases for oracle DBAs

Maxym Kharchenko

Using Apache Spark as ETL engine. Pros and Cons

Provectus

Applications on Hadoop

markgrover

This is a technical deep dive about Cloudera Impala, the project that makes scalable parallel databse technology available to the Hadoop community for the first time. Impala is an open-sourced code base that allows users to issue low-latency queries to data stored in HDFS and Apache HBase using familiar SQL operators. Presenter Marcel Kornacker, creator of Impala, begins with an overview of Impala from the user's perspective, followed by an overview of Impala's architecture and implementation, and will conclude with a comparison of Impala with Dremel and Apache Hive, commercial MapReduce alternatives and traditional data warehouse infrastructure.

Cloudera Impala: A Modern SQL Engine for Hadoop

Cloudera, Inc.

Application architectures with Hadoop – Big Data TechCon 2014

hadooparchbook

NoSQL databases pros and cons

Fabio Fumarola

Cloudera Impala: A modern SQL Query Engine for Hadoop

Cloudera, Inc.

SQL Engines for Hadoop - The case for Impala

markgrover

Building a Business on Hadoop, HBase, and Open Source Distributed Computing

Bradford Stephens

Efficient data access is one of the key factors for having a high performance data processing pipeline. Determining the layout of data values in the filesystem often has fundamental impacts on the performance of data access. In this talk, we will show insights on how data layout affects the performance of data access. We will first explain how modern columnar file formats like Parquet and ORC work and explain how to use them efficiently to store data values. Then, we will present our best practice on how to store datasets, including guidelines on choosing partitioning columns and deciding how to bucket a table.

Why you should care about data layout in the file system with Cheng Lian and ...

Databricks

Apache Spark is an excellent tool to accelerate your analytics, whether you’re doing ETL, Machine Learning, or Data Warehousing. However, to really make the most of Spark it pays to understand best practices for data storage, file formats, and query optimization. This talk will cover best practices I’ve applied over years in the field helping customers write Spark applications as well as identifying what patterns make sense for your use case.

Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...

Databricks

Mais procurados (20)

Scaling etl with hadoop shapira 3

Architecting Applications with Hadoop

Impala Architecture presentation

Hadoop Summit 2015: Hive at Yahoo: Letters from the Trenches

Functional Programming and Big Data

Data Pipelines in Hadoop - SAP Meetup in Tel Aviv

10 concepts the enterprise decision maker needs to understand about Hadoop

Building a Hadoop Data Warehouse with Impala

Cloudera Impala: A Modern SQL Engine for Apache Hadoop

Hadoop databases for oracle DBAs

Using Apache Spark as ETL engine. Pros and Cons

Applications on Hadoop

Cloudera Impala: A Modern SQL Engine for Hadoop

Application architectures with Hadoop – Big Data TechCon 2014

NoSQL databases pros and cons

Cloudera Impala: A modern SQL Query Engine for Hadoop

SQL Engines for Hadoop - The case for Impala

Building a Business on Hadoop, HBase, and Open Source Distributed Computing

Why you should care about data layout in the file system with Cheng Lian and ...

Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...

Destaque

Machine Learning & Graph Processing w/ Spark and Accumulo

Rahul Singh

Talk Abstract Argyle fraud solution extends Accumulo's scalable data management to new analytic areas. Julia machine learning models used for identifying telecom and financial services fraud leverage instant rowkey lookups from Accumulo, while distributed SQL engine provides interface for common data visualization tools. Speaker Arshak Navruzyan VP of Products, Argyle Data My objective is to make machine learning accessible to any organization or individual that wants to transform the world through data. I am currently VP of Products at Argyle Data focused on petabyte-scale fraud applications using machine learning and Accumulo. Previously I held senior engineering and product management roles at Alpine Data Labs, Endeca and Oracle.

Accumulo Summit 2015: Fraud Analytics using Accumulo, Julia and Fast SQL [Lev...

Accumulo Summit

The rapidly increasing amount of semantic network data today provides a wealth of insight into how entities interact and relate with one another. In order to tap into this valuable source of information, organizations require a secure and scalable repository in which to store and explore these interactions and relationships. In this talk we will discuss Apache Rya, an Accumulo-based graph store capable of storing billions of Resource Description Framework (RDF) triples and providing a rich SPARQL query endpoint for exploring complex subgraph relationships. We will talk about two indexing strategies that Rya uses to address some of the challenges associated with storing and querying large graph datasets. In particular, we will discuss how our SPARQL (SPARQL Protocol and RDF Query Language) query caching framework allows users to greatly improve query performance by storing and incrementally maintaining query results using Apache Fluo. We will also discuss our Accumulo-based entity centric index. Inspired by Facebook’s horizontally partitioned graph index, Unicorn , Apache Rya’s entity centric index is a novel way of storing graphs in Accumulo that draws on document partitioned indexing techniques. This graph partitioning and indexing strategy limits network traffic and enables distributed join processing by utilizing a variation of Accumulo’s IntersectingIterator framework to perform joins server side. The work presented herein was funded by the Office of Naval Research, under contract # N00014-12-C-0365, supporting this effort. – Speaker – Dr. Caleb Meier Software Engineer, Parsons Caleb Meier has been a Software Engineer at Parsons Government Services for the last two years. Since joining Parsons, he has investigated and implemented a number of features to improve the query performance of Apache Rya. Caleb earned his Ph.D. in Mathematics from the University of California, San Diego and a B.A. in Mathematics from Yale University. In his spare time he enjoys climbing, biking, playing soccer and spending time with his delightful wife Leslie. — More Information — For more information see http://www.accumulosummit.com/

Accumulo Summit 2016: Accumulo Indexing Strategies for Searching Semantic Net...

Accumulo Summit

Talk Abstract An overview of the new client API planned for the release of Accumulo 2.0. Problems with the old API are described, along with lessons learned. The benefits of the new API are explained, and code snippets are provided to demonstrate the overall design and contrast it with the old API, in order to assist users interested in transitioning. Speaker Christopher Tubbs Computer Systems Researcher, National Security Agency Christopher is a researcher with the National Security Agency. He holds a Bachelor's degree in both Physics and Computer Science from Eastern Michigan University. He is an open source enthusiast and advocate for open source development, as well as data privacy and security. He has been contributing to the Accumulo project since 2009, prior to its release to the Apache Software Foundation in 2011. He is currently a committer and PMC member on the project, and an ASF member, as well as the Accumulo package maintainer for the Fedora project.

Accumulo Summit 2015: Accumulo 2.0: A New Client API [API]

Accumulo Summit

Talk Abstract Accumulo has a solid theoretical foundation, endowing it with huge scalability, high reliability, and the makings of class-leading performance for NoSQL operations. Several publications show Accumulo achieving multi-petabyte scalability and outperforming other databases in its class by orders of magnitude. However, there are challenges arising in practice that slow down that performance and introduce bottlenecks. The root of Accumulo's distributed scale and performance while maintaining consistency comes from a multi-level amplification. Zookeeper bootstraps the consistency with a highly durable quorum. The Accumulo root table uses buffering and caching to boost that performance for sorted key/value operations. With the metadata tablets and data tables, Accumulo continues to boost performance and divides and conqures a highly scalable key/value space to leverage the resources of a large cluster. The challenge arrises when metadata operations at the core of Accumulo bottleneck performance for the entire cluster. In this talk we will describe the Accumulo metadata operations model in detail. With a couple of prototypical application scenarios, we will show a few areas that are current bottlenecks or that we can expect to be bottlenecks in the near future. We will also propose modifications to the current model and outline projects that the community can take on to keep Accumulo in the lead for performance and scalability. Speaker Adam Fuchs Chief Technology Officer, Sqrrl As the Chief Technology Officer and co-founder of Sqrrl, Adam Fuchs is responsible for ensuring that Sqrrl is leading the world in Big Data Infrastructure technology. Previously at the National Security Agency, Adam was an innovator and technical director for several database projects, handling some of the world’s largest and most diverse data sets. He is a co-founder of the Apache Accumulo project. Adam has a BS in Computer Science from the University of Washington and has completed extensive graduate-level course work at the University of Maryland.

Accumulo Summit 2015: Ferrari on a Bumpy Road: Shock Absorbers to Smooth Out ...

Accumulo Summit

Talk Abstract Apache ZooKeeper plays a central role within the Accumulo architecture. Its quorum consistency model supports an overall Accumulo architecture with no single points of failure. Beyond that, Accumulo leverages ZooKeeper to store and communication configuration information for users and tables, as well as operational states of processes and tablets. For most Accumulo users, ZooKeeper is a black box full of goodness. Unfortunately, operational challenges mean we often have to delve into the dark depths to decipher what's going on when something goes wrong. In this talk, we will cover some basics about ZooKeeper's role, what it's good at and what it's not. Then we will discuss ways to debug what's stored inside of ZooKeeper, including how to overcome challenges with ZooKeeper's sometimes difficult ACL model. Speaker Michael Allen Security Architect, Sqrrl Michael Allen is Sqrrl's security architect. Before joining the team, Michael finished up 9 years working for PGP Corporation (and, post-acquisition, Symantec) in a variety of roles developing encryption software. In addition to encryption systems, Michael has extensive experience working with Java and Java-based web applications. He holds an MS in computer science from UC Santa Cruz, and a BA in Computer Science from Pomona College. When he's not making things up at work, Michael makes things up with other actors performing improvisational theater.

Accumulo Summit 2015: Zookeeper, Accumulo, and You [Internals]

Accumulo Summit

Talk Abstract Aggregation has long been a use case of Accumulo Iterators. Iterators' ability to reduce data during compaction and scanning can greatly simplify an aggregation system built on Accumulo. This talk will first review how Accumulo's Iterators/Combiners work in the context of aggregating values. I'll then step back and look at the abstraction of aggregation functions as commutative operations and the several benefits that result by making this abstraction. We will see how it becomes no harder to introduce powerful operations such as cardinality estimation and approximate top-k than it is to sum integers. I will show how to integrate these ideas into Accumulo with an example schema and Iterator. Finally, a practical aggregation use case will be discussed to highlight the concepts from the talk. Speakers Gadalia O'Bryan Senior Solutions Architect, Koverse Gadalia O'Bryan is a Sr. Solutions Architect at Koverse, where she leads customer projects and contributes to key feature and algorithm design, such as Koverse's Aggregation Framework. Prior to Koverse, Gadalia was a mathematician for the National Security Agency. She has an M.A. in mathematics from UCLA and has been working with Accumulo for the past 6 years. Bill Slacum Software Engineer, Koverse Bill is an Accumulo committer and PMC member who has been working on large scale query and analytic frameworks since 2010. He holds BS's in computer science and financial economics from UMBC. Having never used his passport to leave the United States, he is currently a national man of mystery.

Accumulo Summit 2015: Building Aggregation Systems on Accumulo [Leveraging Ac...

Accumulo Summit

Apache Accumulo and the Data Lake

Aaron Cordova

Talk Abstract As with all open-source databases, Accumulo developers often compete between building exciting new features and hacking on performance and stability. As the core features solidify and expand, we see many opportunities to improve performance. An effective methodology for performance improvement is scientific in nature, and follows a well-definite modeling and simulation approach, matching theory to experimentation in an iterative fashion. Ingest performance is one of the most differentiating characteristics of Accumulo. However, there is much room for improvement for typical ingest-heavy applications. Accumulo supports two mechanisms to bring data in: streaming ingest and bulk ingest. In bulk ingest, the goal is to maximize throughput without constraining latency. Bulk ingest involves creating a set of files that conform to Accumulo's internal RFile format and then registering those files with Accumulo. MapReduce provides a framework for generating, sorting, and storing key/value pairs, which form the primary elements of preparing RFiles for bulk ingest. MapReduce has been used many times over the years to break sorting records, such as Terasort. We can expect it is a reasonable choice for maximizing bulk ingest throughput. However, the theory often proves challenging to implement as there are many performance pitfalls along the way. In this talk, we dive deep into optimizing MapReduce for Accumulo bulk ingest. We share detailed theoretical and empirical performance models, we discuss techniques for profiling performance, and we suggest reusable techniques for squeezing the maximum performance out of enterprise-grade Accumulo bulk ingest. Speaker Chris McCubbin Director of Data Science, Sqrrl Chris is the Director of Data Science for Sqrrl. He has extensive experience with the Hadoop ecosystem and applying scientific computation algorithms to real-world datasets. Previously, Chris developed Big Data analysis tools for the Intelligence Community and applied artificial intelligence techniques to unmanned vehicle systems. He holds a MS in Computer Science and BS in Computer Science and Mathematics from the University of Maryland.

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Accumulo Summit

Cloudera Breakfast Series, Analytics Part 1: Use All Your Data

Cloudera, Inc.

Dutch Interactive Awards - Nominees and Jury feedback

Antoaneta Kyoseva

Cloudera Federal Forum 2014: A 360 Degree View of the Insider Threat

Cloudera, Inc.

Introducing Cloudera Director at Big Data Bash

Andrei Savu

Introduction to Apache Accumulo

Aaron Cordova

Cloudera vs Hortonworks - Similarities and Differencies Hortonworks Apache Ranger Apache Knox Apache Falcon Cloudera Project Rhino Project Sentry HADOOP (HDFS) (C+H) Res. Manager Yarn (C+H) Warehouse DB Presto (H) MapReduce PIG(C+H) Search Engines SolrCloud (C+H) Analytics Columnar Store Accumulo (C+H) Impala(C) Machine Learning Spark ML (C+H) Mahout(H) HBase(C+H) Data Streaming Storm(H) Spark Streaming(C+H) HIVE (C+H)

Big data advance topics - part 2.pptx

Moldovan Radu Adrian

Cloud environments are increasingly becoming a popular deployment option for Hadoop. Enterprises can take advantage of the added flexibility and elasticity of the cloud for both long-running clusters, temporary deployments or for spikey workloads. However, as more and more users choose cloud environments for critical Hadoop workloads, they are often forced to compromise on key aspects of their data platform. Cloudera Director enables the full fidelity of the Enterprise Data Hub in the cloud, without compromises. Announced with the recent 5.2 release, Cloudera Director is the simple, reliable way to deploy and scale Hadoop in the cloud, while maintaining an open and neutral platform with enterprise-grade capabilities. During this webinar, Tushar Shanbhag, Director of Product Management, will look at why Hadoop cloud environments are becoming so popular and some of the challenges around Hadoop in the cloud. He will then provide an in-depth overview of Cloudera Director, its key features, and how it alleviates these common challenges. Finally, he will discuss some key use cases and provide insight into what’s next for Cloudera and Hadoop in the cloud.

Cloudera Director: Unlock the Full Potential of Hadoop in the Cloud

Cloudera, Inc.

Your data is a strategic asset, and the benefits of your Hadoop deployment hinge on uptime, reliability, and expert guidance. Companies that have a specific plan with clear success criteria, benchmarks, and production goals are able to get the most from their journey to enterprise data hub (EDH), starting on day one. By leveraging insights from a proactive study of tens of thousands of nodes under subscription, Cloudera Enterprise customers can even minimize downstream issues before they occur. Learn how Cloudera: - Helps eliminate known issues and avoid common cluster misconfigurations - Guides better utilization of the EDH according to comparative analysis - Ensures enterprises optimize support resources for faster issue resolution

The Benefits of Predictive and Proactive Support for an Enterprise Data Hub

Cloudera, Inc.

Samsung’s First 90-Days Building a Next-Generation Analytics Platform

Cloudera, Inc.

Nhom 16 big data

Duy Phan

Destaque (20)

Machine Learning & Graph Processing w/ Spark and Accumulo

Accumulo Summit 2015: Fraud Analytics using Accumulo, Julia and Fast SQL [Lev...

Accumulo Summit 2016: Accumulo Indexing Strategies for Searching Semantic Net...

Accumulo Summit 2015: Accumulo 2.0: A New Client API [API]

Accumulo Summit 2015: Ferrari on a Bumpy Road: Shock Absorbers to Smooth Out ...

Accumulo Summit 2015: Zookeeper, Accumulo, and You [Internals]

Accumulo Summit 2015: Building Aggregation Systems on Accumulo [Leveraging Ac...

Apache Accumulo and the Data Lake

Accumulo Summit 2015: Performance Models for Apache Accumulo: The Heavy Tail ...

Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?

Cloudera Breakfast Series, Analytics Part 1: Use All Your Data

Dutch Interactive Awards - Nominees and Jury feedback

Cloudera Federal Forum 2014: A 360 Degree View of the Insider Threat

Introducing Cloudera Director at Big Data Bash

Introduction to Apache Accumulo

Big data advance topics - part 2.pptx

Cloudera Director: Unlock the Full Potential of Hadoop in the Cloud

The Benefits of Predictive and Proactive Support for an Enterprise Data Hub

Samsung’s First 90-Days Building a Next-Generation Analytics Platform

Nhom 16 big data

Semelhante a SQL on Accumulo

SQLCAT: Tier-1 BI in the World of Big Data

Denny Lee

Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets. If you want to do multi-dimension analysis on large data sets (billion+ rows) with low query latency (sub-seconds), Kylin is a good option. Kylin also provides seamless integration with existing BI tools (e.g Tableau).

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Xu Jiang

Connecting Hadoop and Oracle

Tanel Poder

The real estate market is one of the most competitive in terms of pricing, and as a result, prices tend to vary significantly based on a variety of factors. Forecasting property prices is an important module in decision making for both buyers and investors in supporting budget allocation, finding property finding stratagems, and determining suitable policies, making it one of the top fields to apply the concepts of machine learning to optimise and predict the prices with high accuracy. The literature study provides a clear concept and will benefit any next endeavours. The majority of writers have come to the conclusion that artificial neural networks are more effective at forecasting the future, but in the actual world, there are other algorithms that should have been taken into account. In order to maximise profits, investors base their judgments on market trends. Developers are curious in future trends because it might help them weigh the advantages and downsides and assist them create new products.

Hive_Pig.pptx

PAVANKUMARNOOKALA

Etu Solution Day 2014 Track-D: 掌握Impala和Spark

James Chen

SQL on Hadoop benchmarks using TPC-DS query set

Kognitio

Hadoop for the Absolute Beginner

Ike Ellis

Building a reliable pipeline of data ingress, batch computation, and data egress with Hadoop can be a major challenge. Most folks start out with cron to manage workflows, but soon discover that doesn't scale past a handful of jobs. There are a number of open-source workflow engines with support for Hadoop, including Azkaban (from LinkedIn), Luigi (from Spotify), and Apache Oozie. Having deployed all three of these systems in production, Joe will talk about what features and qualities are important for a workflow system.

Workflow Engines for Hadoop

Joe Crobak

Building a Web Application with Kafka as your Database

confluent

Big SQL 3.0: Datawarehouse-grade Performance on Hadoop - At last!

Nicolas Morales

SQL Police

Marcus Davage

This session will cover a series of use cases where you can store your data cheaply in files and analyze the data with Apache Spark, as well as use cases where you want to store your data into a different data source to access with Spark DataFrames. Here’s an example outline of some of the topics that will be covered in the talk: Use cases to store in file systems for use with Apache Spark: - Analyzing a large set of data files. - Doing ETL of a large amount of data. - Applying Machine Learning & Data Science to a large dataset. - Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.

Not your Father's Database: Not Your Father’s Database: How to Use Apache® Sp...

Databricks

If you also got the Big Data itch, here is something to ease the pain :-) Answers to this questions will be available soon (more info in the attached link) Which Big Data Appliance should YOU use? (click on the attached link for Poll results) Appliances are Small and Quick, Right? Revealing the 6 Types of Big Data Appliances Uncovering the Main Players Challenges, Pitfalls, and Winning the Big Data Game Where is all this leading YOU to?

Piranha vs. mammoth predator appliances that chew up big data

Jack (Yaakov) Bezalel

Kb 40 kevin_klineukug_reading20070717[1]

shuwutong

Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014

cdmaxime

Analyzing transactional data residing in Oracle databases is becoming increasingly common, especially as the data sizes and complexity increase and transactional stores are no longer to keep pace with the ever-increasing storage. Although there are many techniques available for loading Oracle data, getting up-to-date data into your data warehouse store is a more difficult problem. VMware Continuent provides provides data replication from Oracle to data warehouses and analytics engines, to derive insight from big data for better business decisions. Learn practical tips on how to get your data warehouse loading projects off the ground quickly and efficiently when replicating from Oracle into Hadoop, Amazon Redshift, and HP Vertica.

Replicate from Oracle to data warehouses and analytics

Continuent

Data scientists write SQL queries everyday. Very often they know how to write correct queries but don’t know why their queries are slow. This is more obvious in Spark than in Redshift as Spark requires additional tuning such as caching while Redshift does heavy lifting behind the scene. In this talk I will cover a few lessons we learned from migrating one of the biggest table here (900M+ rows/day) from AWS Redshift to Spark. Specifically: – Why and how do we migrate? – How do we tune the query for Spark to gain 10x speed vs direct translated from Redshift – How do we scale the team on Spark (with 80+ people in our data science team)

Migrating from Redshift to Spark at Stitch Fix: Spark Summit East talk by Sky...

Spark Summit

Meir Dudai - Concurrency

sqlserver.co.il

Dat308

sqlserver.co.il

Tableau on Hadoop Meet Up: Advancing from Extracts to Live Connect

Remy Rosenbaum

Semelhante a SQL on Accumulo (20)

SQLCAT: Tier-1 BI in the World of Big Data

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Connecting Hadoop and Oracle

Hive_Pig.pptx

Etu Solution Day 2014 Track-D: 掌握Impala和Spark

SQL on Hadoop benchmarks using TPC-DS query set

Hadoop for the Absolute Beginner

Workflow Engines for Hadoop

Building a Web Application with Kafka as your Database

Big SQL 3.0: Datawarehouse-grade Performance on Hadoop - At last!

SQL Police

Not your Father's Database: Not Your Father’s Database: How to Use Apache® Sp...

Piranha vs. mammoth predator appliances that chew up big data

Kb 40 kevin_klineukug_reading20070717[1]

Cloudera Impala - Las Vegas Big Data Meetup Nov 5th 2014

Replicate from Oracle to data warehouses and analytics

Migrating from Redshift to Spark at Stitch Fix: Spark Summit East talk by Sky...

Meir Dudai - Concurrency

Dat308

Tableau on Hadoop Meet Up: Advancing from Extracts to Live Connect

Mais de Donald Miner

Machine Learning Vital Signs: Metrics and Monitoring of AI in Production This talk details the tracking of machine learning models in production to ensure model reliability, consistency, and performance into the future. Production models are interacting with the real world, and it is terrifying that often times nobody has any idea how they are performing on live data. The world changes! Bias and variance can creep into your models over time and you should know when that happens.

Machine Learning Vital Signs

Donald Miner

EDHREC @ Data Science MD

Donald Miner

Donald Miner will do a quick introduction to Apache Hadoop, then discuss the different ways Python can be used to get the job done in Hadoop. This includes writing MapReduce jobs in Python in various different ways, interacting with HBase, writing custom behavior in Pig and Hive, interacting with the Hadoop Distributed File System, using Spark, and integration with other corners of the Hadoop ecosystem. The state of Python with Hadoop is far from stable, so we'll spend some honest time talking about the state of these open source projects and what's missing will also be discussed.

Hadoop with Python

Donald Miner

This talk will go over table design and row key design approaches for indexing large amounts of data in Apache Accumulo. We'll do an overview of how to store geographical data, entity relationship graphs, natural language text, numbers, and more in Accumulo. This will serve as a starting point to learning how to effectively store different types of data in Accumulo as well as showcase the capabilities of Accumulo for handling varying situations.

Survey of Accumulo Techniques for Indexing Data

Donald Miner

Data, The New Currency

Donald Miner

Speaker: Steve Touw, CTO, 42six Solutions a CSC Company Amino is an open source analytical framework that focuses on a “building-blocks” approach to data discovery by pre-computing features about data at the most granular level possible and then allows analysts and data scientists to easily combine those features into more complex questions. The magic behind Amino is found in it’s custom Accumulo index; that index strives to provide fast scans, highly dimensional scans, data compression, and a simple query structure. The index leverages Accumulo iterators to do much of the scan time logic which has no limit on dimensionality of the query. Iterators are what makes Accumulo unique and enables the Amino index to execute the complex queries.

The Amino Analytical Framework - Leveraging Accumulo to the Fullest

Donald Miner

Hadoop for Data Science

Donald Miner

MapReduce Design Patterns

Donald Miner

Data science and Hadoop

Donald Miner

Mais de Donald Miner (9)

Machine Learning Vital Signs

EDHREC @ Data Science MD

Hadoop with Python

Survey of Accumulo Techniques for Indexing Data

Data, The New Currency

The Amino Analytical Framework - Leveraging Accumulo to the Fullest

Hadoop for Data Science

MapReduce Design Patterns

Data science and Hadoop

Último

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Histor y of HAM Radio presentation slide

vu2urc

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

SQL on Accumulo

1. SQL-on-Accumulo A talk about how we may be able to do it one day DONALD MINER 1 May 5th, 2014

2. A brief history of Hadoop and SQL 2 time 1980 2000

3. A brief history of Hadoop and SQL 3 time 1980 2000

4. 4 BIG DATASQL

5. A brief history of Hadoop and SQL 5 time 1980 2000 SQL-on-Hadoop

6. SQL-on-Accumulo would be nice Problem: Accumulo is just a data store  We’ll have to do query somewhere else 6

7. 7 WWHBD?(What Would HBase Do?)

8. WWHBD? - Hive • Hive  Runs in MapReduce  Map col family and col qualifiers to columns  Maintained by Hive community • Impala and Shark inherit functionality from Hive 8

9. WWHBD? - next level Problem: Hive, Impala, and Shark don’t know how HBase works … and don’t care • Apache Phoenix  Specifically SQL-on-HBase  Currently Apache incubator project  Client-embedded JDBC driver  Uses series of scans and coprocessors • Pivotal’s HAWQ and PXF  PXF is external table functionality in HAWQ  Native support for HAWQ: uses push down filters, range scans, etc. to efficiently slurp data into HAWQ 9

10. ACCUMULO-143 people. technology. integrity. 10

11. SQL-on-Accumulo Status Hive (and somewhat Impala and Shark) • Github project by Brian Femiano [1]  Doesn’t work on new versions  Hasn’t been touched in 9 months  Wasn’t committed into trunk • Some rumors that some orgs have done it themselves (but no public information) people. technology. integrity. 11 [1] https://github.com/bfemiano/accumulo-hive-storage-manager (google for “accumulo hive”)

12. SQL-on-Accumulo Status Phoenix • Discussion on mailing list last week • Some differences between iterators and coprocessors makes this interesting Pivotal’s HAWQ and PXF • In development • Will support visibility labels • Pushdown and optimizations with iterators people. technology. integrity. 12

13. Visibility Design Problems 13 These problems are unique to Accumulo • SELECT and visibility labels  Assume two cells, only uniqueness is visibility… Which do I pick in a SELECT?  Timestamps have this problem, but have a logical assumption (most recent) • Authorizations in SQL  How do you tell the execution engine which authorizations to use?  Table definition? (hard to change)  SQL statement? (extend SQL language?)  Based on login? (how do you downgrade?)

14. What are the next steps? I guess that’s up to the community 14

15. QUIZ: What is this definition trying to say? Big Data: • Volume • Variety • Velocity • Veracity 15 A warning about SQL-on-Accumulo

16. QUIZ: What is this definition trying to say? Big Data: • Volume • Variety • Velocity • Veracity Answer: RDBMS/SQL suck at all these things 16 A warning about SQL-on-Accumulo

17. QUIZ: What is this definition trying to say? Big Data: • Volume • Variety • Velocity • Veracity Answer: RDBMS/SQL suck at all these things 17 A warning about SQL-on-Accumulo What does SQL-on-Accumulo still suck at? *Added context for my internet viewers since this could be controversial if taken literally and I’m not talking to my slides: I’m trying to say that SQL-on-X can’t solve all of the worlds problems, but it can solve a good number of them very well. It also tees up the conversation that SQL is not the end-all-be-all… there are ways that it could be made better to adapt to “the big data use case”. Don’t take this the wrong way, SQL-on-Hadoop and SQL-on- Accumulo would be incredible useful, but it doesn’t solve 100% of the problems.

18. SQL-on-Accumulo DONALD MINER 18 dminer@clearedgeit.com @donaldpminer Questions?

SQL on Accumulo

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a SQL on Accumulo

Semelhante a SQL on Accumulo (20)

Mais de Donald Miner

Mais de Donald Miner (9)

Último

Último (20)

SQL on Accumulo