Building Data Applications with Apache Druid

•

0 gostou•275 visualizações

One of the most popular use cases for Apache Druid is building data applications. Data applications exist to deliver data into the hands of everyone on a team in a business, and are used by these teams to make faster, better decisions. To fulfill this role, they need to support granular drill down, because the devil is in the details, but also be extremely fast, because otherwise people won't use them! In this talk, Gian Merlino will cover: *The unique technical challenges of powering data-driven applications *What attributes of Druid make it a good platform for data applications *Some real-world data applications powered by Druid

Tecnologia

Building data applications
with Apache Druid
October 2020
Gian Merlino, Cofounder & CTO, Imply
1

Who am I?
2
Gian Merlino
Committer & PMC chair at
Cofounder at (we’re hiring!)

Druid in the wild
5
100+ billion rows/day
1+ trillion rows, 1+ year retained
100s of servers
sub-second to few seconds query latency
mix of streaming and batch ingest

Online DB pattern
6
Data lakeData
source
Data mover
(Apache Kafka,
Apache Airflow, etc)
Data
source
Data
source
Query
engine
Query
engine
Query
engine
Pure
storage
Data lake
Data lake
Direct to lake
Online DB
Stores an
optimized
copy
Online
app

7
● Scale-out, fault-tolerant
architecture
● No downtime for software updates
● No downtime for data
management
● Heavily optimized storage format
● Integrated storage format and
query engine
Druid as an online DB

Data apps
● Interactive query speeds
● Always online
● Fresh data from streams
● Quality of service
● Price/performance
8

Interactive query speeds
9
Secondary indexes
Operate on
compressed data Late materializationCompression
INDEX
[0,1,2](11100000)
[3,4] (00011000)
[5,6,7](0000111)
DATA
0
0
0
1
1
2
2
2
DICT
DC = 0
LA = 1
SF = 2

Interactive query speeds
10
artist
(STRING)
__time
(LONG)
1293840000000
1293840000000
1293840000000
1293840000000
1293840000000
1293840000000
1293840000000
1293840000000
DATA
DICT
INDEX
0
0
0
1
1
2
2
2
Justin = 0
Ke$ha = 1
Miley = 2
[0,1,2](11100000)
[3,4](00011000)
[5,6,7](0000111)
25
42
17
170
112
67
53
94
DATA2
1
2
1
1
0
0
0
[0,2](10100000)
[1,3,4](01011000)
[5,6,7](00000111)
DICT
DC = 0
LA = 1
SF = 2
INDEX
1800
2912
1953
3194
5690
1100
8423
9080
city
(STRING)
count
(LONG)
price
(LONG)
Dictionary encoded
(sorted)
Bitmap index
(stored compressed)

Always online
11
Coordinator
Apache
ZooKeeper
Master server
Historical Indexer Historical Indexer
Data server
Deep storage
Broker
Query server
Streaming
data
Batch
data

Fresh data from streams
12
Coordinator
Apache
ZooKeeper
Master server
Historical Indexer Historical Indexer
Data server
Deep storage
Broker
Query server
Streaming
data
Batch
data

Quality of service
14
Coordinator
Apache
ZooKeeper
Master server
Historical Indexer Historical Indexer
Data server
Deep storage
Broker
Query server
Streaming
data
Batch
data

Price/performance
Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid.
vs. leading open source SQL engines

Price/performance
Data sourced from Imply benchmarks.
vs. a leading cloud-based data warehouse

Stay in touch
18
@druidio
Join the community
(Mailing lists, Slack, meetups)
https://druid.apache.org/community/
Follow the Druid project on Twitter!

Mais conteúdo relacionado

Mais procurados

Data Analytics and Processing at Snap - Druid Meetup LA - September 2018

Charles Allen

Target is one of the largest retailers in the United States, with brick-and-mortar stores in all 50 states and one of the most-visited ecommerce sites in the country. In addition to typical merchandising functions like assortment planning, pricing and inventory management, Target also operates a large supply chain, financial/banking operations and property management organizations. As a data-driven organization, we need a data analytics platform that can address the unique needs of each of these various business units, while scaling to hundreds of thousands of users and accommodating an ever-increasing amount of data. In this talk we’ll cover why Target chose to create our own analytics platform and specifically how Druid makes this platform successful. We’ll cover how we utilize key features in Druid, such as union datasources, arbitrary granularities, real-time ingestion, complex aggregation expressions and lightning-fast query response to provide analytics to users at all levels of the organization. We’ll also cover how Druid’s speed and flexibility allow us to provide interactive analytics to front-line, edge-of-business consumers to address hundreds of unique use-cases across several business units.

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

Imply

Benchmarking Apache Druid

Matt Sarrel

Apache Druid ingests and enables instant query on many billions of events in real-time. But how? In this talk, each of the components of an Apache Druid cluster is described – along with the data and query optimisations at its core – that unlock fresh, fast data for all. Bio: Peter Marshall (https://linkedin.com/in/amillionbytes/) leads outreach and engineering across Europe for Imply (http://imply.io/), a company founded by the original developers of Apache Druid. He has 20 years architecture experience in CRM, EDRM, ERP, EIP, Digital Services, Security, BI, Analytics, and MDM. He is TOGAF certified and has a BA (hons) degree in Theology and Computer Studies from the University of Birmingham in the United Kingdom.

Druid: Under the Covers (Virtual Meetup)

Imply

Check out the webinar: https://imply.io/videos/whats-new-imply-3-3-apache-druid-0-18 The most recent Imply 3.3 release, based on Apache 0.18 brings several major new features, including joins, query laning and Clarity Alerts. These new features deliver increased design flexibility during design, and provide improved ingestion performance, and sub-second response times to help accelerate data warehouse and data lake deployments, and add real-time analytics in general.

What’s New in Imply 3.3 & Apache Druid 0.18

Imply

Apache Druid Design and Future prospect

c-bslim

Check out the full webinar: https://imply.io/videos/why-data-warehouses-cannot-support-hot-analytics Today’s data warehouses - whether traditional, specialized or cloud-based - are good at supporting cold analytics, such as reporting, where query times can take minutes. But they cannot cost-effectively support hot analytics—interactive ad hoc analytics usually performed by larger groups of users against batch or streaming data. Examples of hot analytics include clickstream analytics; service, network and application performance monitoring; and risk analytics. Data warehouses struggle with hot analytics use cases because they are too slow, unable to scale, or too expensive. Learn how a new class of real-time data platforms overcome these limitations, and how companies implement a “temperature-based” approach to analytics.

Why data warehouses cannot support hot analytics

Imply

A Non-Standard use Case of Hadoop: High Scale Image Processing and Analytics

DataWorks Summit

Abstract:- BI and analytics are at the top of corporate agendas. Competition is intense, and, more than ever, organizations require fast access to insights about their customers, markets, and internal operations to make better decisionsäóîoften, in real time. Enterprises face challenges powering real-time business analytics and systems of engagement (SOEs). Analytic applications and SOEs need to be fast and consistent, but traditional database approaches, including RDBMS and first-generation NoSQL solutions, can be complex, a challenge to maintain, and costly. Companies should aim to simplify traditional systems and architectures while also reducing vendors. One way to do this is by embracing an emerging hybrid memory architecture, which removes an entire caching layer from your front-end application. This talk discusses real-world examples of implementing this pattern to improve application agility and reduce operational database spend.

Real-Time Analytics in Transactional Applications by Brian Bulkowski

Data Con LA

Insights into Customer Behavior from Clickstream Data by Ronald Nowling

Spark Summit

Optimizing Presto Connector on Cloud Storage

Kai Sasaki

July 2014 HUG : Pushing the limits of Realtime Analytics using Druid

Yahoo Developer Network

We went through the journey of deploying Apache Druid clusters on Kubernetes(K8s) and created a druid-operator (https://github.com/druid-io/druid-operator). This talk introduces the druid kubernetes operator, how to use it to deploy druid clusters and how it works under the hood. We will share how we use this operator to deploy Druid clusters at Splunk. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Druid is a complex stateful distributed system and a Druid cluster consists of multiple web services such as Broker, Historical, Coordinator, Overlord, MiddleManager etc each deployed with multiple replicas. Deploying a single web service on K8s requires creating few K8s resources via YAML files and it multiplies due to multiple services inside of a Druid cluster. Now doing it for multiple Druid clusters (dev, staging, production environments) makes it even more tedious and error prone. K8s enables creation of application (such as Druid) specific extension, called “Operator”, that combines kubernetes and application specific knowledge into a reusable K8s extension that makes deploying complex applications simple.

Splunk: Druid on Kubernetes with Druid-operator

Imply

Do you know who is knocking on your network’s door? Have new regulations left you scratching your head on how to handle what is happening in your network? Network flow data helps answer many questions across a multitude of use cases including network security, performance, capacity planning, routing, operational troubleshooting and more. Today’s modern day streaming data pipelines need to include tools that can scale to meet the demands of these service providers while continuing to provide responsive answers to difficult questions. In addition to stream processing, data needs to be stored in a redundant, operationally focused database to provide fast, reliable answers to critical questions. Together, Kafka and Druid work together to create such a pipeline. In this talk Eric Graham and Rachel Pedreschi will discuss these pipelines and cover the following topics: -Network flow use cases and why this data is important. -Reference architectures from production systems at a major international Bank. -Why Kafka and Druid and other OSS tools for Network Flows. -A demo of one such system.

How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi, Imply ...

confluent

It is common for consumer Internet companies to start off with popular third-party tools for analytics needs. Then, when the user base and the company grows, they end up building their own analytics data pipeline and query engine to cope with their data scale, satisfy custom data enrichment and reporting needs and achieve high quality of their data. That’s exactly the path that was taken at Grammarly, the popular online proofreading service. In this session, Grammarly will share how they improved business and marketing analytics, previously done with Mixpanel, by building their own in-house analytics engine and application on top of Apache Spark. Chernetsov wil touch upon several Spark tweaks and gotchas that they experienced along the way: – Outputting data to several storages in a single Spark job – Dealing with Spark memory model, building a custom spillable data-structure for your data traversal – Implementing a custom query language with parser combinators on top of Spark sql parser – Custom query optimizer and analyzer when you want not exactly sql – Flexible-schema storage and query against multi-schema data with schema conflicts – Custom aggregation functions in Spark SQL

Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...

Databricks

The architecture of data analytics PaaS on AWS

Treasure Data, Inc.

LendingClub RealTime BigData Platform with Oracle GoldenGate

Rajit Saha

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko

Data Con LA

You run your SQL-centric infrastructure for 10 years and slowly starting to note you can’t do this way anymore – everything is getting too expensive but your business requires things which are simply impossible without radical changes. This is exact situation we had 2 years before. So we’d like to show our experience: - Why and how we came into Big Data? - Why we choose Apache and Hadoop? - What to do and what is already done? - What lessons were learned? - Hadoop and relational databases: fight or synergy? - Reactive Big Data manifest.

BIG DATA: From mammoth to elephant

Roman Nikitchenko

Programmatic Bidding Data Streams & Druid

Charles Allen

Mais procurados (20)

Data Analytics and Processing at Snap - Druid Meetup LA - September 2018

Building an Enterprise-Scale Dashboarding/Analytics Platform Powered by the C...

Benchmarking Apache Druid

Druid: Under the Covers (Virtual Meetup)

What’s New in Imply 3.3 & Apache Druid 0.18

Apache Druid Design and Future prospect

Why data warehouses cannot support hot analytics

A Non-Standard use Case of Hadoop: High Scale Image Processing and Analytics

Real-Time Analytics in Transactional Applications by Brian Bulkowski

Insights into Customer Behavior from Clickstream Data by Ronald Nowling

Optimizing Presto Connector on Cloud Storage

July 2014 HUG : Pushing the limits of Realtime Analytics using Druid

Splunk: Druid on Kubernetes with Druid-operator

How To Use Kafka and Druid to Tame Your Router Data (Rachel Pedreschi, Imply ...

Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...

The architecture of data analytics PaaS on AWS

LendingClub RealTime BigData Platform with Oracle GoldenGate

NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko

BIG DATA: From mammoth to elephant

Programmatic Bidding Data Streams & Druid

Semelhante a Building Data Applications with Apache Druid

Data Con LA 2020 Description Apache Druid is a cloud-native open-source database that enables developers to build highly-scalable, low-latency, real-time interactive dashboards and apps to explore huge quantities of data. This column-oriented database provides the microsecond query response times required for ad-hoc queries and programmatic analytics. Druid natively streams data from Apache Kafka (and more) and batch loads just about anything. At ingestion, Druid partitions data based on time so time-based queries run significantly faster than traditional databases, plus Druid offers SQL compatibility. Druid is used in production by AirBnB, Nielsen, Netflix and more for real-time and historical data analytics. This talk provides an introduction to Apache Druid including: Druid's core architecture and its advantages, Working with streaming and batch data in Druid, Querying data and building apps on Druid and Real-world examples of Apache Druid in action Speaker Matt Sarrel, Imply Data, Developer Evangelist

Apache Druid 101

Data Con LA

Building IoT and Big Data Solutions on Azure

Ido Flatow

Streaming Data and Stream Processing with Apache Kafka

confluent

Big data remains a rapidly evolving field with new applications and infrastructure appearing every year. In this talk, Matei Zaharia will cover new trends in 2016 / 2017 and how Apache Spark is moving to meet them. In particular, he will talk about work Databricks is doing to make Apache Spark interact better with native code (e.g. deep learning libraries), support heterogeneous hardware, and simplify production data pipelines in both streaming and batch settings through Structured Streaming. Speaker: Matei Zaharia Video: http://go.databricks.com/videos/spark-summit-east-2017/what-to-expect-big-data-apache-spark-2017 This talk was originally presented at Spark Summit East 2017.

What to Expect for Big Data and Apache Spark in 2017

Databricks

MongoDB .local Houston 2019: Building an IoT Streaming Analytics Platform to ...

MongoDB

Watch full webinar here: https://bit.ly/2QaVfE7 Faster, more agile data management is at the heart of government modernization. However, Traditional data delivery systems are limited in realizing a modernized and future-proof data architecture. This webinar will address how data virtualization can modernize existing systems and enable new data strategies. Join this session to learn how government agencies can use data virtualization to: - Enable governed, inter-agency data sharing - Simplify data acquisition, search and tagging - Streamline data delivery for transition to cloud, data science initiatives, and more

Modern Data Management for Federal Modernization

Denodo

Qo Introduction V2

Joe_F

data-mesh-101.pptx

TarekHamdi8

Game Analytics at London Apache Druid Meetup

Jelena Zanko

Confluent kafka meetupseattle jan2017

Nitin Kumar

We will take a look at an introduction and overview of Azure Analysis Services: Microsoft‘s cloud-based analytical engine and Platform as a Service (PaaS) offerings and how to leverage SQL Server Data Tools to build and deploy a tabular data model to Azure Analysis Services. We will then connect with Power BI Desktop and the Power BI portal to build visualizations. We will discuss Azure Analysis Services features and capabilities, use cases, provisioning and deployment, managing and monitoring, tools, and report creation. Azure Analysis Service became Globally Available in April 2017, and Power BI has released several major updates as well.

Leveraging Azure Analysis Services Tabular Data Models with Power BI by Tim M...

KTL Solutions

Life is a database. How you manage data defines business. ALTIBASE HDB with its Hybrid architecture combines the extreme speed of an In-Memory Database with the storage capacity of an On-Disk Database’ in a single unified engine. ALTIBASE® HDB™ is the only Hybrid DBMS in the industry that combines an in-memory DBMS with an on-disk DBMS, with a single uniform interface, enabling real-time access to large volumes of data, while simplifying and revolutionizing data processing. ALTIBASE XDB is the world’s fastest in-memory DBMS, featuring unprecedented high performance, and supports SQL-99 standard for wide applicability. Altibase is provider of In-Memory data solutions for real-time access, analysis and distribution of high volumes of data in mission-critical environments. Please visit our website (www.altibase.com) to learn more about our products and read more about our case studies. Or contact us at info@altibase.com. We look forward to helping you!

The Most Trusted In-Memory database in the world- Altibase

Altibase

Data Infrastructure at LinkedIn

Amy W. Tang

Data Streaming with Apache Kafka & MongoDB - EMEA

Andrew Morgan

Webinar: Data Streaming with Apache Kafka & MongoDB

MongoDB

Tapdata Product Intro

Tapdata

This session is recommended for anyone interested in understanding how to use AWS big data services to develop real-time analytics applications. In this session, you will get an overview of a number of Amazon's big data and analytics services that enable you to build highly scaleable cloud applications that immediately and continuously analyze large sets of distributed data. We'll explain how services like Amazon Kinesis, EMR and Redshift can be used for data ingestion, processing and storage to enable real-time insights and analysis into customer, operational and machine generated data and log files. We'll explore system requirements, design considerations, and walk through a specific customer use case to illustrate the power of real-time insights on their business.

Getting Started with Real-time Analytics

Amazon Web Services

A Gen3 Perspective of Disparate Data

Robert Grossman

Most data visualisation solutions today still work on data sources which are stored persistently in a data store, using the so called “data at rest” paradigms. More and more data sources today provide a constant stream of data, from IoT devices to Social Media streams. These data stream publish with high velocity and messages often have to be processed as quick as possible. For the processing and analytics on the data, so called stream processing solutions are available. But these only provide minimal or no visualisation capabilities. One was is to first persist the data into a data store and then use a traditional data visualisation solution to present the data. If latency is not an issue, such a solution might be good enough. An other question is which data store solution is necessary to keep up with the high load on write and read. If it is not an RDBMS but an NoSQL database, then not all traditional visualisation tools might already integrate with the specific data store. An other option is to use a Streaming Visualisation solution. They are specially built for streaming data and often do not support batch data. A much better solution would be to have one tool capable of handling both, batch and streaming data. This talk presents different architecture blueprints for integrating data visualisation into a fast data solution and highlights some of the products available to implement these blueprints.

Streaming Visualization

Guido Schmutz

Many companies today move mountains of data using ETL (extract, transform, load) technology. But data volumes are growing too large to move, customers are now expecting real-time data, and ETL costs now account for 10-15% of computing capacity. In this slide presentation, you can see how data virtualization enables data structures that were designed independently to be leveraged together, in real time, and without data movement, reducing complexity, lowering IT costs, and minimizing risk.

Data Virtualization and ETL

Lily Luo

Semelhante a Building Data Applications with Apache Druid (20)

Apache Druid 101

Building IoT and Big Data Solutions on Azure

Streaming Data and Stream Processing with Apache Kafka

What to Expect for Big Data and Apache Spark in 2017

MongoDB .local Houston 2019: Building an IoT Streaming Analytics Platform to ...

Modern Data Management for Federal Modernization

Qo Introduction V2

data-mesh-101.pptx

Game Analytics at London Apache Druid Meetup

Confluent kafka meetupseattle jan2017

Leveraging Azure Analysis Services Tabular Data Models with Power BI by Tim M...

The Most Trusted In-Memory database in the world- Altibase

Data Infrastructure at LinkedIn

Data Streaming with Apache Kafka & MongoDB - EMEA

Webinar: Data Streaming with Apache Kafka & MongoDB

Tapdata Product Intro

Getting Started with Real-time Analytics

A Gen3 Perspective of Disparate Data

Streaming Visualization

Data Virtualization and ETL

Mais de Imply

We have rearchitected Pivot from the ground up for enhanced dimensional analysis while ensuring that it is even faster, if that was even possible. Pivot 2.0 has plenty of new ways for you to visualize your data so that you can figure out the complex relationships between your data and enhanced the comparative analysis to quickly gain insight. In this webinar, will walk you through the exciting new features that are coming soon to Pivot.

Pivot 2.0 - The next generation visualization tool for your streaming data

Imply

Nicolas Trésegnie, Chief Architect at SuperAwesome Abstract: SuperAwesome's mission is to make the internet safer for kids. At the core of SuperAwesome's analytics is Druid. In this talk, we walk through how we run Druid on spot instances. We explain the consequences in terms of cost and reliability, how we managed to build a reliable system despite the risks, and how you could do the same. Nicolas works as Chief Architect at SuperAwesome, where is is looking after the overall architecture of the systems and the infrastructure. He is all about automation and how technology can be used to achieve business goals. Nicolas studied Computer Science and Bioinformatics, and he is now pursuing an MBA at Imperial.

Druid in Spot Instances

Imply

Druid has been the production workhorse for the past 2+ years at Zeotap powering the core Audience planning across our Connect and Targeting products. Though Druid is best suited for data having time as a dimension as it partitions data based on time first, we have used Druid to serve ML powered enhanced insights and Estimation of potential dataset sizes, to assist us with our core business case of Audience planning. These are datasets without timestamp a.k.a non-temporal with high scale and having nested dimensions. These have been achieved using nuanced data modelling to store the data sets and achieve millisecond latency retrieval on top of the same. The core of the presentation would be on the data modelling journey to achieve these use cases detailing the query access patterns. We also delve upon the architecture - ingestion into druid sink and processing including ML. In the end we go over the production setup and configurations and provide the performance tunings applied. The presentation would have the following heads: The presentation would have the following heads * Business case in Ad-Tech and Mar-Tech vertical * Audience Planner Usecase 1 - Insights -Lambda Architecture and data flow -Deep dive on data model -Takeaways *Audience Planner Usecase 2 - Estimator -Architecture and data flow -Stratified sampling explained -Data model to solve nested data - deep dive -Takeaways *Audience Planner Usecase 3 - Skew correction -Skew correction model -Query Access -Data model in Druid to accommodate output from ML models -Takeaways *Production setup, config and Tunings *Production Operation experience takeaways

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Imply

At Nielsen Identity, we leverage Druid to provide our customers with real-time analytics tools for various use-cases, including in-flight analytics, reporting and building target audiences. The common challenge of these use-cases is counting distinct elements in real-time at scale. We’ve been using Druid to solve these problems for the past 4 years, and gained a lot of experience with it. In this talk, we will share some of the best practices and tips we’ve gathered over the years, including: *Data modeling *Ingestion *Retention and deletion *Query optimization

Nielsen: Casting the Spell - Druid in Practice

Imply

Druid is a powerful real-time database, and part of that power is the level of control you get over cluster configuration, allowing you to get maximum performance for your specific data and query types. In this talk, Gian Merlino, one of the original authors of Druid and CTO and co-founder of Imply, will walk you through some advanced techniques that can provide a multiplier to your Druid performance. Afterwards, he’ll take your questions about performance, or anything else Druid-related.

Maximizing Apache Druid performance: Beyond the basics

Imply

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experi...

Imply

In this session, TrafficGuard’s Head of Data Science, Raigon Jolly, will discuss how TrafficGuard uses Druid and its partnership with Imply to: - Provide granular reporting to clients in near-real time - Monitor rules and concept drift - Staying ahead of the moving target that is ad fraud - Facilitate performance tuning and right-sizing infrastructure so our team can focus on innovation of our core product

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

Imply

As Twitch grew, both the amount of data we received and the number of employees interested in the data grew rapidly. In order to continue empowering decision making as we scaled, we turned to using Druid and Imply to provide self service analytics to both our technical and non technical staff allowing them to drill into high level metrics in lieu of reading generated reports. In this talk, learn how Twitch implemented a common analytics platform for the needs of many different teams supporting hundreds of users, thousands of queries, and ~5 billion events each day. This session will explain our Druid architecture in detail, including: -The end-to-end architecture deployed on Amazon that includes Kinesis, RDS, S3, Druid, Pivot and Tableau -How the data is brought together to deliver a unified view of live customer engagement and historical trends -Operational best practices we learnt scaling Druid -An example walk through using the platform

Self Service Analytics at Twitch

Imply

Talk abstract: Users are demanding access to large, multi-petabyte, multi-dimension, real-time datasets to answer business critical questions. Providing a self-service interface that meets the performance expectations of these users can be challenging. Enter Apache Druid: an open source analytics database powering real-time, ad hoc, lightning fast analytics. It is used for clickstream analytics, network telemetry, fraud detection, application monitoring and so much more by companies like Apple, Netflix, Twitter, and AirBnb. Druid can ingest millions of records per second and deliver sub-second response times on OLAP-style slice and dice queries. In this talk, we will start with an overview of Apache Druid followed by a look at several examples of how Druid is being used in the real-world. We'll finish up with Q&A and some virtual networking. Speaker Bio: Mike McLaughlin is a senior field engineer at Imply. He helps customers run and optimize Apache Druid in production. He has 20 years experience developing, architecting, and deploying software.

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

Imply

Benchmarking Apache Druid

Imply

Mais de Imply (10)

Pivot 2.0 - The next generation visualization tool for your streaming data

Druid in Spot Instances

Zeotap: Data Modeling in Druid for Non temporal and Nested Data

Nielsen: Casting the Spell - Druid in Practice

Maximizing Apache Druid performance: Beyond the basics

How Netflix Uses Druid in Real-time to Ensure a High Quality Streaming Experi...

How TrafficGuard uses Druid to Fight Ad Fraud and Bots

Self Service Analytics at Twitch

Apache Druid: Lightning Fast Analytics on Real-time and Historical Data (Atla...

Benchmarking Apache Druid

Último

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

MINDCTI Revenue Release Quarter One 2024

MIND CTI

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Building Data Applications with Apache Druid

1. Building data applications with Apache Druid October 2020 Gian Merlino, Cofounder & CTO, Imply 1

2. Who am I? 2 Gian Merlino Committer & PMC chair at Cofounder at (we’re hiring!)

3. 3 Building data apps

4. Druid powers data apps 4

5. Druid in the wild 5 100+ billion rows/day 1+ trillion rows, 1+ year retained 100s of servers sub-second to few seconds query latency mix of streaming and batch ingest

6. Online DB pattern 6 Data lakeData source Data mover (Apache Kafka, Apache Airflow, etc) Data source Data source Query engine Query engine Query engine Pure storage Data lake Data lake Direct to lake Online DB Stores an optimized copy Online app

7. 7 ● Scale-out, fault-tolerant architecture ● No downtime for software updates ● No downtime for data management ● Heavily optimized storage format ● Integrated storage format and query engine Druid as an online DB

8. Data apps ● Interactive query speeds ● Always online ● Fresh data from streams ● Quality of service ● Price/performance 8

9. Interactive query speeds 9 Secondary indexes Operate on compressed data Late materializationCompression INDEX [0,1,2](11100000) [3,4] (00011000) [5,6,7](0000111) DATA 0 0 0 1 1 2 2 2 DICT DC = 0 LA = 1 SF = 2

10. Interactive query speeds 10 artist (STRING) __time (LONG) 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 1293840000000 DATA DICT INDEX 0 0 0 1 1 2 2 2 Justin = 0 Ke$ha = 1 Miley = 2 [0,1,2](11100000) [3,4](00011000) [5,6,7](0000111) 25 42 17 170 112 67 53 94 DATA2 1 2 1 1 0 0 0 [0,2](10100000) [1,3,4](01011000) [5,6,7](00000111) DICT DC = 0 LA = 1 SF = 2 INDEX 1800 2912 1953 3194 5690 1100 8423 9080 city (STRING) count (LONG) price (LONG) Dictionary encoded (sorted) Bitmap index (stored compressed)

11. Always online 11 Coordinator Apache ZooKeeper Master server Historical Indexer Historical Indexer Data server Deep storage Broker Query server Streaming data Batch data

12. Fresh data from streams 12 Coordinator Apache ZooKeeper Master server Historical Indexer Historical Indexer Data server Deep storage Broker Query server Streaming data Batch data

13. Quality of service 13

14. Quality of service 14 Coordinator Apache ZooKeeper Master server Historical Indexer Historical Indexer Data server Deep storage Broker Query server Streaming data Batch data

15. Price/performance Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid. vs. leading open source SQL engines

16. Price/performance Data sourced from Imply benchmarks. vs. a leading cloud-based data warehouse

17. 17 Wrapping up

18. Stay in touch 18 @druidio Join the community (Mailing lists, Slack, meetups) https://druid.apache.org/community/ Follow the Druid project on Twitter!

19. Time for questions @gianmerlino 19 Thank you! Apache Druid is an independent project of The Apache Software Foundation. More information can be found at https://druid.apache.org. Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

Building Data Applications with Apache Druid

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Building Data Applications with Apache Druid

Semelhante a Building Data Applications with Apache Druid (20)

Mais de Imply

Mais de Imply (10)

Último

Último (20)

Building Data Applications with Apache Druid