Queue Based Solr Indexing with Collection Management: Presented by Devansh Dhutia, Gannett Co.

•

3 gostaram•2,163 visualizações

Lucidworks

Lucene/Solr Revolution 2015

Tecnologia

O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X

Queue Based Indexing & Collection Management
Devansh Dhutia
Platform Architect

3
01
•  National & Local newspaper/media company
•  92+ Markets in 33 states

5
01
Agenda
•  Solr @ Gannett
•  Current State
•  Collection Management
•  Queuing Solution
•  Future Work
•  Questions

6
02
@
Site Search
CMS Search
Analytics
Personalization
40+ Applications
20M+
Integral pillar of Gannett’s Digital Platform
total
documents
800,000+ per
month
Growing rapidly
100,000+ requests
per minute
Highly Available
~100ms average
response time
Extremely Fast
8 nodes
256 gb memory
per availability zone

8
01
Current State
•  Synchronous Operations
•  Near Realtime
•  Time Consuming schema changes
•  Visible outage impact

9
01
Collection Management
•  Create Collection
•  Deploy Batch Indexer
•  Index new Collection
•  Update Alias to new Collection
•  Run catch up
•  Deploy Search/Index Apps

13
01
Outage Problems
•  Spinning Wheel
•  Duplicate content
•  Unable to ﬁnd new content
•  Frustrated editors
•  Ux & other presentation layers

14
01
Enter Queues
•  Asynchronous Write Operations
•  Near Realtime
•  Faster schema changes
•  Auto scale indexing workers
•  Low authoring outage impact
•  Eventually consistent

16
01
RabbitMQ
•  Clustered & Highly Available
•  FIFO
•  pub/sub model
•  Consistent Hash / Multiple Queues

18
01
Components
•  Realtime Queue
•  Batch Queue
•  Prep Queue
•  Deadletter Queue
•  Indexing Service
•  Prep mode
•  Batch Push Service

19
01
Future Work
•  Continuous Delivery of schema
•  Build payload in one zone only
•  Automated Deadletter handling
•  Earlier notiﬁcation of potential failure

20
01
Thank you
Interested in joining our team at Gannett?
http://www.gannett.com/careers

Mais conteúdo relacionado

Mais procurados

Searching for Better Code: Presented by Grant Ingersoll, Lucidworks

Lucidworks

Presto meetup 2015-03-19 @Facebook

Treasure Data, Inc.

TweetMogaz - The Arabic Tweets Platform: Presented by Ahmed Adel, BADR

Lucidworks

Data can be viewed as the exhaust of online activity. With the rise of cloud-based data platforms, barriers to data storage and transfer have crumbled. The demand for creative applications and learning from those datasets has accelerated. Rapid acceleration can quickly accrue disorder, and disorderly data design can turn the deepest data lake into an impenetrable swamp. In this talk, I will discuss the evolution of the data science workflow at Expedia with a special emphasis on Learning to Rank problems. From the heroic early days of ad-hoc Spark exploration to our first production sort model on the cloud, we will explore the process of industrializing the workflow. Layered over our story, I will share some best practices and suggestions on how to keep your data productive, or even pull your organization out of the data swamp.

Rental Cars and Industrialized Learning to Rank with Sean Downes

Databricks

Bullet is an open sourced, lightweight, pluggable querying system for streaming data without a persistence layer implemented on top of Storm. It allows you to filter, project, and aggregate on data in transit. It includes a UI and WS. Instead of running queries on a finite set of data that arrived and was persisted or running a static query defined at the startup of the stream, our queries can be executed against an arbitrary set of data arriving after the query is submitted. In other words, it is a look-forward system. Bullet is a multi-tenant system that scales independently of the data consumed and the number of simultaneous queries. Bullet is pluggable into any streaming data source. It can be configured to read from systems such as Storm, Kafka, Spark, Flume, etc. Bullet leverages Sketches to perform its aggregate operations such as distinct, count distinct, sum, count, min, max, and average. An instance of Bullet is currently running at Yahoo against its user engagement data pipeline. We’ll highlight how it is powering internal use-cases such as web page and native app instrumentation validation. Finally, we’ll show a demo of Bullet and go over query performance numbers.

Bullet: A Real Time Data Query Engine

DataWorks Summit

DataEngConf SF16 - High cardinality time series search

Hakka Labs

Meetup070416 Presentations

Ana Rebelo

Hoodie: How (And Why) We built an analytical datastore on Spark

Vinoth Chandar

Building a Vibrant Search Ecosystem @ Bloomberg: Presented by Steven Bower & ...

Lucidworks

Ubiquitous Solr - A Database's Not-So-Evil Twin: Presented by Ayon Sinha, Wal...

Lucidworks

Presto@Netflix Presto Meetup 03-19-15

Zhenxiao Luo

Leveraging the Power of Solr with Spark: Presented by Johannes Weigend, QAware

Lucidworks

At Netflix, the big data platform is the foundation for analytics that drive all product decisions that directly impact our customer experience. As for scale, it is one of the top three largest services running at Netflix, in terms of compute power and data size. In this talk, we will take the audience through a journey to understand how we scale the platform to handle the increasing amount of data (over 500 billion events generated daily), the increasing demand of analytics (which translates to compute power), and the increasing number of users dependent on our platform to make business decisions.

The evolution of the big data platform @ Netflix (OSCON 2015)

Eva Tse

Never Stop Exploring - Pushing the Limits of Solr: Presented by Anirudha Jadh...

Lucidworks

Технологии с открытым исходным кодом, такие как Microsoft Orleans и ElasticSearch, - ключевые элементы архитектуры YouScan. О том, как они помогают справляться с постоянно растущими объемами данных из социальных сетей, об эволюции архитектуры YouScan, я расскажу в данном докладе.

Евгений Бобров "Powered by OSS. Масштабируемая потоковая обработка и анализ б...

Fwdays

Apache InLong is a one-stop data streaming platform, it chooses Apache Pulsar to cache data for forwarding sort. Apache Pulsar has great reliability and stability, which helps the InLong to be more confident for users. This session will share Tencent Big Data Team's journal of adopting Pulsar in their core data engine to process tens of billions of data integration. Besides, some problems they encountered during the process and the improvements on Pulsar they have made will also be shared as an example for future Pulsar users.

How Tencent Applies Apache Pulsar to Apache InLong - Pulsar Summit Asia 2021

StreamNative

Streaming Aggregation in Solr - New Horizons for Search: Presented by Erick E...

Lucidworks

We will start from understanding how Real-Time Analytics can be implemented on Enterprise Level Infrastructure and will go to details and discover how different cases of business intelligence be used in real-time on streaming data. We will cover different Stream Data Processing Architectures and discus their benefits and disadvantages. I'll show with live demos how to build Fast Data Platform in Azure Cloud using open source projects: Apache Kafka, Apache Cassandra, Mesos. Also I'll show examples and code from real projects.

Виталий Бондаренко "Fast Data Platform for Real-Time Analytics. Architecture ...

Fwdays

Webinar 2017. Supercharge your analytics with ClickHouse. Alexander Zaitsev

Altinity Ltd

Kibana + timelion: time series with the elastic stack

Sylvain Wallez

Mais procurados (20)