SlideShare uma empresa Scribd logo

Scalling Solr in the Cloud - OSCON Data 2011

•
•
OSCON Byrum
OSCON Byrum

The document discusses how Nutshell, a CRM application, scaled their use of Solr for searching and deduplication. They implemented several techniques including using a separate Solr core per customer, intelligent indexing strategies, and partitioning customers across multiple Solr servers. These changes helped improve performance and allow Nutshell to scale Solr to support thousands of customers.

Tecnologia
1 de 61
Baixar para ler offline
Scaling Solr in the Cloud By @ablyler & @LindsaySnider FROM @NutshellCRM www.nutshell.com
Agenda About the Speakers About Nutshell Nutshell and Solr Solr Resource Usage Scaling Methods Questions and Answers
About the Speakers BitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com
About Nutshell Web and Mobile CRM application Heavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jQuery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios
Nutshell and Solr Heavy use of Solr for searching, table views, and de-duplication Used for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users Used for de-duplication: Accounts, Contacts
Reads vs Writes 7000 5250 3500 1750 0 Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM Writes Reads Time Average Read Query Time: 2.11ms
Demo www.nutshell.com
Computer Resources CPU Disk I/O Disk RAM Storage
CPU Disk I/O Disk RAM Storage
SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome email
Dark Ages Age of Enlightenment Modern Era Today Separate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch
Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer
Dark Ages Age of Enlightenment Modern Era Today 50 25 0 Auto provisioning Separate Jetty app for each customer
Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API
Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library
Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM
Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
Dark Ages Age of Enlightenment Modern Era Today 300 150 0 Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
Dark Ages Age of Enlightenment Modern Era Today Shared Schema Across Cores Decreases initialization time for each core Decreases memory usage
Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display
Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn’t build indexes
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master)
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
Dark Ages Age of Enlightenment Modern Era Today 1000 500 0 Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
Dark Ages Age of Enlightenment Modern Era Today Intelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login
Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id
Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers
Dark Ages Age of Enlightenment Modern Era Today 8000 4000 0 Intelligent core spin down / up Partitioning of customers to separate Solr servers
SOLR Server 8000 User
SOLR SOLR SOLR Server SOLR Server SOLR Server SOLR 8000 Server UserSOLR 8000 Server User 8000 Server 8000User 8000User Server 8000User 8000User User
SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User 65 SOLR Server 8000 User SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User 96 SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
Scaling Solr in the Cloud auto provisioning 50 Users separate Jetty app for each customer separate Solr core per customer fall back to MySQL for table data 300 Users migrated from Sun JVM to IcedTea JVM shared schema across cores solr index field selection 600 Users splitting reader / writer roles intelligent core spin down / up 7000 Users parititioning of customer to separate Solr servers By @ablyler & @LindsaySnider FROM @NutshellCRM

Recomendados

AAA game in XNA ? Miner wars 2081
AAA game in XNA ? Miner wars 2081AAA game in XNA ? Miner wars 2081
AAA game in XNA ? Miner wars 2081Marek Rosa
 
Nosql storage in windows azure
Nosql storage in windows azureNosql storage in windows azure
Nosql storage in windows azureGal Kogman
 
Eucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus Systems, Inc.
 
Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram Chinta
 
Eucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus Systems, Inc.
 
Scaling search with Solr Cloud
Scaling search with Solr CloudScaling search with Solr Cloud
Scaling search with Solr CloudCominvent AS
 
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11Harish Ganesan
 
Administering and Monitoring SolrCloud Clusters
Administering and Monitoring SolrCloud ClustersAdministering and Monitoring SolrCloud Clusters
Administering and Monitoring SolrCloud ClustersSematext Group, Inc.
 

Recomendados

AAA game in XNA ? Miner wars 2081
AAA game in XNA ? Miner wars 2081AAA game in XNA ? Miner wars 2081
AAA game in XNA ? Miner wars 2081Marek Rosa
 
Nosql storage in windows azure
Nosql storage in windows azureNosql storage in windows azure
Nosql storage in windows azureGal Kogman
 
Eucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus Systems, Inc.
 
Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram Chinta
 
Eucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus 3 Product Overview
Eucalyptus 3 Product OverviewEucalyptus Systems, Inc.
 
Scaling search with Solr Cloud
Scaling search with Solr CloudScaling search with Solr Cloud
Scaling search with Solr CloudCominvent AS
 
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11
Amazon cloud search_vs_apache_solr_vs_elasticsearch_comparison_report_v11Harish Ganesan
 
Administering and Monitoring SolrCloud Clusters
Administering and Monitoring SolrCloud ClustersAdministering and Monitoring SolrCloud Clusters
Administering and Monitoring SolrCloud ClustersSematext Group, Inc.
 
Big Search with Big Data Principles
Big Search with Big Data PrinciplesBig Search with Big Data Principles
Big Search with Big Data PrinciplesOpenSource Connections
 
ApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataOpenSource Connections
 
RADOS for Eucalyptus
RADOS for EucalyptusRADOS for Eucalyptus
RADOS for EucalyptusTakuya ASADA
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systexJames Chen
 
CloudStack technical overview
CloudStack technical overviewCloudStack technical overview
CloudStack technical overviewCloudStack - Open Source Cloud Computing Project
 
No Sql
No SqlNo Sql
No SqlMichael Marth
 
February 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerFebruary 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerYahoo Developer Network
 
1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day 1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day Kimihiko Kitase
 
UKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sUKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sFromDual GmbH
 
MySQL Consegi
MySQL ConsegiMySQL Consegi
MySQL Consegihenriquesidney
 
Microsoft Openness Mongo DB
Microsoft Openness Mongo DBMicrosoft Openness Mongo DB
Microsoft Openness Mongo DBHeriyadi Janwar
 
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBeyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBen Stopford
 
Stathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy Touloumis
 
Geek out 2014-lagergren-final
Geek out 2014-lagergren-finalGeek out 2014-lagergren-final
Geek out 2014-lagergren-finalMarcus Lagergren
 
Solr @ eBay Kleinanzeigen
Solr @ eBay KleinanzeigenSolr @ eBay Kleinanzeigen
Solr @ eBay KleinanzeigenLucidworks (Archived)
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons
 
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,..."Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...lisapaglia
 
Yokozuna
YokozunaYokozuna
YokozunaPDX Web & Design
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconAdrian Cockcroft
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformAdrian Cockcroft
 
OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON Byrum
 
Protecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseProtecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseOSCON Byrum
 

Mais conteúdo relacionado

Semelhante a Scalling Solr in the Cloud - OSCON Data 2011

ApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataOpenSource Connections
 
RADOS for Eucalyptus
RADOS for EucalyptusRADOS for Eucalyptus
RADOS for EucalyptusTakuya ASADA
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systexJames Chen
 
February 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerFebruary 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerYahoo Developer Network
 
1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day 1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day Kimihiko Kitase
 
UKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sUKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sFromDual GmbH
 
Microsoft Openness Mongo DB
Microsoft Openness Mongo DBMicrosoft Openness Mongo DB
Microsoft Openness Mongo DBHeriyadi Janwar
 
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBeyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBen Stopford
 
Stathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy Touloumis
 
Geek out 2014-lagergren-final
Geek out 2014-lagergren-finalGeek out 2014-lagergren-final
Geek out 2014-lagergren-finalMarcus Lagergren
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons
 
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,..."Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...lisapaglia
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconAdrian Cockcroft
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformAdrian Cockcroft
 

Semelhante a Scalling Solr in the Cloud - OSCON Data 2011 (20)

Big Search with Big Data Principles
Big Search with Big Data PrinciplesBig Search with Big Data Principles
Big Search with Big Data Principles
 
ApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big DataApacheCon Europe 2012 -Big Search 4 Big Data
ApacheCon Europe 2012 -Big Search 4 Big Data
 
RADOS for Eucalyptus
RADOS for EucalyptusRADOS for Eucalyptus
RADOS for Eucalyptus
 
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
[Hic2011] using hadoop lucene-solr-for-large-scale-search by systex
 
CloudStack technical overview
CloudStack technical overviewCloudStack technical overview
CloudStack technical overview
 
No Sql
No SqlNo Sql
No Sql
 
February 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with DockerFebruary 2016 HUG: Running Spark Clusters in Containers with Docker
February 2016 HUG: Running Spark Clusters in Containers with Docker
 
1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day 1 Introduction at CloudStack Developer Day
1 Introduction at CloudStack Developer Day
 
UKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA'sUKOUG 2011: MySQL Architectures for Oracle DBA's
UKOUG 2011: MySQL Architectures for Oracle DBA's
 
MySQL Consegi
MySQL ConsegiMySQL Consegi
MySQL Consegi
 
Microsoft Openness Mongo DB
Microsoft Openness Mongo DBMicrosoft Openness Mongo DB
Microsoft Openness Mongo DB
 
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBeyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
 
Stathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IACStathy DevOps in MSP / MKE on IAC
Stathy DevOps in MSP / MKE on IAC
 
Geek out 2014-lagergren-final
Geek out 2014-lagergren-finalGeek out 2014-lagergren-final
Geek out 2014-lagergren-final
 
Solr @ eBay Kleinanzeigen
Solr @ eBay KleinanzeigenSolr @ eBay Kleinanzeigen
Solr @ eBay Kleinanzeigen
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
 
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,..."Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
 
Yokozuna
YokozunaYokozuna
Yokozuna
 
Netflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at GlueconNetflix Architecture Tutorial at Gluecon
Netflix Architecture Tutorial at Gluecon
 
SV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source PlatformSV Forum Platform Architecture SIG - Netflix Open Source Platform
SV Forum Platform Architecture SIG - Netflix Open Source Platform
 

Mais de OSCON Byrum

OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON Byrum
 
Protecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseProtecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseOSCON Byrum
 
Using Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open DataUsing Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open DataOSCON Byrum
 
Finite State Machines - Why the fear?
Finite State Machines - Why the fear?Finite State Machines - Why the fear?
Finite State Machines - Why the fear?OSCON Byrum
 
Open Source Automotive Development
Open Source Automotive DevelopmentOpen Source Automotive Development
Open Source Automotive DevelopmentOSCON Byrum
 
How we built our community using Github - Uri Cohen
How we built our community using Github - Uri CohenHow we built our community using Github - Uri Cohen
How we built our community using Github - Uri CohenOSCON Byrum
 
The Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in PythonThe Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in PythonOSCON Byrum
 
Distributed Coordination with Python
Distributed Coordination with PythonDistributed Coordination with Python
Distributed Coordination with PythonOSCON Byrum
 
An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)OSCON Byrum
 
Oscon 2013 Jesse Anderson
Oscon 2013 Jesse AndersonOscon 2013 Jesse Anderson
Oscon 2013 Jesse AndersonOSCON Byrum
 
US Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David MertzUS Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David MertzOSCON Byrum
 
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...OSCON Byrum
 
Big Data for each one of us
Big Data for each one of usBig Data for each one of us
Big Data for each one of usOSCON Byrum
 
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking OSCON Byrum
 
Declarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScriptDeclarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScriptOSCON Byrum
 
Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...OSCON Byrum
 
A Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed ApplicationsA Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed ApplicationsOSCON Byrum
 
Life After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data CloudLife After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data CloudOSCON Byrum
 
Faster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypesFaster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypesOSCON Byrum
 
Comparing open source private cloud platforms
Comparing open source private cloud platformsComparing open source private cloud platforms
Comparing open source private cloud platformsOSCON Byrum
 

Mais de OSCON Byrum (20)

OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom FifieldOSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
OSCON 2013 - Planning an OpenStack Cloud - Tom Fifield
 
Protecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent LicenseProtecting Open Innovation with the Defensive Patent License
Protecting Open Innovation with the Defensive Patent License
 
Using Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open DataUsing Cascalog to build an app with City of Palo Alto Open Data
Using Cascalog to build an app with City of Palo Alto Open Data
 
Finite State Machines - Why the fear?
Finite State Machines - Why the fear?Finite State Machines - Why the fear?
Finite State Machines - Why the fear?
 
Open Source Automotive Development
Open Source Automotive DevelopmentOpen Source Automotive Development
Open Source Automotive Development
 
How we built our community using Github - Uri Cohen
How we built our community using Github - Uri CohenHow we built our community using Github - Uri Cohen
How we built our community using Github - Uri Cohen
 
The Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in PythonThe Vanishing Pattern: from iterators to generators in Python
The Vanishing Pattern: from iterators to generators in Python
 
Distributed Coordination with Python
Distributed Coordination with PythonDistributed Coordination with Python
Distributed Coordination with Python
 
An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)An overview of open source in East Asia (China, Japan, Korea)
An overview of open source in East Asia (China, Japan, Korea)
 
Oscon 2013 Jesse Anderson
Oscon 2013 Jesse AndersonOscon 2013 Jesse Anderson
Oscon 2013 Jesse Anderson
 
US Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David MertzUS Patriot Act OSCON2012 David Mertz
US Patriot Act OSCON2012 David Mertz
 
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
OSCON 2012 US Patriot Act Implications for Cloud Computing - Diane Mueller, A...
 
Big Data for each one of us
Big Data for each one of usBig Data for each one of us
Big Data for each one of us
 
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
BodyTrack: Open Source Tools for Health Empowerment through Self-Tracking
 
Declarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScriptDeclarative web data visualization using ClojureScript
Declarative web data visualization using ClojureScript
 
Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...Using and Building Open Source in Google Corporate Engineering - Justin McWil...
Using and Building Open Source in Google Corporate Engineering - Justin McWil...
 
A Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed ApplicationsA Look at the Network: Searching for Truth in Distributed Applications
A Look at the Network: Searching for Truth in Distributed Applications
 
Life After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data CloudLife After Sharding: Monitoring and Management of a Complex Data Cloud
Life After Sharding: Monitoring and Management of a Complex Data Cloud
 
Faster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypesFaster! Faster! Accelerate your business with blazing prototypes
Faster! Faster! Accelerate your business with blazing prototypes
 
Comparing open source private cloud platforms
Comparing open source private cloud platformsComparing open source private cloud platforms
Comparing open source private cloud platforms
 

Último

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slidespraypatel2
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAndikSusilo4
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?XfilesPro
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad
 

Último (20)

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Slack Application Development 101 Slides
Slack Application Development 101 SlidesSlack Application Development 101 Slides
Slack Application Development 101 Slides
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptx
 
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & ApplicationAzure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
 
Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)Injustice - Developers Among Us (SciFiDevCon 2024)
Injustice - Developers Among Us (SciFiDevCon 2024)
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?How to Remove Document Management Hurdles with X-Docs?
How to Remove Document Management Hurdles with X-Docs?
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 

Scalling Solr in the Cloud - OSCON Data 2011

  • 1. Scaling Solr in the Cloud By @ablyler & @LindsaySnider FROM @NutshellCRM www.nutshell.com
  • 2. Agenda About the Speakers About Nutshell Nutshell and Solr Solr Resource Usage Scaling Methods Questions and Answers
  • 3. About the Speakers BitLeap / Barracuda Networks Developers on the Backup Appliance / Cloud Scaled databases and storage systems Used SugarCRM & Salesforce.com
  • 4. About Nutshell Web and Mobile CRM application Heavy use of OpenSource technologies: Gentoo Nginx / PHP / ZendFramework / jQuery MySQL / Solr / Gearman Jenkins / Redmine / Cacti / Nagios
  • 5. Nutshell and Solr Heavy use of Solr for searching, table views, and de-duplication Used for searching / display: Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users Used for de-duplication: Accounts, Contacts
  • 6. Reads vs Writes 7000 5250 3500 1750 0 Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM Writes Reads Time Average Read Query Time: 2.11ms
  • 8. Computer Resources CPU Disk I/O Disk RAM Storage
  • 9. CPU Disk I/O Disk RAM Storage
  • 10. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 11. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 12. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 13. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 14. SOLR Resources Heavy CPU Disk I/O Disk RAM Storage Light
  • 15. Dark Ages Age of Enlightenment Modern Era Today Auto Provisioning Setup new Solr Jetty app Create MySQL database Populate MySQL / Solr with demo data Send welcome email
  • 16. Dark Ages Age of Enlightenment Modern Era Today Separate Jetty per Customer Uses a ton of memory Separate schema / Solr for each customer Ran into upper limit morning before launch
  • 17. Dark Ages Age of Enlightenment Modern Era Today Auto provisioning Separate Jetty app for each customer
  • 18. Dark Ages Age of Enlightenment Modern Era Today 50 25 0 Auto provisioning Separate Jetty app for each customer
  • 19. Dark Ages Age of Enlightenment Modern Era Today Solr Core per Customer Allows for management of Solr on a per customer basis: creating / stopping Contained within a single Jetty app Shared schema between all cores Easily managed via simple HTTP API
  • 20. Dark Ages Age of Enlightenment Modern Era Today Fallback to MySQL Landing page of application Allows for graceful handling when Solr is down Abstracted within the application library
  • 21. Dark Ages Age of Enlightenment Modern Era Today Sun JVM to IcedTea JVM IcedTea JVM uses less memory than Sun JVM
  • 22. Dark Ages Age of Enlightenment Modern Era Today Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
  • 23. Dark Ages Age of Enlightenment Modern Era Today 300 150 0 Separate Solr core per customer Fallback to MySQL for table data Migrated from Sun JVM to IcedTea JVM
  • 24. Dark Ages Age of Enlightenment Modern Era Today Shared Schema Across Cores Decreases initialization time for each core Decreases memory usage
  • 25. Dark Ages Age of Enlightenment Modern Era Today Solr Index Field Selection More indexed fields = more used memory Only index fields that are searched Store other non-indexed fields for display
  • 26. Dark Ages Age of Enlightenment Modern Era Today Splitting of Reader / Writer Index building is CPU / disk intensive Writer = Solr with caching disabled Reader = Solr slave, that doesn’t build indexes
  • 27. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master)
  • 28. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 29. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 30. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Document indexers Gearman workers
  • 31. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 32. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 33. Dark Ages Age of Enlightenment Modern Era Today Web Solr MySQL Slave (master- master) Solr Master Range indexer Range indexer Range indexer Range indexer Range indexers Reindex Manager
  • 34. Dark Ages Age of Enlightenment Modern Era Today Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
  • 35. Dark Ages Age of Enlightenment Modern Era Today 1000 500 0 Shared schema across cores Solr index field selection Splitting of reader / writer roles Intelligent indexing / reindexing
  • 36. Dark Ages Age of Enlightenment Modern Era Today Intelligent Core Management Least recently used Solr cores are spun down Solr cores started on login Reindexing is a database flag, and happens on the next login
  • 37. Dark Ages Age of Enlightenment Modern Era Today Partitioning Pairs of readers and writers Partitioned based on account id Left pad the account id w/ zeros to length of two Reverse the account id Take last two digits of the account id
  • 38. Dark Ages Age of Enlightenment Modern Era Today Intelligent core spin down / up Partitioning of customers to separate Solr servers
  • 39. Dark Ages Age of Enlightenment Modern Era Today 8000 4000 0 Intelligent core spin down / up Partitioning of customers to separate Solr servers
  • 40.
  • 42. SOLR SOLR SOLR Server SOLR Server SOLR Server SOLR 8000 Server UserSOLR 8000 Server User 8000 Server 8000User 8000User Server 8000User 8000User User
  • 43. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 44. SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 45. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 46. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 47. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 48. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User SOLR Server 8000 User SOLR Server 8000 User
  • 49. SOLR Server 8000 User SOLR Server 8000 User 123456 SOLR Server 8000 User 1 2 3 4 5 6 SOLR Server 8000 User SOLR 1 2 3 4 5 6 Server 8000 User 65 SOLR Server 8000 User SOLR Server 8000 User
  • 50. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 51. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 52. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 123456 03 SOLR 04 Server 8000 User 1 2 3 4 5 6 65 SOLR Server 66 8000 User SOLR 1 2 3 4 5 6 96 Server 8000 User 97 65 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 53. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 54. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 55. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User 97 SOLR 98 Server 8000 User 99 SOLR Server 8000 User
  • 56. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 57. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 58. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 59. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User SOLR 96 Server 8000 User SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 60. SOLR Server 8000 User 01 SOLR 02 Server 8000 User 03 SOLR 04 Server 8000 User 65 SOLR Server 66 8000 User SOLR Server 8000 User 96 SOLR 97 Server SOLR 8000 User 98 Server 8000 User 99 SOLR Server 8000 User
  • 61. Scaling Solr in the Cloud auto provisioning 50 Users separate Jetty app for each customer separate Solr core per customer fall back to MySQL for table data 300 Users migrated from Sun JVM to IcedTea JVM shared schema across cores solr index field selection 600 Users splitting reader / writer roles intelligent core spin down / up 7000 Users parititioning of customer to separate Solr servers By @ablyler & @LindsaySnider FROM @NutshellCRM