Solr's missing plugin ecosystem
•
1 gostou•1,164 visualizações
My talk at Lucene/Solr Revolution 2017, Las Vegas The improved plugin system being proposed in this talk utilizes PF4J to add bundle packaging (zip/jar), plugin discovery (repositories), one-line install/upgrade and automatic version compatibility checks. Think of it as Homebrew or Apt-Get for Solr :) The hope is that this will encourage hundreds of new plugins being created and thus give Solr developers a sense of community and a new “stage” to perform on.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Solr's missing plugin ecosystem
Semelhante a Solr's missing plugin ecosystem (20)
Mais de Cominvent AS
Mais de Cominvent AS (11)
Último
Último (20)
Solr's missing plugin ecosystem
- 1. Solr’s missing plugin ecosystem Jan Høydahl Cominvent AS
- 2. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A All images used are licensed for commercial re-use
- 3. 3 • Jan Høydahl • Norway, 45 years • Cominvent AS • Lucene/Solr committer & PMC Telecom Fast ESP Solr Freelancer 1995 2000 2007 2017 17 years of Search
- 4. 4
- 6. «Technology ecosystems are product platforms defined by core components made by the platform owner and complemented by applications made by autonomous companies in the periphery» -ft.com/lexicon Ecosystem
- 8. Agenda • Intro • Solr plugins today • What is a plugin • Loading, distribution • User poll • The improved plugin system • Technical overview • Recap • Q & A
- 9. Java classes pre-defined by Solr for extension ZIP/JAR providing a certain feature Plugin - two levels
- 10. • Examples: • QueryParser • UpdateProcessor • SearchComponent • ResponseWriter • TokenFilter • … 10 What is a Solr plugin?
- 11. dataimport 11 Solr contribs • Optional features bundled as separate JARs • Released with Solr in the dist and contrib folders • May contain one or more Java-level plugins velocity
- 12. 12 How are plugins loaded? CLASSPATH SOLR_HOME/lib WEB-INF/lib <lib dir="${solr.install.dir}/contrib/velocity/lib" regex=".*.jar" /> <lib dir="${solr.install.dir}/dist/" regex="solr-velocity-d.*.jar" /> <requestHandler name="/browse" class="solr.SearchHandler" … /> Class Loader ResourceLoader
- 13. 13
- 15. 15
- 16. 16 Who are you?
- 17. 17 Pluginusage
- 18. 18 Rate the following statements Solr plugins are easy to discover Solr plugins are easy to install Solr plugins are easy to configure Solr plugins are easy to upgrade It is easy to match plugin to Solr version I find a plugin for all my needs It is easy to develop my own plugin It is easy to distribute my own plugins
- 20. 20 Questions to plugin developers We need a Solr Plugin SDK All plugins belong in Apache GIT Github is a better home for Solr plugins Commercial plugins in the official repo? The current plugin types are sufficient Other languages than Java
- 21. Agenda • Intro • Solr plugins today • The improved plugin system • POC • The end user perspective • Deverloper perspective • Solr product perspective • Technical overview • Recap • Q & A
- 23. 23 Proof of concept • Integrated PF4J in Solr • Design document • SOLR-10665 with patch and a download to try • CLI (bin/solr plugin) • Plugin repositories
- 24. 24 Design goals • User friendly and familiar • Establish a plugin ecosystem • Spark further modularisation • Extensible and customisable
- 25. 25 The end user perspective Discover plugin Install plugin Stay up to dateDownload Solr «Users download a very slim core without any contribs, huge analysers or esoteric query parsers. A few megs…»
- 26. 26 Downloading Solr • 65% size reduction 140 —> 50 Mb* • From 1min —> 20sec • Escape the «kitchen sink» • Room for more shaving! Core Contribs Test-fw Kuromoji HDFS *After converting contribs etc to plugins
- 27. 27 The end user experience Discover plugin Install plugin Stay up to dateDownload Solr «A new plugins tab in the Admin UI lists available official plugins as well as community plugins, installable with a single click.»
- 29. 29 PluginInstall
- 30. 30 The end user experience Stay up to dateDiscover plugin Install pluginDownload Solr «You will with a single command upgrade to the latest version that is compatible with your Solr version»
- 31. 31 CLI Usage - upgrade
- 32. 32 Developer perspective App Store advantage Simpler documentation Safe upgrades Free/ Premium Visibility Distribution bin/solr plugin install foo Version match SHA checksum PGP signature Paid plugins Try before buy
- 33. 33 product/project perspective Kitchen sink? Attract developers Modularity Licensing issues Smaller core Faster downloads Sense of community New ext. points Break out features JIRA patch -> plugin GPL dependencies User decides
- 34. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Choice of plugin framework • The new plugin bundle • Plugin repositories • Challenges • Recap • Q & A
- 35. 35 Choice of plugin framework Heavy weight, intrusive Jigsaw solves a different problem Not yet released Lightweight and customisable Provides version, dependencies & repository
- 36. 36 PF4J (Plugin Framework for Java) «PF4J is an open source, lightweight plugin framework for Java, with minimal dependencies and very extensible» www.pf4j.org
- 37. 37 The plugin bundle • Defined by PF4J • Simply «a JAR with a MANIFEST» • Or a ZIP with MANIFEST & dependencies • Lives in SOLR_HOME/plugins Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: decebal Build-Jdk: 1.6.0_17 Plugin-Class: com.example.MyPlugin Plugin-Dependencies: x, y, z Plugin-Id: my-plugin Plugin-Provider: John Doe Plugin-Version: 0.0.1 MANIFEST.mf <dependency org="org.pf4j" name="pf4j" rev="..." />
- 38. 38 Example contrib --> plugin ZIP analysis-extras-8.0.0.zip
- 39. 39
- 40. 40 Bootstrapping
- 41. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> before
- 42. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> PluginBundleClassLoader PluginBundleManager P1 P2 P3 1 2 findClass() (parent first) parent (parent last) after
- 43. 43 Plugin repositories • - • Any static website or folder • Each repo defines a plugins.json file <dependency org="org.pf4j" name="pf4j-update" rev="..." /> Solr version requirement
- 44. 44 Supported repositories dist.apache.org +mirrors solr-plugins/free solr-plugins/paid local solr/plugins • Official Apache • Community moderated • User added repos • Local file shares • Manually installed some.site org/repo /mnt/solr/repo
- 45. 45 The Apache dist repo dist.apache.org +mirrors • Custom PluginUpdateRepository • (1) closest mirror for exact version • (2) apache.org/dist • (3) archive.apache.org/dist • Validates SHA sum and PGP sig
- 46. 46 Needs more research • Multi version of dependency jars • Distributed install across cluster • Utilize runtimeLib / blob store? • Plugin upgrade requires restart • How to let plugins add UI menus
- 47. 47 Future possibilities • Collection declares plugin dependency <dependency>my-plugin</dependency> • Slim-JAR + • Auto load more components using SPI • Plugin SDK with samples pexels.com/photo/blackboard-business-chalkboard-concept-355988/CC0
- 48. 48 Next steps • Break up POC into smaller JIRAs • Commit core support to master • Split out a few contribs as plugins • Release 7.x with plugin support • Split out more features as plugins • Currency FieldType • Excel response writer • Kuromoji japanese analyzer • Test-framework? pixabay.com/p-768642/Publicdomain
- 49. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A
- 50. 50 Recap • Already modular • «Kitchen sink» • Hard to locate 3rd party plugins • integration introduces: • a well defined plugin bundle • tooling to work with plugins • an «App Store» approach to distribution • A larger ecosystem!
- 52. Thank You
- 53. Extra slides
- 54. 54 How pluggable are we - really?
- 55. 55 Do you use Solr plugins today?
- 57. 57 Top-3 priority features (contd) Not prioritised by anyone: • A "plugin store" to purchase commercial Solr plugins • Plugins must be based on OSGI • Plugins must be Jigsaw modules
- 58. 58 «All plugins that I used are custom,because [there] doesn’t existanything that fit the needs, or I didn'tfind the plugin at research time» «From a plugin-developer point of view the greatest pain point is keeping up with changes in Lucene, occasionally even interface changes between bug fix releases» «Worst experience is to always reinvent the wheel for plugins after API change in Lucene/Solr»
- 59. 59 Plugin lifecycle CREATED The manager has loaded the ZIP/JAR DISABLED A plugin that will not be started. May e.g. be do to Solr<—>Plugin version mismatch STARTED Active plugin state, contributes extensions STOPPED A plugin can be explicitly stopped, or implicitly during upgrading (unload old, load new)
- 60. 60 Implicit loading? • Lucene Analysis plugins loaded with SPI • Enables use of name, e.g synonym instead of class name org.apache.lucene.analysis.synonym.SynonymFilterFactory • Not exposed by Solr :-( • Some UpdateProcessors are pre-initialized (7.0) • Reference by well-defined name • Statically defined, not for 3rd party URPs • All others need to be explicitly configured