https://github.com/aborroy/alfresco-solr-monitoring https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage

1. Angel Borroy
Search Team
February 13, 2020
Monitoring
Alfresco SOLR
with Prometheus
and Grafana

2. 22
Monitoring Alfresco SOLR with Prometheus and Grafana
• Building Components
• Docker Compose demo
• Creating new Metrics and Dashboards
• Shard Support
• Deployment considerations
Contents

3. 33
Building Components

4. 4
Map of Components
Alfresco
Search Service
1.0 to 1.4+
SOLR 6.6.x
Prometheus
Exporter
SOLR 7.3.0+
SOLR Metrics
Prometheus
Server
2.x
Grafana
UI
6.x
8083
9854 9090
3000
Exporter
Config
XML
JOB
YAML
Dashboard
JSON

5. 5
SOLR Metrics
Classes of meters
• counters - simply count events. Single long value.
• meters - additionally compute rates of events.
Provide a count and mean, 1-, 5-, and 15-minute
rates.
• histograms - calculate approximate distribution of
events according to their values. Provide the
following approximate statistics, with a similar
exponential decay as above: mean (arithmetic
average), median, maximum, minimum, standard
deviation, and 75th, 95th, 98th, 99th and 999th
percentiles.
• timers - measure the number and duration of events.
Count and histogram of timings.
• gauges - offer instantaneous reading of a current
value.
http://localhost:8083/solr/admin/metrics
Metric Registries

6. 6
Prometheus Exporter
https://github.com/aborroy/alfresco-solr-monitoring/blob/master/solr-exporter/solr6-exporter-config.xml
<str>
.metrics["solr.jvm"] | to_entries | .[] | select(.key | startswith("gc.")) | select(.key | endswith(".time")) as $object |
$object.key | split(".")[1] as $item |
($object.value.value / 1000) as $value |
{
name : "solr_metrics_jvm_gc_seconds_total",
type : "COUNTER",
help : "Help",
label_names : ["item"],
label_values : [$item],
value : $value
}
</str>
http://localhost:9854/metrics
Exporter
Config
XML

7. 7
Prometheus Server
https://github.com/aborroy/alfresco-solr-monitoring/blob/master/prometheus/prometheus.yml
scrape_configs:
- job_name: 'solr'
static_configs:
- targets: ['solr-exporter:9854']
http://localhost:9090/targets
JOB
YAML

8. 8
Grafana UI
https://github.com/aborroy/alfresco-solr-monitoring/blob/master/grafana/grafana-solr6-dashboard.json
"targets": [
{
"expr": "increase(solr_metrics_jvm_gc_total{base_url=~"$base_url"}[1m])",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{base_url}} {{item}}",
"refId": "A"
}
]
Dashboard
JSON
http://localhost::3000/d/GqfWBzbWz/solr-dashboard

9. 99
Docker Compose Demo

10. 1010
Demo settings
• Single SOLR server
• Every service running in the
same Docker Compose
• Default templates for exporter
and dashboard

11. 11
Docker Compose
solr-exporter:
image: solr:7.3.0
depends_on:
- solr6
ports:
- "9854:9854"
entrypoint:
- "/opt/solr/contrib/prometheus-exporter/bin/solr-exporter"
- "-p"
- "9854"
- "-b"
- "http://solr6:8983/solr"
- "-f"
- "/opt/solr/contrib/prometheus-exporter/conf/solr-exporter-config.xml"
- "-n"
- "8"
volumes:
- ./solr-exporter/solr6-exporter-config.xml:/opt/solr/contrib/prometheus-exporter/conf/solr-exporter-config.xml

12. 12
Docker Compose
prometheus-server:
image: prom/prometheus
ports:
- 9090:9090
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
grafana-ui:
image: grafana/grafana
ports:
- 3000:3000
environment:
- GF_SECURITY_ADMIN_PASSWORD=secret
links:
- prometheus-server:prometheus
volumes:
- ./grafana/datasources/prometheus.yml:/etc/grafana/provisioning/datasources/prometheus.yml
- ./grafana/dashboards/solr.yml:/etc/grafana/provisioning/dashboards/solr.yml
- ./grafana/grafana-solr6-dashboard.json:/var/lib/grafana/dashboards/grafana-solr6-dashboard.json

13. 1313
Demo Time

14. 1414
Creating new Metrics and Dashboards

15. 1515
Creating new metrics
http://127.0.0.1:8083/solr/admin/metrics
http://127.0.0.1:9854/metrics
http://127.0.0.1:3000
Prometheus
https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage

16. 1616
Exposing new metrics
Prometheus Exporter
<str>
.metrics["solr.jvm"] | to_entries | .[] |
select(.key | startswith("memory.heap.usage")) as $object |
$object.key | split(".")[1] as $item |
$object.value.value as $value |
{
name : "solr_metrics_memory_heap_usage",
type : "GAUGE",
help : "See following URL: https://lucene.apache.org/...",
label_names : ["item"],
label_values : [$item],
value : $value
}
</str>

17. 1717
Adding new panels
Grafana
{
"datasource": "Prometheus",
"id": 61,
"options": {
"fieldOptions": {
"calcs": [
"last"
],
"defaults": {
"max": 1.0,
"min": 0.0,
"thresholds": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 0.75
}
],
"unit": "percentunit"
},
"override": {},
"values": false
},
"orientation": "auto",
},
"targets": [
{
"expr":
"solr_metrics_memory_heap_usage{base_url=~"$
base_url"}",
"format": "time_series",
"intervalFactor": 2,
"legendFormat": "{{base_url}} {{item}}",
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "Memory Heap Usage",
"type": "gauge"
},

18. 1818
Final Considerations

19. 1919
Shard is not supported
• Prometheus Exporter only
supports ZooKeeper
(SolrCloud)
• Alternatively several
Prometheus Exporter
endpoints can be configured,
one for each SOLR ShardX

20. 2020
Alfresco
Deployment
SOLR
Server

21. Thank you!
Monitoring
Alfresco SOLR
with Prometheus
and Grafana

22. 22
References
• Monitoring Alfresco SOLR with Prometheus and Grafana blog post
https://hub.alfresco.com/t5/alfresco-content-services-blog/monitoring-alfresco-solr-with-prometheus-and-grafana/ba-
p/294157
• Docker Compose sample template
https://github.com/aborroy/alfresco-solr-monitoring
https://github.com/aborroy/alfresco-solr-monitoring/tree/add-metric-heap-usage
• SOLR Metrics Reporting
https://lucene.apache.org/solr/guide/6_6/metrics-reporting.html
• SOLR Prometheus Exporter
https://github.com/apache/lucene-solr/tree/master/solr/contrib/prometheus-exporter
• Grafana Dashboard JSON Reference
https://grafana.com/docs/grafana/latest/reference/dashboard/