Monitoring Alfresco SOLR with Prometheus and Grafana

[angelborroy]

Apache SOLR provides a Metrics Reporting REST API from 6.4.0 release. This API can be accessed using the following handler:

http://localhost:8983/solr/admin/metrics

<response>
  <lst name="metrics">
    <lst name="solr.jetty">
      <lst name="org.eclipse.jetty.server.handler.DefaultHandler.1xx-responses">
        <long name="count">0</long>
        <double name="meanRate">0.0</double>
        <double name="1minRate">0.0</double>
        <double name="5minRate">0.0</double>
        <double name="15minRate">0.0</double>
      </lst>
    </lst>
  </lst>
</response>

SOLR provides different types of metrics (counter, meter, histogram, timer and gauge) that are exposed in groups of categories (jvm, jetty, node and core). Additionally, some out of the box reporters are available to export this metrics to analisys tools like SLF4J, Graphite, Ganglia and Prometheus.

Since Alfresco Search Services 1.4 is built on top of Apache Solr 6.6.5, this REST API can be used to monitor service performance.

SOLR Prometheus Exporter

Prometheus exporter is available from SOLR 7.3.0:

https://github.com/apache/lucene-solr/tree/releases/lucene-solr/7.3.0/solr/contrib/prometheus-export...

This application is a REST API that exposes SOLR Metrics in a Prometheus compliant format.

http://localhost:9854/metrics

# HELP solr_ping See following URL: https://lucene.apache.org/solr/guide/ping.html
# TYPE solr_ping gauge
solr_ping{base_url="http://solr6:8983/solr",core="alfresco",} 1.0
solr_ping{base_url="http://solr6:8983/solr",core="archive",} 1.0
# HELP solr_metrics_jetty_response_total See following URL: https://lucene.apache.org/solr/guide/metrics-reporting.html
# TYPE solr_metrics_jetty_response_total counter
solr_metrics_jetty_response_total{base_url="http://solr6:8983/solr",status="1xx",} 0.0
solr_metrics_jetty_response_total{base_url="http://solr6:8983/solr",status="2xx",} 715.0
solr_metrics_jetty_response_total{base_url="http://solr6:8983/solr",status="3xx",} 0.0
solr_metrics_jetty_response_total{base_url="http://solr6:8983/solr",status="4xx",} 0.0
solr_metrics_jetty_response_total{base_url="http://solr6:8983/solr",status="5xx",} 0.0

Prometheus exporter is able to parse SOLR Metrics JSON response in order to provide a subset of metrics by using an XML configuration file.

https://lucene.apache.org/solr/guide/7_3/monitoring-solr-with-prometheus-and-grafana.html

<config>
  <rules>
    <metrics>
      <lst name="request">
        <lst name="query">
          <str name="path">/admin/metrics</str>
          <lst name="params">
            <str name="group">all</str>
            <str name="type">all</str>
            <str name="prefix"></str>
            <str name="property"></str>
          </lst>
        </lst>
        <arr name="jsonQueries">
          <str>
            .metrics["solr.jetty"] | to_entries | .[] | select(.key | startswith("org.eclipse.jetty.server.handler.DefaultHandler")) | select(.key | endswith("xx-responses")) as $object |
            $object.key | split(".") | last | split("-") | first as $status |
            $object.value.count as $value |
            {
            name         : "solr_metrics_jetty_response_total",
            type         : "COUNTER",
            help         : "See following URL: https://lucene.apache.org/solr/guide/metrics-reporting.html",
            label_names  : ["status"],
            label_values : [$status],
            value        : $value
            }
          </str>
        </arr>
      </lst>
    </metrics>
  </rules>
</config>

By using this configuration, SOLR Metrics from SOLR 6.6.5 release can be consumed even using a 7.3.0 Prometheus Exporter version.

Prometheus and Grafana

Once the SOLR Exporter is producing the right format for Prometheus, a new target can be declared in Prometheus configuration file.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:

scrape_configs:

  - job_name: 'solr'

    static_configs:
      - targets: ['solr-exporter:9854']

And Grafana can use this data source to build a Monitoring Dashboard.

Grafana Dashboard for Alfresco SOLR Monitoring

Putting it all together

A Docker Compose template has been provided to test this configuration with Alfresco:

https://github.com/aborroy/alfresco-solr-monitoring

Once the composition is running, not only Alfresco services are available, but also Prometheus Exporter, Prometheus and Grafana are living.

So you can test your ACS 6.2 stack accessing to:

http://localhost:8080/share         - Alfresco Share WebApp
http://localhost:8080/alfresco      - Alfresco Repository
http://localhost:8083/solr          - Alfresco Search Services

And the Monitoring services in:

http://localhost:9854     - Solr Exporter metrics
http://localhost:9090     - Prometheus UI
http://localhost:3000 - Grafana UI

Recap

Extending this base configuration, several customizations can be deployed:

• Prometheus exporter configurations in order to provide different subsets of metrics. For instance, monitoring a bulk ingestion might be different from monitoring a searching system.
• Different Grafana dashboards can be built in order to visualize groups of metrics related with performance, resources consumption or cache usage.