Document new core Prometheus Exporter

[danotorrey]

In the 4.1 release, the ability to export Graylog Dropwizard metrics to Prometheus is being added to the core open-source product in Graylog2/graylog2-server#10767. This works similarly to the existing Graylog Cloud server-side prometheus metrics exporting, except it allows the mappings between the Dropwizard and Prometheus metrics to be customized at runtime in a yaml file.

Expected Content

• Add the new configuration parameters to the server.conf docs page.
• Add a new page or section that describes the minimum details needed to understand, enable, and use the feature. There is no need to explain how to set up Prometheus. However, we should explain how to define the individual mappings.

Functionality Description

The configuration properties described in the server.conf file can be used to enable and customize the feature.

This Prometheus exporter heavily uses Prometheus metric labels to consolidate individual traditional Graylog Dropwizard metrics in to a combined metric with labels, which make it easer to view and analyze similar metrics together.

The initial export mappings are described here in prometheus-exporter.yml. Essentially, this is a mapping between Dropwizard metric names (wildcard patterns) and prometheus export names. The matching patterns are described in detail here.

The following example definition uses most of the export, mapping and labels functionality, note the metric_name and metric_pattern. All metrics exported to Prometheus will have the gl_ prefix appended to the name. So, the name in the below example will result as gl_auth_actions.

  - metric_name: "auth_actions"
    match_pattern: "org.graylog.plugins.cloud.auth.*.failure.http.*"
    wildcard_extract_labels:
      - "action"
      - "http_code"
    additional_labels:
      success: "false"
      type: "http"

The wildcard_extract_labels are an ordered sequence of label names to extract for the wildcard asterisks * that appear in the metric name. One label value for each value in place of the wildcard will be created.

For example, for the definition above, the following metric would produce the following labels:

metric: org.graylog.plugins.cloud.auth.tokenRequest.failure.http.500

label: action: tokenRequest
label: action: http_code: 500

The success and type labels are static labels.

Note that all metrics will automatically be assigned a node_id label with the value of the Graylog node id that the metric was exported from. This makes it easy to visualize metrics across Graylog nodes together, and also see which values are coming from which node.

I hope this explanation helps, but perhaps the best way to see how to definitions work is to use the exporter for some known metrics and map them to prometheus metrics and various labels to see what the output looks like.

I am not exactly sure how much detail we should go into in the documentation, since to use this feature, the user probably already knows Prometheus fairly well.

To test this functionality, the instructions to starting a Prometheus instance in Docker can be used:
https://docs.graylog.org/en/4.0/pages/enterprise/cloud_forwarder.html#prometheus-metrics-exports

[danotorrey]

FYI @bernd. Placeholder for core Prometheus exports docs.

[danotorrey]

@bernd I've added some details on how exporting works to the issue description.

@dulanism Recently, we decided to implement this new core Prometheus Exporting feature for Graylog 4.1 for a customer request. I know that you already have a lot of docs work on your plate for 4.1. Do you have any time to write documentation for this feature too? Please do not hesitate to say no if your list is full. Since I know the functionality well, I can also write some minimal documentation for it too, and request your review. Please let me know what works best for you.

[dulanism]

Hi @danotorrey. Yeah, could you write up something minimal? I added a note in a gdoc to traverse what you write into the draft

[danotorrey]

@dulanism Sounds good. Will do. How should I write up the first draft? Should I add that as a gdoc with screenshots, links etc., and let you add that the docs repo? Happy to add to the docs repo directly also. Let me know what works best for you.

[dulanism]

@danotorrey I can add it to the repos. Send what you have in a Google doc when you're ready. it might be worth our while to have a 15 minute convo too. Thanks!

[danotorrey]

@dulanism Sounds good!

[danotorrey]

From discussing with @dulanism I will file an initial PR for this directly in the documentation repo.

[danotorrey]

I have started the documentation, and will file a PR shortly.

[danotorrey]

Closed VIA #1166