Adding health report to logstash integration

[flexitrev]

Proposed commit message

Added data and dashboards including Logstash Health report

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Closes #12465

Screenshots

Node Health

Pipeline Health

[yaauie]

• On the "General Pipeline Information" section:
  • I would not expect TERMINATED pipelines to have load averages; the presence of the 0's and green color in those columns visually tells me that everything is A-okay.
  • Is "Load" in this section short-hand for Worker Utilization? Although it's shorter, I'm wary of overloading the word "Load" to mean new things (from a dev-ops perspective, it already has a very specific meaning related to CPU instructions queued per CPU tick)
  • Can we color-code the whole row based on the "Status" column (and if so, do we have prior-art for color-sets that are less in-your-face than #00ff00 #999999 #ffff00 #ff0000 )?
  • It feels like this is a smash-up of our two initial probes, but what happens when we add more?
    • in our diagnostic tool, we can compare two windows, IIRC it is always "current" against a selectable baseline, which lets us reduce the number of columns.

[strawgate]

• I would not expect TERMINATED pipelines to have load averages; the presence of the 0's and green color in those columns visually tells me that everything is A-okay.

@flexitrev With the above sections covering terminated pipelines, we could change the bottom section to be running pipelines and just filter out terminated pipelines.

Can we color-code the whole row based on the "Status" column (and if so, do we have prior-art for color-sets that are less in-your-face than #00ff00 #999999 #ffff00 #ff0000 )?

I do not believe you can color-code the whole row based on a cell -- you can color code the cell based on its value, though. As for the colors themselves, the color sets are the EUI palettes that ship with Kibana and will follow any changes the kibana team makes to the values for the colors. In general I think we should stick with the built-in palettes for integrations we ship to customers.

[flexitrev]

Added a filter to the Worker utilization by pipeline to exclude terminated pipelines. Should I exclude finished too?

Colors are expected to change with project borealis - If they don't automatically get picked up by us picking builtin colors, we'll update later. https://github.com/elastic/search-team/issues/8943

[flexitrev]

Updated dashboard, simplified worker utilization by pipeline table, modified visualization names, changed status to percentage.

[flexitrev]

Thanks! I tried it at the top level manifest but not on the individual streams -

[flexitrev]

It looks like we removed event.dataset prematurely -- it looks like CI was failing because sample-event.json had two values for event.dataset: https://buildkite.com/elastic/integrations/builds/21145#0194aee4-6d12-43ad-8f03-233889b509f1

build/test-coverage/coverage-logstash-pipeline-1738101594676982809-report.xml

<testcase name="static test: Verify sample_event.json" classname="logstash.health_report" time="0.115375683">
<failure>test case failed: one or more errors found in document: [0] field "event.dataset" should have value in ["logstash.health_report"], it has "logstash.pipeline"</failure>
</testcase>

In the sample_event.json when CI was failing there were two event.dataset values in the document: 13ed93d

"event": {
  "agent_id_status": "verified",
  "ingested": "2025-01-28T18:41:28Z",
  "dataset": "logstash.health_report"
},
"event.dataset": "logstash.pipeline"
}

@strawgate Yes, I thought it best to just keep logstash.health_report. Should I keep both somehow? Rename event.dataset:logstash.pipeline?

[strawgate]

Ideally, there would be two sample events, the node sample event and the pipeline sample event?

[flexitrev]

Ideally, there would be two sample events, the node sample event and the pipeline sample event?

That might require defining 2 datastreams health_report.node and health_report.pipeline

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: e5f3045

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History

• 💚 Build #21504 succeeded a440b14
• 💚 Build #21431 succeeded 5853431
• 💔 Build #21429 failed ea6b9f4
• 💚 Build #21238 succeeded df631ac
• 💚 Build #21228 succeeded 9fcdccd
• 💚 Build #21149 succeeded 13ed93d

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[flexitrev]

Cutting a new branch and creating a new PR - #12601