Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Elastic Agent Dashboards #12524

Closed
wants to merge 5,968 commits into from
Closed

Update Elastic Agent Dashboards #12524

wants to merge 5,968 commits into from

Conversation

strawgate
Copy link
Contributor

@strawgate strawgate commented Jan 30, 2025

Fixes #12488

Benefits from:

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

Related issues

Screenshots

Overview

Overview

Concerning Agents

Problematic Agents

Agent Metrics

Agent Metrics

tommyers-elastic and others added 30 commits December 6, 2024 09:44
The problem is that the `ecs@mappings` component template (introduced in >=8.13.0) 
does not perform type coercion to long when the value was a string. In this 
specific integration scenario, the `event.severity` value was provided as a 
string. So the dynamic mapping never matched, which left `event.severity` with 
the default 'keyword' mapping type, which does not comply with ECS.

This change adds back the static mapping for `event.severity` and modifies the 
grok pattern to perform conversion to long.

Relates: elastic#10848

---------

Co-authored-by: Andrew Kroh <[email protected]>
In certain contexts, the navigation panel links on the netflow dashboard were not functioning correctly. This issue has been rectified by replacing the navigation panel with a 'links' panel, which is now functional in all contexts.
aws-s3 data does not have object values.
Add mappings for `journald.custom` fields, which will fix an automated system test failure.
…ic#11618)

This commit introduces support for Journald in the system integration,
adding a "journald" input for both auth and syslog data.

Conditions are used to select when to run the log input or the journal input, it
defaults to running the Journald input on Debian 12 and Amazon Linux 2023.

The minimum stack version is updated to 8.17.0 so the integration can use the
facilities filtering option from the journald input added in 8.16.0 and some extra fields
from the host provider added in 8.17.0.
- The current parser is not processing the hostname field of syslog messages, so in some cases, the observer.name field remains empty. Added support for parsing out the hostname field of syslog messages and placing it into the observer.name field.

---------

Co-authored-by: Taylor Swanson <[email protected]>
Co-authored-by: Andrew Kroh <[email protected]>
In PAN-OS threat logs, when threat_category is 'domain_edl', the misc field will contain a URL. This change adds parsing of the URL for this case.
…lastic#11766)

Revises the Check Point integration readme to include detailed instructions on utilizing the logfile input feature.
…lastic#11949)

Fixes for some new test data that was showing:
- Errors attempting to run conversions on absent fields.
- Events being dropped because they are too deep.

Checking field presence before conversion was difficult because those
processors were run in `foreach` loops, and Painless doesn't get access
to `_ingest._value`. This was fixed by extracting separate pipelines,
which could make the current value accessible at the start, then set it
back after processing.

To avoid dropping events with fields that are too deeply nested (more
than a depth of 20), some depth truncation logic was added.

There were also a number of field definitions added in order to make
tests pass for the new test data.
Tested against a real endpoint.

Pipeline test cases obtained from a test instance. Up to 10 examples of each
available type are included. Not all types are represented.
…nput (elastic#11984)

Switch the integration package from the one-input-per-data-stream model to the one-input model.

One input per data stream model:

![image](https://github.com/user-attachments/assets/ce60cd18-80fc-4805-aebf-322dcfd5f374)

One input model:

![image](https://github.com/user-attachments/assets/b8ae0dbd-6ca6-4caa-ad5c-3c4a6f573dee)

In the one-input model, there is only one azure-eventhub input running and sending events to the `events` data stream. In the `events` data stream, the ingest pipeline performs these tasks:

- discover and set the `event.dataset` field using the `category` field in the event.
- use the `event.dataset` field to reroute the event to the target data stream.

The discover process uses the following logic:

- if the event doesn't have a category, it sets `event.dataset` to `azure.eventhub` (the generic integration)
- if the event does have a category, it sets `event.dataset` to `azure.platformlogs` (it's probably an Azure log)
- if the event category is supported, it sets `event.dataset` to specific one like `azure.activitylogs` or `azure.signinlogs`.

After the discovery step, the routing rules use the `event.dataset` value to forward the events to the best available target data stream.
…2031)

Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…stic#12032)

Made with ❤️️ by updatecli

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
…lastic#12034)

Bumps [updatecli/updatecli-action](https://github.com/updatecli/updatecli-action) from 2.72.0 to 2.73.0.
- [Release notes](https://github.com/updatecli/updatecli-action/releases)
- [Commits](updatecli/updatecli-action@fb02bdc...11d8c3e)

---
updated-dependencies:
- dependency-name: updatecli/updatecli-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…stic#12035)

- Improved the mechanism which handles splitting key/value pairs
in traffic logs to handle situations where values are not in quotes,
and other edge cases.
- Fix missing space in test case
The kibana.audit data stream was missing a mapping
for the span.id field.

Fixes elastic#11658
Fixes elastic#11130
Fixes elastic#10691
Fixes elastic#10514
Fixes elastic#9839
- Add Alert Insight panel to the `Security Posture` dashboard.
- Add `SMB v1 Connections` visualization into the Unencrypted Traffic Hygiene panel.
- Update the `VPN Connections` visualization by changing it from a bar chart to a line chart.
Add a template to help support users making new feature requests for
existing integrations.
…tic#12048)

For cert updates, in _dev/deploy/docker/files run

openssl req -x509 -newkey rsa:2048 -keyout detection-private.key -out detection-certificate.crt -sha256 -days 3650 -nodes -subj "/C=XX/L=Default City/O=Default Company Ltd/CN=xx.incident-management.eset.systems"
openssl req -x509 -newkey rsa:2048 -keyout device_task-private.key -out device_task-certificate.crt -sha256 -days 3650 -nodes -subj "/C=XX/L=Default City/O=Default Company Ltd/CN=xx.automation.eset.systems"
openssl req -x509 -newkey rsa:2048 -keyout oauth-private.key -out oauth-certificate.crt -sha256 -days 3650 -nodes -subj "/C=XX/L=Default City/O=Default Company Ltd/CN=xx.business-account.iam.eset.systems"

and then distribute the certificate to the system test configs.
…ptions. (elastic#12053)

- Change the UI setting name from `Cloudflare R2 Bucket Name` to `S3-Compatible Bucket Name`.
- Generalise the setting's description.
- Also update README to generalise the option.
Based on a list of field names from a live system.
Hasn't been tested with full data.
* Add ELB Connection logs dashboard
@andrewkroh andrewkroh added the bugfix Pull request that fixes a bug issue label Jan 31, 2025
@elasticmachine
Copy link

💚 Build Succeeded

History

Copy link

Quality Gate failed Quality Gate failed

Failed conditions
75.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@qcorporation qcorporation requested review from a team as code owners February 4, 2025 03:56
@andrewkroh andrewkroh added Integration:1password 1Password Integration:abnormal_security Abnormal Security New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations] labels Feb 4, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix Pull request that fixes a bug issue dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:abnormal_security Abnormal Security Integration:1password 1Password New Integration Issue or pull request for creating a new integration package. Team:Security-Service Integrations Security Service Integrations Team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[elastic_agent] Improvements to the Elastic Agent Metrics Overview dashboard