Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[envoyproxy][mongodb_atlas] Update failing mappings when validation based on mappings enabled #12577

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from

Conversation

mrodm
Copy link
Contributor

@mrodm mrodm commented Feb 3, 2025

Proposed commit message

Updated failing mappings found for envoyproxy and mongodb_atlas packages when elastic-package uses validation based on mappings.

envoyproxy

Added missing mappings for envoy.thread_local_cluster_manager metrics.
Added the same mappings as the other groups (mean, min, max, count and so on).

Example of error found (buildkite link):

test case failed: one or more errors found in mappings in metrics-envoyproxy.stats index template:
[0] field "envoy.thread_local_cluster_manager.worker_0_clusters_inflated.value" is undefined: field definition not found

mongodb_atlas

Update flattened field definitions to not create dynamic templates.

As they were previously, this dynamic template is created:

      "dynamic_templates": [
        {
          "mongodb_atlas.project.additional_info.*": {
            "path_match": "mongodb_atlas.project.additional_info.*",
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        },
      ]

With the change of this PR, this mapping is created as:

        "mongodb_atlas": {
          "properties": {
            "project": {
              "properties": {
                "additional_info": {
                  "type": "flattened"
                },
              },
            },
          },
        },

Example of errors found (buildkite link):

test case failed: one or more errors found in mappings in logs-mongodb_atlas.project index template:
[0] field "mongodb_atlas.project.additional_info._t" is undefined: field definition not found
[1] field "mongodb_atlas.project.additional_info.cid" is undefined: field definition not found
[2] field "mongodb_atlas.project.additional_info.cre" is undefined: field definition not found
[3] field "mongodb_atlas.project.additional_info.description" is undefined: field definition not found
...

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Run elastic-package with mappings validation enabled and run system tests:

cd packages/<package>
elastic-package stack up -v -d --version <kibana_constraint>
export ELASTIC_PACKAGE_FIELD_VALIDATION_TEST_METHOD=mappings
# Add defer-cleanup to check the data ingested in the corresponding indices/data streams
elastic-package test system -v --data-streams memory --defer-cleanup 900s

elastic-package stack down -v

Related issues

Screenshots

@mrodm mrodm self-assigned this Feb 3, 2025
@andrewkroh andrewkroh added Integration:1password 1Password Integration:abnormal_security Abnormal Security New Integration Issue or pull request for creating a new integration package. labels Feb 4, 2025
@mrodm mrodm force-pushed the update_obs-infraobs-integrations_mappings branch from 1778899 to 959efee Compare February 4, 2025 17:22
@mrodm mrodm force-pushed the update_obs-infraobs-integrations_mappings branch from 959efee to e122783 Compare February 4, 2025 19:02
taylor-swanson and others added 9 commits February 4, 2025 15:02
…astic#12571)

Updates the following integrations to ECS 8.17.0:

- fortinet_fortiedr
- fortinet_fortigate
- fortinet_fortimail
- fortinet_fortimanager
- fortinet_fortiproxy
- goflow2
- hashicorp_vault
- imperva
- iptables
…astic#12569)

Update the following integrations to ECS 8.17.0:
- arista_ngfw
- cef
- checkpoint
- cisco_aironet
- cisco_asa
- cisco_ftd
- cisco_ios
- cisco_ise
- cisco_nexus
- cisco_secure_email_gateway
- citrix_waf
…astic#12572)

Updated the following integrations to ECS 8.17.0:
- juniper_srx
- modsecurity
- netflow
- osquery
- panw
- pfsense
- proxysg
…astic#12574)

Updated the following integrations to ECS 8.17.0:
- qnap_nas
- snort
- sonicwall_firewall
- sophos
- squid
- stormshield
- suricata
- tcp
- udp
- watchguard_firebox
- zeek
- The time format used "week-based-year" (Y) instead of "year-of-era" (y). The former
gives very different results when calculating the month and date, since it is week-based.
- Regenerated test files with updated dates.
elastic#12612)

* Fix dashboard query in [Metrics System] Host overview

* update pr link
@mrodm mrodm removed Integration:1password 1Password Integration:abnormal_security Abnormal Security New Integration Issue or pull request for creating a new integration package. labels Feb 5, 2025
marc-gr and others added 7 commits February 5, 2025 14:54
* Add 9.0.0 constraint to sec-windows-platform packages

* Fix typo

* add scanner options to allow small test files

* Update packages/microsoft_exchange_server/data_stream/httpproxy/agent/stream/filestream.yml.hbs

* Update packages/microsoft_exchange_server/data_stream/httpproxy/agent/stream/filestream.yml.hbs

* Update packages/microsoft_exchange_server/data_stream/httpproxy/agent/stream/filestream.yml.hbs

* Update packages/microsoft_exchange_server/data_stream/httpproxy/agent/stream/filestream.yml.hbs

* Update packages/microsoft_exchange_server/data_stream/httpproxy/agent/stream/filestream.yml.hbs
* Docker Enablement for v9.0.0

Signed-off-by: Andreas Gkizas <[email protected]>
…ingress, istio and containerd integrations (elastic#12535)

* Enabling 9.0.0 kibana support for  kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations

Signed-off-by: Andreas Gkizas <[email protected]>
@mrodm mrodm force-pushed the update_obs-infraobs-integrations_mappings branch from e122783 to e9a1f16 Compare February 5, 2025 17:30
@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

elasticmachine commented Feb 5, 2025

💔 Build Failed

Failed CI Steps

History

cc @mrodm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.