[Metricbeat][Azure]Check for metrics dimensions before matching two metrics together

[MichaelKatsoulis]

Proposed commit message

• WHAT: Check for metrics dimensions before matching two metrics together in azure module.
• WHY: Metric definitions are matched although they may contain different dimensions, leading to documents with <empty> value in ES

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

There will no longer be documents with empty value in its dimensions in ES. But this was a bug.

How to test this PR locally

Deploy metricbeat with azure module before and after the PR and set the following configuration for azure.yaml.
Set the right client_id, client_secret, tenant_id and subscription_id.

- module: azure
  metricsets:
  - storage
  enabled: true
  resources:
    - resource_group: ["mbranca-general"]
  period: 300s
  client_id: ''
  client_secret: ''
  tenant_id: ''
  subscription_id: ''
  refresh_list_interval: 600s

Related issues

• Closes [Azure monitor] Metric values are not mapped to the correct metric definition #42536

Use cases

Screenshots

Before the PR. Version 8.17.0. Storage accounts.

After the PR:

Before the PR. Values of azure.dimensions.authentication

After the PR. Values of azure.dimensions.authentication

Before the PR. Values of azure.dimensions.tier

After the PR. Values of azure.dimensions.tier

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MichaelKatsoulis? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[zmoog]

Comparing metric definitions without taking dimensions into account seems incorrect.

Here's an example where, running the code on main, the matchMetrics() function returns true even when the two definitions have different dimensions:

While testing, I noticed that the docs with <empty> dimension sometimes contain the sum of the non-<empty> dimension values.

However, I also found documents where the value for <empty> dimensions seemed not to have any relation with others. Possibly a random assignment due to the incomplete comparison.

I would only suggest adding a test case similar to https://github.com/lastic/beats/blob/71900c4d89fa3b29f8709bb46a61a9ad78a1e9c2/x-pack/metricbeat/module/azure/client_utils_test.go#L61-L85

So we can prove and document the behavior using different dimensions.

LGTM, thank you for taking the time to dive deeper in this issue!

[MichaelKatsoulis]

/test