Support sending custom tags when using pollDBStats

[KaibutsuX]

What does this PR do?

Motivation

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[darccio]

@mtoffl01 Was there any reason behind not allowing tags from the beginning?

Checking the code, I see that for the purpose of this PR, it should be enough to set DD_TAGS or OTEL_PROPAGATORS (if using OTel) to set these global tags.

WDYT @KaibutsuX?

[KaibutsuX]

I don't have enough context or documentation to know what DD_TAGS is or how I would set it as an end-user.

[darccio]

DD_TAGS is properly explained in our Go tracer public documentation.

We understand this should be enough to achieve what you are trying with this PR.

[KaibutsuX]

That links says: "A list of default tags to be added to every span and profile." From my tracing through the code, I could not see that this same list of user-supplied tags was included in these metrics used in pollDBStats

[darccio]

Yes, we understand it may not that be clear from the code. Sorry about that. It inherits the global tags from the tracer config, set by globalconfig.SetStatsTags, which is initialized by newConfig, and retrieved when setting up the DB statsd client.

We'll update the public documentation to make sure it reflects this detail.

[mtoffl01]

Hi @KaibutsuX ,

As @darccio pointed out, tags set in DD_TAGS are applied to DBStats metrics. We are updating the Go tracer public documentation to include "metrics" in the list of data to which tags are applied to, so this should be clear for future readers.

Are you trying to pass your global tags down to DBStats metrics, or are you looking to apply a unique set of tags specifically to those metrics? If the former, DD_TAGS will do the trick. If the latter, this feature is not currently supported, but we'd be happy to learn more about your use case so we can consider implementing it.

Thank you!

[KaibutsuX]

@mtoffl01 I will see if using DD_TAGS works as expected.

You may want to also update the documentation for DD_TAGS as it is marked as an array, but in go code it accepts a map[string]interface{} which implicitly only respects the entries that can be asserted as strings. Further, it's unclear that the labels will be formatted as labelKey:labelValue.(string) especially since the documentation explicitly says that DD_TAGS supports multiple values as csv using commas or spaces.

[mtoffl01]

Hi @KaibutsuX ,

Thanks for the tip! We will change the Default value in the docs and update the description to specify that keys and values should be string-convertible types. In the meantime, keep us posted on your success with DD_TAGS.

[darccio]

@KaibutsuX Is everything solved with the tips we shared? Do you want to still push this into the codebase? Thanks!

[KaibutsuX]

Using the alternative suggestion does not allow me to set the metrics namespace. Instead all tags are sent to the default namespace, which in my use-case makes the tagging unhelpful and results in just more noise. For the time being, I'm still using a local version of this PR to support my own pollDBStats in which I send my tags AND set my own custom namespace.

[mtoffl01]

@KaibutsuX ,
What do you mean by "metric namespace" and "all tags are sent to the default namespace?" Indeed the expectation with DD_TAGS is that all configured key-values will be applied to all traces, metrics and logs submitted by this tracer instance.

If you would like to configure tags to apply only to your DB Stats metrics, please tell us some more about how you use these tags. If I've not understood the "namespace" term, please expand.

Thank you!

[KaibutsuX]

dd-trace-go/contrib/database/sql/option.go

Line 43 in 137578f

sc, err := internal.NewStatsdClient(globalconfig.DogstatsdAddr(), statsTags(c))

Since absolutely everything is being sent to the same namespace, if I'm only interested in 30 / 1000 tags is makes it hard to filter and use.

NewStatsdClient accepts options, one of which would support custom namespaces would be statsd.WithNamespace("my-custom-metrics")