Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

zeitlinger · 2025-01-17T12:53:53Z

Fixes #236

This is a new take on the issue. #349 has been created before and not completed.

So what's the difference now?

The proposed rules are already implemented in the OTel operator and work well: https://github.com/open-telemetry/opentelemetry-operator#configure-resource-attributes - the PR (at least in it's initial form) makes the operator implementation fully compatible with this spec PR.
The specification work on service.instance.id has been completed - which was effectively a blocker before.

Merge requirement checklist

CONTRIBUTING.md guidelines followed.
Change log entry added, according to the guidelines in When to add a changelog entry.
- If your PR does not need a change log, start the PR title with [chore]

dashpole

Love the overall direction. This makes a lot of sense

docs/resource/k8s.md

dmitryax · 2025-01-22T16:32:35Z

We briefly discussed this during the k8s SemConv WG meeting. There is no consensus on the document yet, but we decided to recommend moving it from the k8s resource section of the semantic conventions repo to service because it prescribes a way of populating values of service attributes, not attributes in k8s namespace.

docs/resource/k8s.md

jinja2 · 2025-01-23T21:16:22Z

docs/resource/k8s.md

+- `pod.annotation[resource.opentelemetry.io/service.version]`
+- `pod.label[app.kubernetes.io/version]` (well-known label
+  [app.kubernetes.io/version](https://kubernetes.io/docs/reference/labels-annotations-taints/#app-kubernetes-io-version))
+- `if (contains(container docker image tag, '/') == false) container docker image tag`


Is there a reason we bail if the tag has /? The service attribute conventions don't define a format for version. If none of the other options work, wouldn't using the tag be better (not sure what the sdk defaults to)?

I'm unsure about this part of the spec.

I took this from the way the operator logic - collector does it a bit differently.

I've found the reason - it was a guard for a port number separator as can be seen here: https://github.com/distribution/reference/blob/284a39eaf3368476e0a4c6114a0eec61220acdd9/reference_test.go#L250

I've updated the spec to refer to that library - which is already used by the collector for container.image.id -

... with the notable difference that I think we should include the digest if available, it's the best source of truth for the service version

jinja2 · 2025-01-23T21:21:20Z

docs/resource/k8s.md

+Choose the first value found:
+
+- `pod.annotation[resource.opentelemetry.io/service.instance.id]`
+- `concat([k8s.namespace.name, k8s.pod.name, k8s.container.name], '.')`


We discussed service.instance.id in the semconv meeting and had some doubts about the scenario where a container is restarted without a pod recreate (e.g. kubelet restarting a container for OOM). The confusion was whether this id should change to reflect that a new instance of the container was started which is what would happen i think if we left this to an sdk.

The convention for service.instance.id states that it MUST be unique for each instance of the same service.namespace,service.name pair (in other words service.namespace,service.name,service.instance.id triplet MUST be globally unique). For a pod which is owned by a statefulset this is not the case. I guess the question is whether this id should be used to differentiate between different instances of a service at a given time or across all instances ever run. In case of a statefulset, the id is not unique enough to differentiate between instances of a pod with same sts ordinal but k8s will only run one instance of a pod with a given ordinal (the id will remain same after an sts pod is recreated). If we want a different id across pod recreates, switching to pod uid instead of name would work.

I've generally interpreted "globally unique" here to mean unique at a given time, rather than unique across all time. IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

Are sdks going to ignore any service.instance.id values passed in with the env var OTEL_RESOURCE_ATTRIBUTES? My understanding is that the documented values are currently being calculated and set in this var by operator when auto-instrumenting.

IMO this is somewhat moot since SDKs are going to start generating unique IDs for service.instance.id anyways.

Are sdks going to ignore any service.instance.id values passed in with the env var OTEL_RESOURCE_ATTRIBUTES? My understanding is that the documented values are currently being calculated and set in this var by operator when auto-instrumenting.

no, SDKs must honor whatever attributes are set using OTEL_RESOURCE_ATTRIBUTES

I've generally interpreted "globally unique" here to mean unique at a given time, rather than unique across all time.

that is also my understanding

thanks @jsuereth for the feedback on this question, which I have now included in the PR itself:

Note that the container restart count is not included in the service.instance.id calculation, because it makes
troubleshooting harder when the ID changes on every restart, e.g. in a crash loop.

zeitlinger · 2025-01-27T13:28:15Z

We briefly discussed this during the k8s SemConv WG meeting. There is no consensus on the document yet, but we decided to recommend moving it from the k8s resource section of the semantic conventions repo to service because it prescribes a way of populating values of service attributes, not attributes in k8s namespace.

@open-telemetry/specs-semconv-approvers the service page is automatically generated - how can I move the content of this PR to the service page?

trask · 2025-01-28T21:40:45Z

@open-telemetry/specs-semconv-approvers the service page is automatically generated - how can I move the content of this PR to the service page?

maybe as subsection(s) under https://github.com/open-telemetry/semantic-conventions/blob/main/docs/resource/README.md#service

docs/resource/README.md

zeitlinger · 2025-02-03T14:12:14Z

@jsuereth @tigrannajaryan thanks for the positive feedback in the entities SIG call. A added the notes about the object hierarchy for service.name and value stability for service.instance.id. Please let me know if this addresses everything.

zeitlinger · 2025-02-03T14:25:38Z

@jinja2 thanks for the detailed review - everything should be addressed now

zeitlinger · 2025-02-06T10:31:35Z

@open-telemetry/specs-semconv-approvers please take a look

Approval from the k8s semconv and entities SIG has been granted now

jsuereth · 2025-02-10T19:33:32Z

docs/resource/k8s.md

@@ -439,3 +439,17 @@ A ResourceQuota provides constraints that limit aggregate resource consumption p
 <!-- endsemconv -->

 [DocumentStatus]: https://opentelemetry.io/docs/specs/otel/document-status
+
+## Specify resource attributes using Kubernetes annotations


I'm not sure this is the best place for these docs.

I agree with what's written, just not location. It's ok here for now

@trask I think this was your suggestion - IIUC

it was from @dmitryax #1756 (comment), I have no objection to moving it

I tried adding it to the yaml - along with modifying the jinja template.

It looked strange to have markdown in yaml - so I'm wondering where we could put the text.
The goal was to have the k8s guidelines for service attributes part of the service spec.

Here's how it looks embedded in yaml:

groups: - id: registry.service type: attribute_group display_name: Service Attributes brief: > A service instance. note: > Note: `service.namespace` and `service.name` are not intended to be concatenated for the purpose of forming a single globally unique name for the service. For example the following 2 sets of attributes actually describe 2 different services (despite the fact that the concatenation would result in the same string): ``` # Resource attributes that describes a service. namespace = Company.Shop service.name = shoppingcart ``` ``` # Another set of resource attributes that describe a different service. namespace = Company service.name = Shop.shoppingcart ``` ### Service in Kubernetes

@joaopgrassi or @lmolkova do you have a suggestion?

Since this all non-normative "suggestion-like", I think it would be nicer if it was its own document, inside https://github.com/open-telemetry/semantic-conventions/tree/main/docs/non-normative. So then everything in this PR goes to a new document there. WDYT?

docs/resource/README.md

…ributes

…tice

…d prod (deployment.envirnoment.name)

Co-authored-by: Joao Grassi <[email protected]>

joaopgrassi · 2025-02-12T13:43:47Z

@open-telemetry/specs-semconv-maintainers this seems like a good PR to include in https://changelog.opentelemetry.io. Should we add the label to it? 🤔

trask · 2025-02-12T14:55:25Z

@open-telemetry/specs-semconv-maintainers this seems like a good PR to include in https://changelog.opentelemetry.io. Should we add the label to it? 🤔

sure! I don't think it strictly falls under "New semantic convention areas or stability changes, as well as releases", but I think it's always ok for @open-telemetry/specs-semconv-maintainers to call out particularly interesting or broadly relevant changes on https://changelog.opentelemetry.io

cc @austinlparker

zeitlinger requested review from a team as code owners January 17, 2025 12:53

zeitlinger mentioned this pull request Jan 17, 2025

[processor/k8sattributes] Operator resource attributes open-telemetry/opentelemetry-collector-contrib#37114

Open

dashpole reviewed Jan 17, 2025

View reviewed changes

docs/resource/k8s.md Outdated Show resolved Hide resolved

docs/resource/k8s.md Outdated Show resolved Hide resolved

docs/resource/k8s.md Outdated Show resolved Hide resolved

ChrsMark added the area:k8s label Jan 20, 2025

ChrsMark requested a review from a team January 20, 2025 10:59

ChrsMark reviewed Jan 21, 2025

View reviewed changes

docs/resource/k8s.md Outdated Show resolved Hide resolved

jinja2 reviewed Jan 23, 2025

View reviewed changes

docs/resource/k8s.md Outdated Show resolved Hide resolved

jinja2 reviewed Jan 23, 2025

View reviewed changes

github-actions bot added the enhancement New feature or request label Jan 28, 2025

jinja2 reviewed Jan 30, 2025

View reviewed changes

docs/resource/README.md Outdated Show resolved Hide resolved

zeitlinger force-pushed the k8s-label-recommendations branch from 604cdc9 to d850c85 Compare February 3, 2025 13:53

dashpole approved these changes Feb 5, 2025

View reviewed changes

jinja2 approved these changes Feb 5, 2025

View reviewed changes

jsuereth reviewed Feb 10, 2025

View reviewed changes

jsuereth approved these changes Feb 10, 2025

View reviewed changes

joaopgrassi reviewed Feb 11, 2025

View reviewed changes

docs/resource/README.md Outdated Show resolved Hide resolved

docs/resource/README.md Show resolved Hide resolved

joaopgrassi approved these changes Feb 12, 2025

View reviewed changes

docs/resource/README.md Outdated Show resolved Hide resolved

smithclay and others added 4 commits February 12, 2025 11:03

Add recommendation for well-known k8s labels to existing resource att…

8278a3a

…ributes

update resource attribute transformations based on operator best prac…

53f2da3

…tice

update resource attribute transformations based on operator best prac…

4c488a5

…tice

update resource attribute transformations based on operator best prac…

2467a2f

…tice

zeitlinger and others added 12 commits February 12, 2025 11:03

add fallback to k8s.namespace.name

ef0c5e9

remove part-of, which can sometimes be used to distinguish staging an…

4282baa

…d prod (deployment.envirnoment.name)

add app.kubernetes.io/instance

ee153b4

clarify how version is calculated

b0359d8

clarify how version is calculated

b8e1e33

move to service section

c99add9

add rationale

1bf4551

use "should be calculated"

ea02400

use "should be calculated"

8b31d82

cross link

3329467

fix link to resource attributes

98a09f9

Update docs/resource/README.md

a5ba77b

Co-authored-by: Joao Grassi <[email protected]>

zeitlinger force-pushed the k8s-label-recommendations branch from 2a21f8c to a5ba77b Compare February 12, 2025 10:04

austinlparker added the changelog.opentelemetry.io label Feb 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

zeitlinger commented Jan 17, 2025 •

edited

Loading

dashpole left a comment

dmitryax commented Jan 22, 2025

jinja2 Jan 23, 2025

zeitlinger Jan 27, 2025

zeitlinger Jan 28, 2025

jinja2 Jan 23, 2025

jinja2 Jan 23, 2025 •

edited

Loading

dashpole Jan 24, 2025

jinja2 Jan 24, 2025 •

edited

Loading

zeitlinger Jan 27, 2025

zeitlinger Jan 27, 2025

zeitlinger Feb 3, 2025

zeitlinger commented Jan 27, 2025

trask commented Jan 28, 2025

zeitlinger commented Feb 3, 2025

zeitlinger commented Feb 3, 2025

zeitlinger commented Feb 6, 2025

jsuereth Feb 10, 2025

zeitlinger Feb 11, 2025

trask Feb 11, 2025

zeitlinger Feb 12, 2025

joaopgrassi Feb 12, 2025 •

edited

Loading

joaopgrassi commented Feb 12, 2025

trask commented Feb 12, 2025

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Are you sure you want to change the base?

Kubernetes: How to populate resource attributes based on attributes, labels and transformation #1756

Conversation

zeitlinger commented Jan 17, 2025 • edited Loading

Merge requirement checklist

dashpole left a comment

Choose a reason for hiding this comment

dmitryax commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinja2 Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinja2 Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeitlinger commented Jan 27, 2025

trask commented Jan 28, 2025

zeitlinger commented Feb 3, 2025

zeitlinger commented Feb 3, 2025

zeitlinger commented Feb 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joaopgrassi Feb 12, 2025 • edited Loading

Choose a reason for hiding this comment

joaopgrassi commented Feb 12, 2025

trask commented Feb 12, 2025

zeitlinger commented Jan 17, 2025 •

edited

Loading

jinja2 Jan 23, 2025 •

edited

Loading

jinja2 Jan 24, 2025 •

edited

Loading

joaopgrassi Feb 12, 2025 •

edited

Loading