Skip to content

docs: describe how OTel resource attributes are translated and stored in labels #15509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
carsonip opened this issue Jan 31, 2025 · 16 comments · Fixed by elastic/docs-content#1186
Assignees
Labels
docs good first issue Team:Docs Label for the Observability docs team

Comments

@carsonip
Copy link
Member

There are no docs describing how only a subset of OTel resource attributes are translated to ECS, and how the rest are converted, e.g. replacing . with _, and stored under labels.*. These pages are good candidates to describe this limitation.

@carsonip carsonip added docs Team:Docs Label for the Observability docs team good first issue labels Jan 31, 2025
@harshithsaiv
Copy link

Hi @carsonip !

I would love to contribute to this. Can I give this a try? If possible, could you assign it to me?

Thanks!

@carsonip
Copy link
Member Author

carsonip commented Feb 5, 2025

Thanks @harshithsaiv ! The docs are located in https://github.com/elastic/observability-docs . Let's proceed without adding an assignee to this issue first and let me know if you need any help.

@harshithsaiv
Copy link

harshithsaiv commented Feb 21, 2025

Hi @carsonip Could you please provide the exact list of OTel resource attributes that are translated to ECS fields? This will help ensure the documentation is accurate and comprehensive.

Or I can just generalize it like this

For example:

  • deployment.environment is translated to service.environment in ECS.
  • Other attributes like custom.attribute are converted to labels.custom_attribute.

@carsonip
Copy link
Member Author

Hey @harshithsaiv , the otlp translation logic is in https://github.com/elastic/apm-data/tree/main/input/otlp and watch out for the tricky parts where

  • there's different handling for otlp spans that maps to APM span vs APM transaction
  • some fields may be translated conditionally (depending on attribute value type)
  • labels captures any fields that are not translated

@harshithsaiv
Copy link

Hi @carsonip,

I've updated the OpenTelemetry resource attributes documentation to provide more detailed information about how OTel attributes are mapped to ECS fields. These changes address the documentation gap identified in issue #15509.

Key improvements:

  • Added clear explanations about how unmapped attributes are stored under labels.* with dots replaced by underscores
  • Documented the different handling of OpenTelemetry spans (mapping to either APM transactions or spans based on span kind)
  • Added detailed examples showing the conditional attribute translation based on value types
  • Included JSON examples to better illustrate the transformation process

I noticed there's an error in the CI workflow about AsciiDoc files being migrated to Markdown. I understand these changes might need to be converted to Markdown format as part of the ongoing migration. Would you be able to advise on the best path forward? Should I:

  1. Convert these changes to Markdown format instead?
  2. Wait until after the migration is complete?
  3. Proceed with this PR with some special handling?

Thank you for your guidance and for reviewing these documentation improvements!

@carsonip
Copy link
Member Author

carsonip commented Mar 3, 2025

Thanks for the PR elastic/observability-docs#4834 . I'll either leave some comments on this one, or a new one in markdown format (explained below)

I noticed there's an error in the CI workflow about AsciiDoc files being migrated to Markdown. I understand these changes might need to be converted to Markdown format as part of the ongoing migration. Would you be able to advise on the best path forward? Should I:
Convert these changes to Markdown format instead?
Wait until after the migration is complete?
Proceed with this PR with some special handling?

It was unfortunate timing. The docs have been moved to https://github.com/elastic/docs-content . Therefore, the answer is you should close elastic/observability-docs#4834 and open one in https://github.com/elastic/docs-content

@harshithsaiv
Copy link

Thank you for the update, @carsonip. I'll go ahead and close the PR and open a new one in https://github.com/elastic/docs-content.

@carsonip
Copy link
Member Author

hey @harshithsaiv , just checking in to see if you need any help with the new docs repo. The closed PR looks good at a high level and I'm ready to review your new on in elastic/docs-content once it's ready. Thanks!

@colleenmcginnis
Copy link
Contributor

👋 @theletterf this is the issue I picked up this week, but I was waiting on a resolution in elastic/docs-content#985 before taking any action.

@theletterf
Copy link

As agreed with @AlexanderWert, the changes should happen in the https://github.com/elastic/opentelemetry repo for now.

@carsonip
Copy link
Member Author

As agreed with @AlexanderWert, the changes should happen in the https://github.com/elastic/opentelemetry repo for now.

Just to confirm, in your discussion, was it clear that this piece of documentation is only relevant to apm server and MIS ingest path, and that it is not relevant to EDOT?

@theletterf
Copy link

Oh, since I joined (calculates) not even six days ago, probably not, he he.

If it doesn't apply to EDOT, I'mm all for going forward with an edit here or in the new v3 docs. Would gladly get more context in a call.

@carsonip
Copy link
Member Author

If it doesn't apply to EDOT, I'mm all for going forward with an edit here or in the new v3 docs. Would gladly get more context in a call.

As elastic/docs-content#985 goes in the direction of removing mentions (including limitations etc) of OTel in Elastic Observability, I wonder if we still want to document this limitation of APM Server and MIS around OTel resource attributes at all. Let's have a wider internal sync on this topic.

@carsonip
Copy link
Member Author

Summary of the sync - see elastic/observability-docs#4843 . We can press forward with this documentation task in docs-content

@theletterf
Copy link

theletterf commented Apr 17, 2025

Opening a PR in docs-content using the docs provided by @harshithsaiv.

@colleenmcginnis Do we need to backport (is that the right term?) this to 8.x?

@carsonip
Copy link
Member Author

Do we need to backport (is that the right term?) this to 8.x?

The documented table applies to 8.x, but 8.x are documented in observability-docs repo, making backporting a bit more troublesome. It is up to the docs team to decide whether it is worth the trouble.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs good first issue Team:Docs Label for the Observability docs team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants