fix: Allow overlapping context scopes

[bantonsson]

Fixes #1887

This PR is fully functional, but not complete with edge case checks, tests and comments. It's ment as a basis for discussions.

Changes

This PR changes the single current Context to a more robust stack that is resilient to out of order and overlapping Context scopes. This robustness is needed to be able to support better interop with tokio-rs/tracing.

There are some context_attach benchmarks added, and this implementation performs the same or between 3-17% faster than the old one (while maintaining a working context state).

As for the context benchmarks in opentelemetry-sdk, this implementation performs the same except for the context/is_sampled/?/alt variants where it is 6-7% slower. I haven't dug into this too much at this point.

Update 2025-02-04

Two things:

1. I've moved out the actual code that we're benchmarking in the context benchmark into separate functions that are not inlined into the iterator loop, and there are now no differences in the context benchmark between the old way, and the new context stack.
2. I've added an optimization of the cloning of Context, that increases the throughput of context_attach 60-80% for empty_cx, 230-330% for single_value_cx and 50-80% for span_cx. The optimization also increases the throughput of the spec versions of the context benchmark between 20-50%.

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

Attention: Patch coverage is 97.77778% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.0%. Comparing base (b358202) to head (94ddd7e).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
opentelemetry/src/context.rs	97.7%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #2378   +/-   ##
=====================================
  Coverage   79.0%   79.0%           
=====================================
  Files        120     120           
  Lines      22512   22580   +68     
=====================================
+ Hits       17791   17857   +66     
- Misses      4721    4723    +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cijothomas]

@bantonsson Can you add some details on how this fixes the issue of contexts being dropped out-of-order? Perhaps some code comments will help future readers.

[bantonsson]

@cijothomas I'll clean this up a bit more and add comments describing the ContextStack and how it solves the out of order closing of contexts.

[bantonsson]

@cijothomas Do you know if the integration tests are flaky? The previous run failed but now it succeeded without any functional changes to the code.

[cijothomas]

@cijothomas Do you know if the integration tests are flaky? The previous run failed but now it succeeded without any functional changes to the code.

You can ignore it for now. Integration tests are not marked required for merging PRs. It does has some instability due to spinning up OTel Collector in a docker and multiple tests using the same instance etc. That will be addressed separately.

[shaun-cox]

Interesting. I too am curious to know a bit more about tokio-rs/tracing use cases that cause the context guards to be dropped out of order.

What is the expected behavior of the context stack if new contexts are pushed during out-of-order pops?