Flaky test: opentelemetry::error_sets_span_status intermittently panics with "expected last span data"

[polyglotAI-bot]

Describe the bug

opentelemetry::error_sets_span_status (tests/it/opentelemetry.rs) fails intermittently
in CI. It is not tied to a ClickHouse server version and not tied to a particular commit:
it has failed on main in scheduled runs on 4 separate days in the last week, on two
different server legs, and on an unrelated PR branch on a third leg. The same check passes
on the other legs of the same run.

The assertion that fails is the expect at tests/it/opentelemetry.rs:149:

thread 'opentelemetry::error_sets_span_status' panicked at tests/it/opentelemetry.rs:149:43:
expected last span data

so LAST_SPAN_DATA was None — no span reached the test's span processor on the test's
own thread. It is not the later status assertion (expected error status, got ...).

Steps to reproduce

The failure is non-deterministic. It appears only in CI so far.

1. Run the it suite with the opentelemetry feature enabled — in CI this is the
   cargo llvm-cov test --workspace --no-report --all-features step of test-local.
   The three opentelemetry tests only compile under --all-features, so each CI leg
   executes them once.
2. Repeat. Roughly 4 of the last 14 executions failed.

Evidence of non-determinism

Failing runs (all test-local, all the same assertion, all on unrelated commits):

• 2026-08-24 scheduled, main, leg head — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32692955489/job/97329789872
• 2026-08-23 scheduled, main, leg head — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32619689568/job/97145842648
• 2026-08-22 scheduled, main, leg head — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32553579597/job/96984138868
• 2026-08-17 scheduled, main, leg latest — https://github.com/ClickHouse/clickhouse-rs/actions/runs/31997254612/job/95290996512
• 2026-08-24, PR run, leg pinned to server 26.6 — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32693722038

Passing runs of the same check on the same code: the latest leg passed in each of the
three scheduled runs above where head failed, and the whole scheduled matrix was green on
2026-08-13, 08-14, 08-15, 08-16, 08-18, 08-19, 08-20 and 08-21.

Because the failures cover server legs latest, head and a pinned 26.6, the failure is
not a server-version dependency. tests/it/opentelemetry.rs has not changed since May 2026,
so it is also not a recent change to the test itself.

Local reproduction

Not reproduced on an idle or loaded developer box:

• cargo test --all-features --test it opentelemetry:: — 0 failures in 25 consecutive runs.
• Full it suite, --all-features, 3 runs, under heavy CPU contention (4-core box, load
  average ~30) — error_sets_span_status passed in all 3.

Server: ClickHouse 26.3 in Docker. Client: main at b508875. Rust 1.89.

Additional observation

In the failing CI runs the captured output for the test is empty — the
eprintln!("last span data: {span:?}") inside LastSpanProcessor::on_end
(tests/it/opentelemetry.rs:180) produced nothing on that test's thread. libtest captures
output per thread, so on a failing run either no span was ended at all, or the span was
ended on a different thread than the one running the test body. On a passing run the same
test emits that line for both the query and the response span.

Suspected fragile area (not a confirmed root cause)

The test observes span data through a thread-local:

thread_local! {
    static LAST_SPAN_DATA: Cell<Option<SpanData>> = const { Cell::new(None) };
}

LastSpanProcessor::on_end writes it, and the test body reads it with
LAST_SPAN_DATA.take(). The tracer provider is global and installed once per process
(static ONCE: Once), and the subscriber is installed per test with a thread-local
set_default(), while the three opentelemetry tests run concurrently on different
libtest threads. That makes the assertion depend on the span being ended on exactly the
thread that runs the test body, and on the span being ended before control returns to the
test body — neither of which the test enforces. We were not able to pin down which of these
actually breaks in CI, so this is offered as the area to look at, not as a diagnosis.

Suggested direction

Make the observation independent of the thread the span ends on, and make the test wait for
the span instead of assuming it has already arrived — for example store exported spans in a
process-wide Mutex<Vec<SpanData>> keyed by trace id and look up the trace id the test
created, rather than "the last span on this thread". Forcing the opentelemetry tests onto
a single thread would also remove the cross-thread part, but not the ordering part.

Please do not paper over it with a retry or a sleep — if the span really can be ended on a
foreign thread or after the request future completes, that is worth knowing.

Configuration

Environment

• Client version: main (b508875), also observed on every main commit in the window above
• OS: ubuntu-latest GitHub runner (CI); reproduction attempt on Ubuntu 24.04, 4 cores

ClickHouse server

• ClickHouse Server version: fails on latest, head and pinned 26.6; local attempt on 26.3
• ClickHouse Server non-default settings, if any: none (repo docker-compose.yml)

---

Found by automated CI monitoring of our own PRs, then confirmed against the scheduled main
runs. Reported by @polyglotAI-bot.