Skip to content

lapdog: OTLP traces accepted but not surfaced in LLM Observability UI (no OTel GenAI → LLMObs translation) #378

@adickinson72

Description

@adickinson72

Summary

When lapdog is run in its default --lapdog-mode, the OTLP receiver on
:4318/v1/traces accepts OpenTelemetry traces (200 OK) but the LLM
Observability UI never displays them. The UI only renders spans that arrive
via /evp_proxy/v2/api/v2/llmobs (i.e. via ddtrace's LLMObs SDK or
ddtrace-run).

This makes lapdog effectively unusable for frameworks whose Datadog support
is the OTel path — most notably AWS Strands Agents, which Datadog
themselves recommend instrumenting via OTel GenAI SemConv (see
DataDog/dd-trace-py#14707, where the LLMObs feature request for Strands was
closed by directing the user to the OTel OTLP endpoint instead).

Repro

  1. lapdog start (1.57.0, Homebrew on macOS).
  2. Run a Python app that emits OTel GenAI SemConv v1.37+ spans
    (e.g. Strands Agents via StrandsTelemetry().setup_otlp_exporter())
    pointed at OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces,
    OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf.
  3. Trigger an agent invocation.

Observed in ~/.lapdog/lapdog.log:

POST /v1/traces HTTP/1.1  200 149  "-"  "OTel-OTLP-Exporter-Python/1.42.1"

…and the protobuf body (visible via GET /test/session/requests) contains
the expected attributes: gen_ai.operation.name=chat,
gen_ai.request.model=…, gen_ai.usage.{input,output}_tokens,
session.id, gen_ai.conversation.id, etc.

But:

  • GET /test/session/traces returns [] (the APM trace store stays empty
    for OTLP-delivered traces).
  • The UI's POST /api/v1/logs-analytics/aggregate?type=llmobs queries
    return no spans for the invocation.

Switching to lapdog python app.py works because it forces
ddtrace's auto-instrumentation, which posts directly to
/evp_proxy/v2/api/v2/llmobs — but that path only captures the underlying
HTTP/LiteLLM/Bedrock calls and misses the Strands agent → tool → llm
hierarchy entirely.

Expected

OTel GenAI SemConv spans arriving on /v1/traces should be translated to
LLMObs spans (matching the behavior of the real Datadog Agent's OTLP
receiver) so that they appear in the lapdog UI alongside ddtrace-emitted
LLMObs spans.

Why this matters

Per DataDog/dd-trace-py#14707, OTel is Datadog's recommended path for
Strands. There is no native ddtrace Strands integration (the only Strands
code in dd-trace-py is under ai_guard, not LLMObs). Without OTel
translation in lapdog, Strands users have no local Datadog LLMObs option
and must point at a real DD Agent (e.g. via Docker) to develop against.

Workarounds

  • Run the real Datadog Agent locally instead of lapdog.
  • Use lapdog python … and accept that the Strands agent hierarchy is not
    captured (only the underlying ddtrace-instrumented LLM HTTP calls are).

Environment

  • lapdog 1.57.0 (Homebrew, macOS Darwin 25.5.0)
  • Python 3.11 / 3.13
  • ddtrace 4.x, opentelemetry-sdk 1.42.1
  • Strands Agents 1.x

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions