[BUG][OBSERVABILITY]: All traces show "anonymous" user — ObservabilityMiddleware runs before HttpAuthMiddleware due to LIFO middleware registration order

[ShailedraSharma]

🐞 Bug Summary

The internal observability dashboard (/admin/observability) shows anonymous
for every trace regardless of the authenticated user. This makes the User Email
filter, per-user request tracking, and the User column in the trace table
completely non-functional.

🔍 Root Cause

In main.py, middleware is registered in this order:

Line 3043: app.add_middleware(HttpAuthMiddleware) ← registered first
Line 3155: app.add_middleware(ObservabilityMiddleware) ← registered last

Starlette processes middleware in LIFO order, so ObservabilityMiddleware
runs before HttpAuthMiddleware. At the time user_email is read in
observability_middleware.py:134-140, request.state.user has not been set yet:

# observability_middleware.py line 134-140
user_email = None
if hasattr(request.state, "user") and hasattr(request.state.user, "email"):
    user_email = request.state.user.email  # Always None — auth hasn't run yet

# Trace started with user_email=None
trace_id = self.service.start_trace(
    user_email=user_email,   # ← None
    ...
)

# call_next() runs here — HttpAuthMiddleware executes and sets request.state.user
# But trace is already started with anonymous
response = await call_next(request)

🔄 Steps to Reproduce

1. Deploy ContextForge v1.0.1 with OBSERVABILITY_ENABLED=true
2. Make any authenticated MCP tool call using a valid API token
3. Open Admin UI → Observability → Traces
4. All entries show anonymous in the User column regardless of which user made the call
5. Filtering by User Email returns no results for any valid email

✅ Expected Behavior

Traces show the authenticated user's email in the User column, enabling per-user
filtering and attribution in the observability dashboard.

❌ Actual Behavior

All traces show anonymous. The User Email filter is non-functional.

🩹 Suggested Fix

Option 1 — Move ObservabilityMiddleware registration before HttpAuthMiddleware:

Register ObservabilityMiddleware at a lower line number than HttpAuthMiddleware
in main.py so it runs after auth in LIFO order.

Option 2 — Extract user_email after call_next():

Move the user_email extraction to after call_next() completes, then update
the trace with the correct user:

response = await call_next(request)

# Auth has now run — request.state.user is set
if hasattr(request.state, "user") and hasattr(request.state.user, "email"):
    user_email = request.state.user.email
    self.service.update_trace(trace_id, user_email=user_email)

🖥️ Environment

• ContextForge version: v1.0.1
• Image: ghcr.io/ibm/mcp-context-forge:v1.0.1
• Deployment: Kubernetes via mcp-stack Helm chart
• OBSERVABILITY_ENABLED=true
• OTEL_ENABLE_OBSERVABILITY=true

📝 Additional Notes

This also affects OTEL traces shipped to Tempo/Jaeger — spans are started before
auth runs so they will also lack user attribution. The structured logger
(mcpgateway.services.structured_logger) correctly captures user_email after
auth and is unaffected by this bug.