Genesis LLO Support in ADOT SDK

[yiyuan-he]

What does this pull request do?

Adds support to handle LLO from third-party instrumentation SDKs in ADOT SDK.

The following SDKs are supported:

• Traceloop/Openllmetry
• OpenInference
• OpenLit

Note: OTel dependencies in ADOT SDK have been loosened as a short-term workaround to support the various conflicting dependency requirements of third-party instrumentation SDKs.

Test plan

Built this custom ADOT SDK into various sample apps and exported the span and logs data to the OTLP X-Ray and Logs endpoint, respectively, to validate the LLO extraction and transformation to Gen AI Events.

Configurations tested:

• LangChain + Traceloop/Openllmetry
• LangChan + OpenInference
• LangChain + OpenLit
• CrewAI + Traceloop/Openllmetry
• CrewAI + OpenInference
• CrewAI + OpenLit

Environment variable configuration:

λ  env OTEL_METRICS_EXPORTER=none \
       OTEL_TRACES_EXPORTER=otlp \
       OTEL_LOGS_EXPORTER=otlp \
       OTEL_PYTHON_DISTRO=aws_distro \
       OTEL_PYTHON_CONFIGURATOR=aws_configurator \
       OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
       OTEL_EXPORTER_OTLP_LOGS_HEADERS="x-aws-log-group=test,x-aws-log-stream=default" \
       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \
       OTEL_RESOURCE_ATTRIBUTES="service.name=langchain-app" \
       OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED="true" \
       OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="true" \
       OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="http,sqlalchemy,psycopg2,pymysql,sqlite3,aiopg,asyncpg,mysql_connector,botocore,boto3,urllib3,requests,starlette" \
       AGENT_OBSERVABILITY_ENABLED="true" \
       python app.py

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[yiyuan-he]

gen_ai.system is not always available in the span. This is a best attempt at still retrieving a relevant value.

[yiyuan-he]

Support for more third-party SDKs can be added by following this pattern.

I don't think we can go further with isolating these rules since there is no consistent and generic way to determine which SDK family is being instrumented. Also, many of these third-party SDKs may have overlapping rules for generic OTel attributes such as gen_ai.prompt.{n}.content and gen_ai.completion.{n}.content.

[yiyuan-he]

Changed order of pipeline initialization so we can pass the global logger provider instance to the span pipeline if AGENT_OBSERVABILITY_ENABLED flag is enabled.

[yiyuan-he]

LLO handling is gated behind this configuration. If false, then the ADOT SDK will have default behavior in span and logs pipelines.