From aaedab9bcd69363043aaf47602288f96faf5fcfe Mon Sep 17 00:00:00 2001 From: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Date: Mon, 27 Jan 2025 16:23:45 -0500 Subject: [PATCH 01/17] chore: update changelog for version 2.19.2 (#12088) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Nicole Cybul --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a29299f8987..6c72665472f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,21 @@ Changelogs for versions not listed here can be found at https://github.com/DataD --- +## 2.19.2 +### Bug Fixes + +- Tracing + - celery: Fixes an issue where `celery.apply` spans from Celery prerun got closed too soon leading to span tags being missing. + - openai: Fixes a patching issue where asynchronous moderation endpoint calls resulted in coroutine scheduling errors. + - openai: Ensures the OpenAI integration is compatible with Python versions 3.12 and 3.13. + - vertexai: Resolves an issue with `chat.send_message()` where the content keyword argument was not parsed correctly. +- LLM Observability + - This fix resolves an issue where annotating a span with non latin-1 (but valid utf-8) input/output values resulted in encoding errors. +- Lib-Injection + - Fixes incorrect telemetry data payload format. + +--- + ## 2.19.1 ### Bug Fixes From e3045a19a20e0348f040aa589ef3c2517db74618 Mon Sep 17 00:00:00 2001 From: Nick Ripley Date: Mon, 27 Jan 2025 16:30:29 -0500 Subject: [PATCH 02/17] fix(profiling): fix SystemError when collecting memory profiler events (#12075) We added locking to the memory profiler to address crashes. These locks are mostly "try" locks, meaning we bail out if we can't acquire them right away. This was done defensively to mitigate the possibility of deadlock until we fully understood why the locks are needed and could guarantee their correctness. But as a result of using try locks, the `iter_events` function in particular can fail if the memory profiler lock is contended when it tries to collect profiling events. The function then returns NULL, leading to SystemError exceptions because we don't set an error. Even if we set an error, returning NULL isn't the right thing to do. It'll basically mean we wait until the next profile iteration, still accumulating events in the same buffer, and try again to upload the events. So we're going to get multiple iteration's worth of events. The right thing to do is take the lock unconditionally in `iter_events`. We can allocate the new tracker outside the memory allocation profiler lock so that we don't need to worry about reentrancy/deadlock issues if we start profiling that allocation. Then, the only thing we do under the lock is swap out the global tracker, so it's safe to take the lock unconditionally. Fixes #11831 TODO - regression test? --- ddtrace/profiling/collector/_memalloc.c | 22 +++++++++++++------ ...loc-iter-events-null-780fd50bbebbf616.yaml | 4 ++++ 2 files changed, 19 insertions(+), 7 deletions(-) create mode 100644 releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml diff --git a/ddtrace/profiling/collector/_memalloc.c b/ddtrace/profiling/collector/_memalloc.c index 1f2b87e0433..1e1b9dbf52e 100644 --- a/ddtrace/profiling/collector/_memalloc.c +++ b/ddtrace/profiling/collector/_memalloc.c @@ -394,20 +394,28 @@ iterevents_new(PyTypeObject* type, PyObject* Py_UNUSED(args), PyObject* Py_UNUSE } IterEventsState* iestate = (IterEventsState*)type->tp_alloc(type, 0); - if (!iestate) + if (!iestate) { + PyErr_SetString(PyExc_RuntimeError, "failed to allocate IterEventsState"); return NULL; + } memalloc_assert_gil(); - /* reset the current traceback list */ - if (memlock_trylock(&g_memalloc_lock)) { - iestate->alloc_tracker = global_alloc_tracker; - global_alloc_tracker = alloc_tracker_new(); - memlock_unlock(&g_memalloc_lock); - } else { + /* Reset the current traceback list. Do this outside lock so we can track it, + * and avoid reentrancy/deadlock problems, if we start tracking the raw + * allocator domain */ + alloc_tracker_t* tracker = alloc_tracker_new(); + if (!tracker) { + PyErr_SetString(PyExc_RuntimeError, "failed to allocate new allocation tracker"); Py_TYPE(iestate)->tp_free(iestate); return NULL; } + + memlock_lock(&g_memalloc_lock); + iestate->alloc_tracker = global_alloc_tracker; + global_alloc_tracker = tracker; + memlock_unlock(&g_memalloc_lock); + iestate->seq_index = 0; PyObject* iter_and_count = PyTuple_New(3); diff --git a/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml b/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml new file mode 100644 index 00000000000..52a43cbd2a1 --- /dev/null +++ b/releasenotes/notes/profiling-memalloc-iter-events-null-780fd50bbebbf616.yaml @@ -0,0 +1,4 @@ +--- +fixes: + - | + profiling: fix SystemError from the memory profiler returning NULL when collecting events From 55767a78063ed17de0e954a6db4b9ff64b762f1d Mon Sep 17 00:00:00 2001 From: William Conti <58711692+wconti27@users.noreply.github.com> Date: Tue, 28 Jan 2025 08:44:03 -0500 Subject: [PATCH 03/17] chore(tracing): refactor web server integrations to use the core module (#12035) ## Motivation Refactors all web server integrations still using `tracer.trace` to instead use `core.context_with_data`. This is in preparation for supporting AWS API Gateway to ensure all web servers share the same code path. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/_trace/trace_handlers.py | 56 +++++++++- .../contrib/internal/aiohttp/middlewares.py | 105 ++++++++---------- ddtrace/contrib/internal/bottle/trace.py | 61 +++++----- .../contrib/internal/cherrypy/middleware.py | 53 +++++---- ddtrace/contrib/internal/falcon/middleware.py | 61 ++++------ ddtrace/contrib/internal/molten/patch.py | 52 ++++----- ddtrace/contrib/internal/pyramid/trace.py | 72 ++++++------ ddtrace/contrib/internal/sanic/patch.py | 61 +++++----- ...b.cherrypy.test_middleware.test_child.json | 1 + ...b.cherrypy.test_middleware.test_error.json | 1 + ...b.cherrypy.test_middleware.test_fatal.json | 1 + ...cherrypy.test_middleware.test_success.json | 1 + ...ver.test_multiple_requests_sanic_http.json | 2 + ...multiple_requests_sanic_http_pre_21.9.json | 2 + ...c.test_sanic_server.test_sanic_errors.json | 2 + ...nic_server.test_sanic_errors_pre_21.9.json | 2 + 16 files changed, 290 insertions(+), 243 deletions(-) diff --git a/ddtrace/_trace/trace_handlers.py b/ddtrace/_trace/trace_handlers.py index dab3f743146..3ebb28f33c8 100644 --- a/ddtrace/_trace/trace_handlers.py +++ b/ddtrace/_trace/trace_handlers.py @@ -109,11 +109,14 @@ def _get_parameters_for_new_span_directly_from_context(ctx: core.ExecutionContex def _start_span(ctx: core.ExecutionContext, call_trace: bool = True, **kwargs) -> "Span": span_kwargs = _get_parameters_for_new_span_directly_from_context(ctx) call_trace = ctx.get_item("call_trace", call_trace) - tracer = (ctx.get_item("middleware") or ctx["pin"]).tracer + tracer = ctx.get_item("tracer") or (ctx.get_item("middleware") or ctx["pin"]).tracer distributed_headers_config = ctx.get_item("distributed_headers_config") if distributed_headers_config: trace_utils.activate_distributed_headers( - tracer, int_config=distributed_headers_config, request_headers=ctx["distributed_headers"] + tracer, + int_config=distributed_headers_config, + request_headers=ctx["distributed_headers"], + override=ctx.get_item("distributed_headers_config_override"), ) distributed_context = ctx.get_item("distributed_context") if distributed_context and not call_trace: @@ -126,6 +129,42 @@ def _start_span(ctx: core.ExecutionContext, call_trace: bool = True, **kwargs) - return span +def _set_web_frameworks_tags(ctx, span, int_config): + span.set_tag_str(COMPONENT, int_config.integration_name) + span.set_tag_str(SPAN_KIND, SpanKind.SERVER) + span.set_tag(_SPAN_MEASURED_KEY) + + analytics_enabled = ctx.get_item("analytics_enabled") + analytics_sample_rate = ctx.get_item("analytics_sample_rate", True) + + # Configure trace search sample rate + if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: + span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, analytics_sample_rate) + + +def _on_web_framework_start_request(ctx, int_config): + request_span = ctx.get_item("req_span") + _set_web_frameworks_tags(ctx, request_span, int_config) + + +def _on_web_framework_finish_request( + span, int_config, method, url, status_code, query, req_headers, res_headers, route, finish +): + trace_utils.set_http_meta( + span=span, + integration_config=int_config, + method=method, + url=url, + status_code=status_code, + query=query, + request_headers=req_headers, + response_headers=res_headers, + route=route, + ) + if finish: + span.finish() + + def _on_traced_request_context_started_flask(ctx): current_span = ctx["pin"].tracer.current_span() if not ctx["pin"].enabled or not current_span: @@ -761,6 +800,10 @@ def listen(): core.on("azure.functions.request_call_modifier", _on_azure_functions_request_span_modifier) core.on("azure.functions.start_response", _on_azure_functions_start_response) + # web frameworks general handlers + core.on("web.request.start", _on_web_framework_start_request) + core.on("web.request.finish", _on_web_framework_finish_request) + core.on("test_visibility.enable", _on_test_visibility_enable) core.on("test_visibility.disable", _on_test_visibility_disable) core.on("test_visibility.is_enabled", _on_test_visibility_is_enabled, "is_enabled") @@ -769,6 +812,14 @@ def listen(): core.on("rq.queue.enqueue_job", _propagate_context) for context_name in ( + # web frameworks + "aiohttp.request", + "bottle.request", + "cherrypy.request", + "falcon.request", + "molten.request", + "pyramid.request", + "sanic.request", "flask.call", "flask.jsonify", "flask.render_template", @@ -779,6 +830,7 @@ def listen(): "django.template.render", "django.process_exception", "django.func.wrapped", + # non web frameworks "botocore.instrumented_api_call", "botocore.instrumented_lib_function", "botocore.patched_kinesis_api_call", diff --git a/ddtrace/contrib/internal/aiohttp/middlewares.py b/ddtrace/contrib/internal/aiohttp/middlewares.py index ddb2d35fbc6..63a60734d3b 100644 --- a/ddtrace/contrib/internal/aiohttp/middlewares.py +++ b/ddtrace/contrib/internal/aiohttp/middlewares.py @@ -2,15 +2,10 @@ from aiohttp.web_urldispatcher import SystemRoute from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils from ddtrace.contrib.asyncio import context_provider -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes from ddtrace.ext import http -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection @@ -35,47 +30,42 @@ async def attach_context(request): # application configs tracer = app[CONFIG_KEY]["tracer"] service = app[CONFIG_KEY]["service"] - distributed_tracing = app[CONFIG_KEY]["distributed_tracing_enabled"] - # Create a new context based on the propagated information. - trace_utils.activate_distributed_headers( - tracer, - int_config=config.aiohttp, - request_headers=request.headers, - override=distributed_tracing, - ) - - # trace the handler - request_span = tracer.trace( - schematize_url_operation("aiohttp.request", protocol="http", direction=SpanDirection.INBOUND), - service=service, - span_type=SpanTypes.WEB, - ) - request_span.set_tag(_SPAN_MEASURED_KEY) - - request_span.set_tag_str(COMPONENT, config.aiohttp.integration_name) - - # set span.kind tag equal to type of request - request_span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - # Configure trace search sample rate # DEV: aiohttp is special case maintains separate configuration from config api analytics_enabled = app[CONFIG_KEY]["analytics_enabled"] - if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: - request_span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, app[CONFIG_KEY].get("analytics_sample_rate", True)) - - # attach the context and the root span to the request; the Context - # may be freely used by the application code - request[REQUEST_CONTEXT_KEY] = request_span.context - request[REQUEST_SPAN_KEY] = request_span - request[REQUEST_CONFIG_KEY] = app[CONFIG_KEY] - try: - response = await handler(request) - if isinstance(response, web.StreamResponse): - request.task.add_done_callback(lambda _: finish_request_span(request, response)) - return response - except Exception: - request_span.set_traceback() - raise + # Create a new context based on the propagated information. + + with core.context_with_data( + "aiohttp.request", + span_name=schematize_url_operation("aiohttp.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, + service=service, + tags={}, + tracer=tracer, + distributed_headers=request.headers, + distributed_headers_config=config.aiohttp, + distributed_headers_config_override=app[CONFIG_KEY]["distributed_tracing_enabled"], + headers_case_sensitive=True, + analytics_enabled=analytics_enabled, + analytics_sample_rate=app[CONFIG_KEY].get("analytics_sample_rate", True), + ) as ctx: + req_span = ctx.span + + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.aiohttp)) + + # attach the context and the root span to the request; the Context + # may be freely used by the application code + request[REQUEST_CONTEXT_KEY] = req_span.context + request[REQUEST_SPAN_KEY] = req_span + request[REQUEST_CONFIG_KEY] = app[CONFIG_KEY] + try: + response = await handler(request) + if isinstance(response, web.StreamResponse): + request.task.add_done_callback(lambda _: finish_request_span(request, response)) + return response + except Exception: + req_span.set_traceback() + raise return attach_context @@ -122,19 +112,22 @@ def finish_request_span(request, response): # SystemRoute objects exist to throw HTTP errors and have no path route = aiohttp_route.resource.canonical - trace_utils.set_http_meta( - request_span, - config.aiohttp, - method=request.method, - url=str(request.url), # DEV: request.url is a yarl's URL object - status_code=response.status, - request_headers=request.headers, - response_headers=response.headers, - route=route, + core.dispatch( + "web.request.finish", + ( + request_span, + config.aiohttp, + request.method, + str(request.url), # DEV: request.url is a yarl's URL object + response.status, + None, # query arg = None + request.headers, + response.headers, + route, + True, + ), ) - request_span.finish() - async def on_prepare(request, response): """ diff --git a/ddtrace/contrib/internal/bottle/trace.py b/ddtrace/contrib/internal/bottle/trace.py index 3aabb4ccc81..58778a36ee1 100644 --- a/ddtrace/contrib/internal/bottle/trace.py +++ b/ddtrace/contrib/internal/bottle/trace.py @@ -5,13 +5,8 @@ import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection from ddtrace.internal.utils.formats import asbool @@ -42,24 +37,21 @@ def wrapped(*args, **kwargs): resource = "{} {}".format(request.method, route.rule) - trace_utils.activate_distributed_headers( - self.tracer, int_config=config.bottle, request_headers=request.headers - ) - - with self.tracer.trace( - schematize_url_operation("bottle.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "bottle.request", + span_name=schematize_url_operation("bottle.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=self.service, resource=resource, - span_type=SpanTypes.WEB, - ) as s: - s.set_tag_str(COMPONENT, config.bottle.integration_name) - - # set span.kind to the type of request being performed - s.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - s.set_tag(_SPAN_MEASURED_KEY) - # set analytics sample rate with global config enabled - s.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.bottle.get_analytics_sample_rate(use_global_config=True)) + tags={}, + tracer=self.tracer, + distributed_headers=request.headers, + distributed_headers_config=config.bottle, + headers_case_sensitive=True, + analytics_sample_rate=config.bottle.get_analytics_sample_rate(use_global_config=True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.bottle)) code = None result = None @@ -91,16 +83,21 @@ def wrapped(*args, **kwargs): method = request.method url = request.urlparts._replace(query="").geturl() full_route = "/".join([request.script_name.rstrip("/"), route.rule.lstrip("/")]) - trace_utils.set_http_meta( - s, - config.bottle, - method=method, - url=url, - status_code=response_code, - query=request.query_string, - request_headers=request.headers, - response_headers=response.headers, - route=full_route, + + core.dispatch( + "web.request.finish", + ( + req_span, + config.bottle, + method, + url, + response_code, + request.query_string, + request.headers, + response.headers, + full_route, + False, + ), ) return wrapped diff --git a/ddtrace/contrib/internal/cherrypy/middleware.py b/ddtrace/contrib/internal/cherrypy/middleware.py index 909a043175f..70f43059905 100644 --- a/ddtrace/contrib/internal/cherrypy/middleware.py +++ b/ddtrace/contrib/internal/cherrypy/middleware.py @@ -11,12 +11,10 @@ from ddtrace.constants import ERROR_MSG from ddtrace.constants import ERROR_STACK from ddtrace.constants import ERROR_TYPE -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes from ddtrace.internal import compat -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import SpanDirection from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -77,20 +75,23 @@ def _setup(self): cherrypy.request.hooks.attach("after_error_response", self._after_error_response, priority=5) def _on_start_resource(self): - trace_utils.activate_distributed_headers( - self._tracer, int_config=config.cherrypy, request_headers=cherrypy.request.headers - ) - - cherrypy.request._datadog_span = self._tracer.trace( - SPAN_NAME, - service=trace_utils.int_service(None, config.cherrypy, default="cherrypy"), + with core.context_with_data( + "cherrypy.request", + span_name=SPAN_NAME, span_type=SpanTypes.WEB, - ) + service=trace_utils.int_service(None, config.cherrypy, default="cherrypy"), + tags={}, + tracer=self._tracer, + distributed_headers=cherrypy.request.headers, + distributed_headers_config=config.cherrypy, + headers_case_sensitive=True, + ) as ctx: + req_span = ctx.span - cherrypy.request._datadog_span.set_tag_str(COMPONENT, config.cherrypy.integration_name) + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.cherrypy)) - # set span.kind to the type of request being performed - cherrypy.request._datadog_span.set_tag_str(SPAN_KIND, SpanKind.SERVER) + cherrypy.request._datadog_span = req_span def _after_error_response(self): span = getattr(cherrypy.request, "_datadog_span", None) @@ -135,18 +136,22 @@ def _close_span(self, span): url = compat.to_unicode(cherrypy.request.base + cherrypy.request.path_info) status_code, _, _ = valid_status(cherrypy.response.status) - trace_utils.set_http_meta( - span, - config.cherrypy, - method=cherrypy.request.method, - url=url, - status_code=status_code, - request_headers=cherrypy.request.headers, - response_headers=cherrypy.response.headers, + core.dispatch( + "web.request.finish", + ( + span, + config.cherrypy, + cherrypy.request.method, + url, + status_code, + None, + cherrypy.request.headers, + cherrypy.response.headers, + None, + True, + ), ) - span.finish() - # Clear our span just in case. cherrypy.request._datadog_span = None diff --git a/ddtrace/contrib/internal/falcon/middleware.py b/ddtrace/contrib/internal/falcon/middleware.py index b4ec5434777..513ce6cce39 100644 --- a/ddtrace/contrib/internal/falcon/middleware.py +++ b/ddtrace/contrib/internal/falcon/middleware.py @@ -1,14 +1,8 @@ import sys from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.ext import http as httpx -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.schema import SpanDirection from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -27,26 +21,27 @@ def __init__(self, tracer, service=None, distributed_tracing=None): def process_request(self, req, resp): # Falcon uppercases all header names. headers = dict((k.lower(), v) for k, v in req.headers.items()) - trace_utils.activate_distributed_headers(self.tracer, int_config=config.falcon, request_headers=headers) - span = self.tracer.trace( - schematize_url_operation("falcon.request", protocol="http", direction=SpanDirection.INBOUND), - service=self.service, + with core.context_with_data( + "falcon.request", + span_name=schematize_url_operation("falcon.request", protocol="http", direction=SpanDirection.INBOUND), span_type=SpanTypes.WEB, - ) - span.set_tag_str(COMPONENT, config.falcon.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - - # set analytics sample rate with global config enabled - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.falcon.get_analytics_sample_rate(use_global_config=True)) - - trace_utils.set_http_meta( - span, config.falcon, method=req.method, url=req.url, query=req.query_string, request_headers=req.headers - ) + service=self.service, + tags={}, + tracer=self.tracer, + distributed_headers=headers, + distributed_headers_config=config.falcon, + headers_case_sensitive=True, + analytics_sample_rate=config.falcon.get_analytics_sample_rate(use_global_config=True), + ) as ctx: + req_span = ctx.span + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.falcon)) + + core.dispatch( + "web.request.finish", + (req_span, config.falcon, req.method, req.url, None, req.query_string, req.headers, None, None, False), + ) def process_resource(self, req, resp, resource, params): span = self.tracer.current_span() @@ -69,8 +64,7 @@ def process_response(self, req, resp, resource, req_succeeded=None): if resource is None: status = "404" span.resource = "%s 404" % req.method - span.set_tag(httpx.STATUS_CODE, status) - span.finish() + core.dispatch("web.request.finish", (span, config.falcon, None, None, status, None, None, None, None, True)) return err_type = sys.exc_info()[0] @@ -87,20 +81,13 @@ def process_response(self, req, resp, resource, req_succeeded=None): route = req.root_path or "" + req.uri_template - trace_utils.set_http_meta( - span, - config.falcon, - status_code=status, - response_headers=resp._headers, - route=route, - ) - # Emit span hook for this response # DEV: Emit before closing so they can overwrite `span.resource` if they want config.falcon.hooks.emit("request", span, req, resp) - # Close the span - span.finish() + core.dispatch( + "web.request.finish", (span, config.falcon, None, None, status, None, None, resp._headers, route, True) + ) def _is_404(err_type): diff --git a/ddtrace/contrib/internal/molten/patch.py b/ddtrace/contrib/internal/molten/patch.py index 38fa949243c..dfec47eb17d 100644 --- a/ddtrace/contrib/internal/molten/patch.py +++ b/ddtrace/contrib/internal/molten/patch.py @@ -5,15 +5,11 @@ from wrapt import wrap_function_wrapper as _w from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils from ddtrace.contrib.internal.trace_utils import unwrap as _u -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes +from ddtrace.internal import core from ddtrace.internal.compat import urlencode -from ddtrace.internal.constants import COMPONENT from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation from ddtrace.internal.schema.span_attribute_schema import SpanDirection @@ -89,25 +85,21 @@ def patch_app_call(wrapped, instance, args, kwargs): request = molten.http.Request.from_environ(environ) resource = func_name(wrapped) - # request.headers is type Iterable[Tuple[str, str]] - trace_utils.activate_distributed_headers( - pin.tracer, int_config=config.molten, request_headers=dict(request.headers) - ) - - with pin.tracer.trace( - schematize_url_operation("molten.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "molten.request", + span_name=schematize_url_operation("molten.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=trace_utils.int_service(pin, config.molten), resource=resource, - span_type=SpanTypes.WEB, - ) as span: - span.set_tag_str(COMPONENT, config.molten.integration_name) - - # set span.kind tag equal to type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - # set analytics sample rate with global config enabled - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, config.molten.get_analytics_sample_rate(use_global_config=True)) + tags={}, + tracer=pin.tracer, + distributed_headers=dict(request.headers), # request.headers is type Iterable[Tuple[str, str]] + distributed_headers_config=config.molten, + headers_case_sensitive=True, + analytics_sample_rate=config.molten.get_analytics_sample_rate(use_global_config=True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.molten)) @wrapt.function_wrapper def _w_start_response(wrapped, instance, args, kwargs): @@ -125,11 +117,13 @@ def _w_start_response(wrapped, instance, args, kwargs): except ValueError: pass - if not span.get_tag(MOLTEN_ROUTE): + if not req_span.get_tag(MOLTEN_ROUTE): # if route never resolve, update root resource - span.resource = "{} {}".format(request.method, code) + req_span.resource = "{} {}".format(request.method, code) - trace_utils.set_http_meta(span, config.molten, status_code=code) + core.dispatch( + "web.request.finish", (req_span, config.molten, None, None, code, None, None, None, None, False) + ) return wrapped(*args, **kwargs) @@ -143,11 +137,13 @@ def _w_start_response(wrapped, instance, args, kwargs): request.path, ) query = urlencode(dict(request.params)) - trace_utils.set_http_meta( - span, config.molten, method=request.method, url=url, query=query, request_headers=request.headers + + core.dispatch( + "web.request.finish", + (req_span, config.molten, request.method, url, None, query, request.headers, None, None, False), ) - span.set_tag_str("molten.version", molten.__version__) + req_span.set_tag_str("molten.version", molten.__version__) return wrapped(environ, start_response, **kwargs) diff --git a/ddtrace/contrib/internal/pyramid/trace.py b/ddtrace/contrib/internal/pyramid/trace.py index 9942c673d4e..9b221bd3f09 100644 --- a/ddtrace/contrib/internal/pyramid/trace.py +++ b/ddtrace/contrib/internal/pyramid/trace.py @@ -6,12 +6,8 @@ # project import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import _SPAN_MEASURED_KEY -from ddtrace.constants import SPAN_KIND -from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes +from ddtrace.internal import core from ddtrace.internal.constants import COMPONENT from ddtrace.internal.logger import get_logger from ddtrace.internal.schema import schematize_service_name @@ -67,29 +63,30 @@ def trace_tween_factory(handler, registry): service = settings.get(SETTINGS_SERVICE) or schematize_service_name("pyramid") tracer = settings.get(SETTINGS_TRACER) or ddtrace.tracer enabled = asbool(settings.get(SETTINGS_TRACE_ENABLED, tracer.enabled)) - distributed_tracing = asbool(settings.get(SETTINGS_DISTRIBUTED_TRACING, True)) + + # ensure distributed tracing within pyramid settings matches config + config.pyramid.distributed_tracing_enabled = asbool(settings.get(SETTINGS_DISTRIBUTED_TRACING, True)) if enabled: # make a request tracing function def trace_tween(request): - trace_utils.activate_distributed_headers( - tracer, int_config=config.pyramid, request_headers=request.headers, override=distributed_tracing - ) - - span_name = schematize_url_operation("pyramid.request", protocol="http", direction=SpanDirection.INBOUND) - with tracer.trace(span_name, service=service, resource="404", span_type=SpanTypes.WEB) as span: - span.set_tag_str(COMPONENT, config.pyramid.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - span.set_tag(_SPAN_MEASURED_KEY) - # Configure trace search sample rate + with core.context_with_data( + "pyramid.request", + span_name=schematize_url_operation("pyramid.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, + service=service, + resource="404", + tags={}, + tracer=tracer, + distributed_headers=request.headers, + distributed_headers_config=config.pyramid, + headers_case_sensitive=True, # DEV: pyramid is special case maintains separate configuration from config api - analytics_enabled = settings.get(SETTINGS_ANALYTICS_ENABLED) - - if (config._analytics_enabled and analytics_enabled is not False) or analytics_enabled is True: - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, settings.get(SETTINGS_ANALYTICS_SAMPLE_RATE, True)) + analytics_enabled=settings.get(SETTINGS_ANALYTICS_ENABLED), + analytics_sample_rate=settings.get(SETTINGS_ANALYTICS_SAMPLE_RATE, True), + ) as ctx, ctx.span as req_span: + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.pyramid)) setattr(request, DD_TRACER, tracer) # used to find the tracer in templates response = None @@ -110,8 +107,8 @@ def trace_tween(request): finally: # set request tags if request.matched_route: - span.resource = "{} {}".format(request.method, request.matched_route.name) - span.set_tag_str("pyramid.route.name", request.matched_route.name) + req_span.resource = "{} {}".format(request.method, request.matched_route.name) + req_span.set_tag_str("pyramid.route.name", request.matched_route.name) # set response tags if response: status = response.status_code @@ -119,17 +116,22 @@ def trace_tween(request): else: response_headers = None - trace_utils.set_http_meta( - span, - config.pyramid, - method=request.method, - url=request.path_url, - status_code=status, - query=request.query_string, - request_headers=request.headers, - response_headers=response_headers, - route=request.matched_route.pattern if request.matched_route else None, + core.dispatch( + "web.request.finish", + ( + req_span, + config.pyramid, + request.method, + request.path_url, + status, + request.query_string, + request.headers, + response_headers, + request.matched_route.pattern if request.matched_route else None, + False, + ), ) + return response return trace_tween diff --git a/ddtrace/contrib/internal/sanic/patch.py b/ddtrace/contrib/internal/sanic/patch.py index 8e53ed41dc8..1babf7d44a0 100644 --- a/ddtrace/contrib/internal/sanic/patch.py +++ b/ddtrace/contrib/internal/sanic/patch.py @@ -4,14 +4,10 @@ import wrapt from wrapt import wrap_function_wrapper as _w -import ddtrace from ddtrace import config -from ddtrace.constants import _ANALYTICS_SAMPLE_RATE_KEY -from ddtrace.constants import SPAN_KIND from ddtrace.contrib import trace_utils -from ddtrace.ext import SpanKind from ddtrace.ext import SpanTypes -from ddtrace.internal.constants import COMPONENT +from ddtrace.internal import core from ddtrace.internal.logger import get_logger from ddtrace.internal.schema import schematize_service_name from ddtrace.internal.schema import schematize_url_operation @@ -47,7 +43,10 @@ def update_span(span, response): # and so use 500 status_code = getattr(response, "status", 500) response_headers = getattr(response, "headers", None) - trace_utils.set_http_meta(span, config.sanic, status_code=status_code, response_headers=response_headers) + + core.dispatch( + "web.request.finish", (span, config.sanic, None, None, status_code, None, None, response_headers, None, False) + ) def _wrap_response_callback(span, callback): @@ -200,31 +199,35 @@ def _create_sanic_request_span(request): headers = request.headers.copy() - trace_utils.activate_distributed_headers(ddtrace.tracer, int_config=config.sanic, request_headers=headers) - - span = pin.tracer.trace( - schematize_url_operation("sanic.request", protocol="http", direction=SpanDirection.INBOUND), + with core.context_with_data( + "sanic.request", + span_name=schematize_url_operation("sanic.request", protocol="http", direction=SpanDirection.INBOUND), + span_type=SpanTypes.WEB, service=trace_utils.int_service(None, config.sanic), resource=resource, - span_type=SpanTypes.WEB, - ) - span.set_tag_str(COMPONENT, config.sanic.integration_name) - - # set span.kind to the type of operation being performed - span.set_tag_str(SPAN_KIND, SpanKind.SERVER) - - sample_rate = config.sanic.get_analytics_sample_rate(use_global_config=True) - if sample_rate is not None: - span.set_tag(_ANALYTICS_SAMPLE_RATE_KEY, sample_rate) - - method = request.method - url = "{scheme}://{host}{path}".format(scheme=request.scheme, host=request.host, path=request.path) - query_string = request.query_string - if isinstance(query_string, bytes): - query_string = query_string.decode() - trace_utils.set_http_meta(span, config.sanic, method=method, url=url, query=query_string, request_headers=headers) - - return span + tags={}, + pin=pin, + distributed_headers=headers, + distributed_headers_config=config.sanic, + headers_case_sensitive=True, + analytics_sample_rate=config.sanic.get_analytics_sample_rate(use_global_config=True), + ) as ctx: + req_span = ctx.span + + ctx.set_item("req_span", req_span) + core.dispatch("web.request.start", (ctx, config.sanic)) + + method = request.method + url = "{scheme}://{host}{path}".format(scheme=request.scheme, host=request.host, path=request.path) + query_string = request.query_string + if isinstance(query_string, bytes): + query_string = query_string.decode() + + core.dispatch( + "web.request.finish", (req_span, config.sanic, method, url, None, query_string, headers, None, None, False) + ) + + return req_span async def sanic_http_lifecycle_handle(request): diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json index 1887282b06f..933eda3e420 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_child.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json index 49759fee3d3..58fde524828 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_error.json @@ -25,6 +25,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json index 4153c8c2b18..a5a7c1ed14f 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_fatal.json @@ -25,6 +25,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json index d1eb9fc5bee..ce13988add0 100644 --- a/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json +++ b/tests/snapshots/tests.contrib.cherrypy.test_middleware.test_success.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json index 3803d07f360..66ca28d9606 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http.json @@ -23,6 +23,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -72,6 +73,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json index 4aac2721c02..aca376a974a 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_multiple_requests_sanic_http_pre_21.9.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -70,6 +71,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json index 435bbbc7b23..ab2aeaec920 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -58,6 +59,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, diff --git a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json index c03813a43d6..ceabecd1ae7 100644 --- a/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json +++ b/tests/snapshots/tests.contrib.sanic.test_sanic_server.test_sanic_errors_pre_21.9.json @@ -22,6 +22,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, @@ -54,6 +55,7 @@ "span.kind": "server" }, "metrics": { + "_dd.measured": 1, "_dd.top_level": 1, "_dd.tracer_kr": 1.0, "_sampling_priority_v1": 1, From 16d5280956bc259e77691536229824d3183a4181 Mon Sep 17 00:00:00 2001 From: Christophe Papazian <114495376+christophe-papazian@users.noreply.github.com> Date: Tue, 28 Jan 2025 15:48:22 +0100 Subject: [PATCH 04/17] ci(tracer): make serverless test unrequired (#12121) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- .gitlab/benchmarks.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab/benchmarks.yml b/.gitlab/benchmarks.yml index e922d315444..692a61ea93f 100644 --- a/.gitlab/benchmarks.yml +++ b/.gitlab/benchmarks.yml @@ -96,6 +96,7 @@ benchmark-serverless: tags: ["arch:amd64"] when: on_success needs: [ "benchmark-serverless-trigger" ] + allow_failure: true script: - git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/serverless-tools.git ./serverless-tools && cd ./serverless-tools - ./ci/check_trigger_status.sh From 4f0bcb5a59378fb6015230f9f9ad04199fc380b5 Mon Sep 17 00:00:00 2001 From: Christophe Papazian <114495376+christophe-papazian@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:28:20 +0100 Subject: [PATCH 05/17] chore(asm): clean libddwaf loading (#12102) Depending of the timing, libddwaf loading process could create triggers that would create loops in our instrumentation. From what I investigated: - if loaded too early, it could have bad interactions with gevent. - if loaded too late, it could be self instrumented by the tracer, creating a loop, as ctypes is using Popen and subprocess. while keeping the late loading introduced by 2 previous PRs - https://github.com/DataDog/dd-trace-py/pull/11987 - https://github.com/DataDog/dd-trace-py/pull/12013 this PR introduced a mechanism to bypass tracer instrumentation during ctypes loading, to avoid a possible loop that would prevent the WAF to be loaded. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/appsec/_ddwaf/ddwaf_types.py | 3 +++ ddtrace/appsec/_processor.py | 26 +++++++++++--------- ddtrace/contrib/internal/subprocess/patch.py | 6 +++++ ddtrace/settings/asm.py | 1 + 4 files changed, 24 insertions(+), 12 deletions(-) diff --git a/ddtrace/appsec/_ddwaf/ddwaf_types.py b/ddtrace/appsec/_ddwaf/ddwaf_types.py index e998b6048f6..5942a2fa184 100644 --- a/ddtrace/appsec/_ddwaf/ddwaf_types.py +++ b/ddtrace/appsec/_ddwaf/ddwaf_types.py @@ -22,9 +22,12 @@ if system() == "Linux": try: + asm_config._bypass_instrumentation_for_waf = True ctypes.CDLL(ctypes.util.find_library("rt"), mode=ctypes.RTLD_GLOBAL) except Exception: # nosec pass + finally: + asm_config._bypass_instrumentation_for_waf = False ARCHI = machine().lower() diff --git a/ddtrace/appsec/_processor.py b/ddtrace/appsec/_processor.py index 6102ba1ded2..83b65584a66 100644 --- a/ddtrace/appsec/_processor.py +++ b/ddtrace/appsec/_processor.py @@ -4,6 +4,7 @@ from json.decoder import JSONDecodeError import os import os.path +from typing import TYPE_CHECKING from typing import Any from typing import Dict from typing import List @@ -11,6 +12,11 @@ from typing import Set from typing import Tuple from typing import Union + + +if TYPE_CHECKING: + import ddtrace.appsec._ddwaf as ddwaf + import weakref from ddtrace._trace.processor import SpanProcessor @@ -167,14 +173,17 @@ def __post_init__(self) -> None: def delayed_init(self) -> None: try: if self._rules is not None and not hasattr(self, "_ddwaf"): - self._ddwaf = ddwaf.DDWaf( + from ddtrace.appsec._ddwaf import DDWaf # noqa: E402 + import ddtrace.appsec._metrics as metrics # noqa: E402 + + self.metrics = metrics + self._ddwaf = DDWaf( self._rules, self.obfuscation_parameter_key_regexp, self.obfuscation_parameter_value_regexp ) - _set_waf_init_metric(self._ddwaf.info) + self.metrics._set_waf_init_metric(self._ddwaf.info) except Exception: # Partial of DDAS-0005-00 log.warning("[DDAS-0005-00] WAF initialization failed") - raise self._update_required() def _update_required(self): @@ -193,7 +202,7 @@ def _update_rules(self, new_rules: Dict[str, Any]) -> bool: if asm_config._asm_static_rule_file is not None: return result result = self._ddwaf.update_rules(new_rules) - _set_waf_updates_metric(self._ddwaf.info) + self.metrics._set_waf_updates_metric(self._ddwaf.info) self._update_required() return result @@ -241,7 +250,7 @@ def waf_callable(custom_data=None, **kwargs): return self._waf_action(span._local_root or span, ctx, custom_data, **kwargs) _asm_request_context.set_waf_callback(waf_callable) - _asm_request_context.add_context_callback(_set_waf_request_metrics) + _asm_request_context.add_context_callback(self.metrics._set_waf_request_metrics) if headers is not None: _asm_request_context.set_waf_address(SPAN_DATA_NAMES.REQUEST_HEADERS_NO_COOKIES, headers) _asm_request_context.set_waf_address( @@ -436,10 +445,3 @@ def on_span_finish(self, span: Span) -> None: del self._span_to_waf_ctx[s] except Exception: # nosec B110 pass - - -# load waf at the end only to avoid possible circular imports with gevent -import ddtrace.appsec._ddwaf as ddwaf # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_init_metric # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_request_metrics # noqa: E402 -from ddtrace.appsec._metrics import _set_waf_updates_metric # noqa: E402 diff --git a/ddtrace/contrib/internal/subprocess/patch.py b/ddtrace/contrib/internal/subprocess/patch.py index 80d05b107bb..2d66edd4737 100644 --- a/ddtrace/contrib/internal/subprocess/patch.py +++ b/ddtrace/contrib/internal/subprocess/patch.py @@ -327,6 +327,8 @@ def unpatch() -> None: @trace_utils.with_traced_module def _traced_ossystem(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) if isinstance(args[0], str): for callback in _STR_CALLBACKS.values(): callback(args[0]) @@ -393,6 +395,8 @@ def _traced_osspawn(module, pin, wrapped, instance, args, kwargs): @trace_utils.with_traced_module def _traced_subprocess_init(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) cmd_args = args[0] if len(args) else kwargs["args"] if isinstance(cmd_args, (list, tuple, str)): if kwargs.get("shell", False): @@ -425,6 +429,8 @@ def _traced_subprocess_init(module, pin, wrapped, instance, args, kwargs): @trace_utils.with_traced_module def _traced_subprocess_wait(module, pin, wrapped, instance, args, kwargs): try: + if asm_config._bypass_instrumentation_for_waf: + return wrapped(*args, **kwargs) binary = core.get_item("subprocess_popen_binary") with pin.tracer.trace(COMMANDS.SPAN_NAME, resource=binary, span_type=SpanTypes.SYSTEM) as span: diff --git a/ddtrace/settings/asm.py b/ddtrace/settings/asm.py index 16937399501..fc2c9aa13fb 100644 --- a/ddtrace/settings/asm.py +++ b/ddtrace/settings/asm.py @@ -245,6 +245,7 @@ class ASMConfig(Env): default=r"^[+-]?((0b[01]+)|(0x[0-9A-Fa-f]+)|(\d+\.?\d*(?:[Ee][+-]?\d+)?|\.\d+(?:[Ee][+-]" + r"?\d+)?)|(X\'[0-9A-Fa-f]+\')|(B\'[01]+\'))$", ) + _bypass_instrumentation_for_waf = False def __init__(self): super().__init__() From c4448ea4e5580cfa55684ef108a5dafafe1f6f75 Mon Sep 17 00:00:00 2001 From: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Date: Tue, 28 Jan 2025 17:11:42 -0500 Subject: [PATCH 06/17] fix(llmobs): propagate distributed headers via signal dispatching, not config (#12089) This PR makes a change to our shared distributed tracing header injection method to dispatch signals/events instead of relying on the global config settings, which is only modifiable via env vars. This fixes distributed tracing for users that might rely solely on the `LLMObs.enable()` setup config. Programmatic `LLMObs.enable()/disable()` calls do not set the global `config._llmobs_enabled` boolean setting, which is only controlled by the `DD_LLMOBS_ENABLED` env var. This was problematic for users that relied on manual `LLMObs.enable()` setup (i.e. no env vars) because our distributed tracing injection code only checks the global config to inject llmobs parent IDs into request headers. If users manually enabled LLMObs without any env vars, then this would not be reflected in the global config value and thus LLMObs parent IDs would never be injected into the request headers. We can't check directly if LLMObs is enabled in the http injection module because: 1. This would require us to import significant product-specific LLMObs-code into the shared http injector helper module which would impact non-LLMObs users' app performance 2. Circular imports in LLMObs which imports http injector logic to use in its own helpers Instead of doing our check based on the global `config._llmobs_enabled` setting, we now send a tracing event to our shared product listeners, and register a corresponding `LLMObs._inject_llmobs_context()` hook to be called for all inject() calls if LLMObs is enabled (we check the LLMObs instance, not the global config setting value). ~One risk and why I don't like changing global config settings is because this then implies that it is no longer global or tied to an env var (I want to push for env var configuration where possible over manual overriding/enabling). If a global enabled config can be toggled indiscriminately then this could open a can of worms for enabling/disabling logic in our LLMObs service, which isn't really designed to be toggled on/off multiple times in the app's lifespan. However if some users cannot rely on env vars, then I don't see any other solution that does not couple tracer internal code with LLMObs code which is a no-option.~ (UPDATE: we avoided this issue by using signal dispatching) ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ddtrace/llmobs/_llmobs.py | 8 ++- ddtrace/propagation/http.py | 7 +-- ...nable-updates-config-45379a7a30e2e0e3.yaml | 5 ++ tests/llmobs/test_llmobs_service.py | 1 - tests/llmobs/test_propagation.py | 54 ++++++++++++------- tests/tracer/test_propagation.py | 24 --------- 6 files changed, 49 insertions(+), 50 deletions(-) create mode 100644 releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml diff --git a/ddtrace/llmobs/_llmobs.py b/ddtrace/llmobs/_llmobs.py index 768f4bdb292..bc5ddabcf00 100644 --- a/ddtrace/llmobs/_llmobs.py +++ b/ddtrace/llmobs/_llmobs.py @@ -285,6 +285,7 @@ def _stop_service(self) -> None: # Remove listener hooks for span events core.reset_listeners("trace.span_start", self._on_span_start) core.reset_listeners("trace.span_finish", self._on_span_finish) + core.reset_listeners("http.span_inject", self._inject_llmobs_context) forksafe.unregister(self._child_after_fork) @@ -369,6 +370,7 @@ def enable( # Register hooks for span events core.on("trace.span_start", cls._instance._on_span_start) core.on("trace.span_finish", cls._instance._on_span_finish) + core.on("http.span_inject", cls._instance._inject_llmobs_context) atexit.register(cls.disable) telemetry_writer.product_activated(TELEMETRY_APM_PRODUCT.LLMOBS, True) @@ -1162,6 +1164,11 @@ def submit_evaluation( cls._instance._llmobs_eval_metric_writer.enqueue(evaluation_metric) + def _inject_llmobs_context(self, span_context: Context, request_headers: Dict[str, str]) -> None: + if self.enabled is False: + return + _inject_llmobs_parent_id(span_context) + @classmethod def inject_distributed_headers(cls, request_headers: Dict[str, str], span: Optional[Span] = None) -> Dict[str, str]: """Injects the span's distributed context into the given request headers.""" @@ -1179,7 +1186,6 @@ def inject_distributed_headers(cls, request_headers: Dict[str, str], span: Optio if span is None: log.warning("No span provided and no currently active span found.") return request_headers - _inject_llmobs_parent_id(span.context) HTTPPropagator.inject(span.context, request_headers) return request_headers diff --git a/ddtrace/propagation/http.py b/ddtrace/propagation/http.py index fdaf97410ad..381acabb1bc 100644 --- a/ddtrace/propagation/http.py +++ b/ddtrace/propagation/http.py @@ -28,6 +28,7 @@ from ddtrace._trace.span import _get_64_lowest_order_bits_as_int from ddtrace._trace.span import _MetaDictType from ddtrace.appsec._constants import APPSEC +from ddtrace.internal.core import dispatch from ddtrace.settings.asm import config as asm_config from ..constants import AUTO_KEEP @@ -1052,6 +1053,7 @@ def parent_call(): :param dict headers: HTTP headers to extend with tracing attributes. :param Span non_active_span: Only to be used if injecting a non-active span. """ + dispatch("http.span_inject", (span_context, headers)) if not config._propagation_style_inject: return if non_active_span is not None and non_active_span.context is not span_context: @@ -1089,11 +1091,6 @@ def parent_call(): for key in span_context._baggage: headers[_HTTP_BAGGAGE_PREFIX + key] = span_context._baggage[key] - if config._llmobs_enabled: - from ddtrace.llmobs._utils import _inject_llmobs_parent_id - - _inject_llmobs_parent_id(span_context) - if PROPAGATION_STYLE_DATADOG in config._propagation_style_inject: _DatadogMultiHeader._inject(span_context, headers) if PROPAGATION_STYLE_B3_MULTI in config._propagation_style_inject: diff --git a/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml b/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml new file mode 100644 index 00000000000..4bf312f4680 --- /dev/null +++ b/releasenotes/notes/fix-llmobs-enable-updates-config-45379a7a30e2e0e3.yaml @@ -0,0 +1,5 @@ +--- +fixes: + - | + LLM Observability: Resolves an issue where explicitly only using ``LLMObs.enable()`` to configure LLM Observability + without environment variables would not automatically propagate distributed tracing headers. diff --git a/tests/llmobs/test_llmobs_service.py b/tests/llmobs/test_llmobs_service.py index de428999147..744fde885e9 100644 --- a/tests/llmobs/test_llmobs_service.py +++ b/tests/llmobs/test_llmobs_service.py @@ -60,7 +60,6 @@ def test_service_enable_proxy_default(): assert llmobs_instance.tracer == dummy_tracer assert isinstance(llmobs_instance._llmobs_span_writer._clients[0], LLMObsProxiedEventClient) assert run_llmobs_trace_filter(dummy_tracer) is not None - llmobs_service.disable() diff --git a/tests/llmobs/test_propagation.py b/tests/llmobs/test_propagation.py index e3ab9c80d66..27421e6b12f 100644 --- a/tests/llmobs/test_propagation.py +++ b/tests/llmobs/test_propagation.py @@ -57,20 +57,24 @@ def test_propagate_correct_llmobs_parent_id_simple(run_python_code_in_subprocess """ code = """ import json +import mock -from ddtrace import tracer -from ddtrace.ext import SpanTypes +from ddtrace.internal.utils.http import Response +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("LLMObs span", span_type=SpanTypes.LLM) as root_span: - with tracer.trace("Non-LLMObs span") as child_span: - headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} - HTTPPropagator.inject(child_span.context, headers) +with mock.patch( + "ddtrace.internal.writer.HTTPWriter._send_payload", return_value=Response(status=200, body="{}"), +): + LLMObs.enable(ml_app="test-app", api_key="", agentless_enabled=True) + with LLMObs.workflow("LLMObs span") as root_span: + with LLMObs._instance.tracer.trace("Non-LLMObs span") as child_span: + headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} + HTTPPropagator.inject(child_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" env["DD_TRACE_ENABLED"] = "0" stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) @@ -93,21 +97,33 @@ def test_propagate_llmobs_parent_id_complex(run_python_code_in_subprocess): """ code = """ import json +import mock -from ddtrace import tracer -from ddtrace.ext import SpanTypes +from ddtrace.internal.utils.http import Response +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("LLMObs span", span_type=SpanTypes.LLM) as root_span: - with tracer.trace("Non-LLMObs span") as child_span: - headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} - HTTPPropagator.inject(child_span.context, headers) +with mock.patch( + "ddtrace.internal.writer.HTTPWriter._send_payload", return_value=Response(status=200, body="{}"), +): + from ddtrace import auto # simulate ddtrace-run startup to ensure env var configs also propagate + with LLMObs.workflow("LLMObs span") as root_span: + with LLMObs._instance.tracer.trace("Non-LLMObs span") as child_span: + headers = {"_DD_LLMOBS_SPAN_ID": str(root_span.span_id)} + HTTPPropagator.inject(child_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" - env["DD_TRACE_ENABLED"] = "0" + env.update( + { + "DD_LLMOBS_ENABLED": "1", + "DD_TRACE_ENABLED": "0", + "DD_AGENTLESS_ENABLED": "1", + "DD_API_KEY": "", + "DD_LLMOBS_ML_APP": "test-app", + } + ) stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) assert stderr == b"", (stdout, stderr) @@ -124,7 +140,7 @@ def test_propagate_llmobs_parent_id_complex(run_python_code_in_subprocess): def test_no_llmobs_parent_id_propagated_if_no_llmobs_spans(run_python_code_in_subprocess): - """Test that the correct LLMObs parent ID (None) is extracted from the headers in a simple distributed scenario. + """Test that the correct LLMObs parent ID ('undefined') is extracted from headers in a simple distributed scenario. Service A (subprocess) has spans, but none are LLMObs spans. Service B (outside subprocess) has a LLMObs span. Service B's span should have no LLMObs parent ID as there are no LLMObs spans from service A. @@ -132,17 +148,17 @@ def test_no_llmobs_parent_id_propagated_if_no_llmobs_spans(run_python_code_in_su code = """ import json -from ddtrace import tracer +from ddtrace.llmobs import LLMObs from ddtrace.propagation.http import HTTPPropagator -with tracer.trace("Non-LLMObs span") as root_span: +LLMObs.enable(ml_app="ml-app", agentless_enabled=True, api_key="") +with LLMObs._instance.tracer.trace("Non-LLMObs span") as root_span: headers = {} HTTPPropagator.inject(root_span.context, headers) print(json.dumps(headers)) """ env = os.environ.copy() - env["DD_LLMOBS_ENABLED"] = "1" env["DD_TRACE_ENABLED"] = "0" stdout, stderr, status, _ = run_python_code_in_subprocess(code=code, env=env) assert status == 0, (stdout, stderr) diff --git a/tests/tracer/test_propagation.py b/tests/tracer/test_propagation.py index 533e4974250..b073cd72e3b 100644 --- a/tests/tracer/test_propagation.py +++ b/tests/tracer/test_propagation.py @@ -4,7 +4,6 @@ import os import pickle -import mock import pytest import ddtrace @@ -3387,29 +3386,6 @@ def test_DD_TRACE_PROPAGATION_STYLE_INJECT_overrides_DD_TRACE_PROPAGATION_STYLE( assert result == expected_headers -def test_llmobs_enabled_injects_llmobs_parent_id(): - with override_global_config(dict(_llmobs_enabled=True)): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_called_once_with(context) - - -def test_llmobs_disabled_does_not_inject_parent_id(): - with override_global_config(dict(_llmobs_enabled=False)): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_not_called() - - -def test_llmobs_parent_id_not_injected_by_default(): - with mock.patch("ddtrace.llmobs._utils._inject_llmobs_parent_id") as mock_llmobs_inject: - context = Context(trace_id=1, span_id=2) - HTTPPropagator.inject(context, {}) - mock_llmobs_inject.assert_not_called() - - @pytest.mark.parametrize( "span_context,expected_headers", [ From cb41f8e77aa71bfffbb63f7ee1a9808e659189d2 Mon Sep 17 00:00:00 2001 From: Munir Abdinur Date: Wed, 29 Jan 2025 08:39:44 -0500 Subject: [PATCH 07/17] feat(provider): expose context provider in ddtrace.trace (#12135) ddtrace v3.0 is set to remove `ddtrace.providers` module from the public API. However we should still allow users to use their own ContextProviders. This PR ensures the BaseContextProvider remains in the public API and can be used in `tracer.configure(...)`. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- .github/CODEOWNERS | 2 ++ ddtrace/provider.py | 5 +++-- ddtrace/trace/__init__.py | 2 ++ docs/advanced_usage.rst | 2 +- .../feat-expose-base-context-provider-530ebec2225f6c8d.yaml | 5 +++++ 5 files changed, 13 insertions(+), 3 deletions(-) create mode 100644 releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 84e6c873642..b1162da2348 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -166,12 +166,14 @@ ddtrace/internal/remoteconfig @DataDog/remote-config @DataDog/apm-core-pyt tests/internal/remoteconfig @DataDog/remote-config @DataDog/apm-core-python # API SDK +ddtrace/trace/ @DataDog/apm-sdk-api-python ddtrace/_trace/ @DataDog/apm-sdk-api-python ddtrace/opentelemetry/ @DataDog/apm-sdk-api-python ddtrace/internal/opentelemetry @DataDog/apm-sdk-api-python ddtrace/opentracer/ @DataDog/apm-sdk-api-python ddtrace/propagation/ @DataDog/apm-sdk-api-python ddtrace/filters.py @DataDog/apm-sdk-api-python +ddtrace/provider.py @DataDog/apm-sdk-api-python ddtrace/pin.py @DataDog/apm-sdk-api-python ddtrace/sampler.py @DataDog/apm-sdk-api-python ddtrace/sampling_rule.py @DataDog/apm-sdk-api-python diff --git a/ddtrace/provider.py b/ddtrace/provider.py index 4a275ece8c8..7b9867de01a 100644 --- a/ddtrace/provider.py +++ b/ddtrace/provider.py @@ -7,7 +7,8 @@ deprecate( - "The context provider interface is deprecated.", - message="The trace context is an internal interface and will no longer be supported.", + "The context provider interface is deprecated", + message="Import BaseContextProvider from `ddtrace.trace` instead.", category=DDTraceDeprecationWarning, + removal_version="3.0.0", ) diff --git a/ddtrace/trace/__init__.py b/ddtrace/trace/__init__.py index f709310d589..8473c0b99f0 100644 --- a/ddtrace/trace/__init__.py +++ b/ddtrace/trace/__init__.py @@ -1,6 +1,7 @@ from ddtrace._trace.context import Context from ddtrace._trace.filters import TraceFilter from ddtrace._trace.pin import Pin +from ddtrace._trace.provider import BaseContextProvider from ddtrace._trace.span import Span from ddtrace._trace.tracer import Tracer @@ -15,4 +16,5 @@ "Tracer", "Span", "tracer", + "BaseContextProvider", ] diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst index 05879acac37..392e1406fb7 100644 --- a/docs/advanced_usage.rst +++ b/docs/advanced_usage.rst @@ -221,7 +221,7 @@ context management. If there is a case where the default is insufficient then a custom context provider can be used. It must implement the -:class:`ddtrace.provider.BaseContextProvider` interface and can be configured +:class:`ddtrace.trace.BaseContextProvider` interface and can be configured with:: tracer.configure(context_provider=MyContextProvider) diff --git a/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml b/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml new file mode 100644 index 00000000000..010d2bf52bd --- /dev/null +++ b/releasenotes/notes/feat-expose-base-context-provider-530ebec2225f6c8d.yaml @@ -0,0 +1,5 @@ +--- +deprecations: + - | + tracing: Moves ``ddtrace.provider.BaseContextProvider`` to ``ddtrace.trace.BaseContextProvider``. + The ``ddtrace.provider`` module is deprecated and will be removed in v3.0.0. From b4add532b3e79240456a1a3d0da0956f1c87233e Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Wed, 29 Jan 2025 17:28:08 +0000 Subject: [PATCH 08/17] Make stackv2 sampler use pthreads --- .../profiling/stack_v2/src/sampler.cpp | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp index 7ad9ad692b2..c365c36e9dc 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp @@ -10,6 +10,47 @@ using namespace Datadog; + +// Helper class for spawning a std::thread with control over its default stack size +#ifdef __linux__ +#include +#include +template +pthread_t create_thread_with_stack(size_t stack_size, Function&& f, Args&&... args) { + pthread_attr_t attr; + pthread_attr_init(&attr); + if (stack_size > 0) { + pthread_attr_setstacksize(&attr, stack_size); + } + + // DAS: I think std::bind here is necessary? + pthread_t thread; + auto* thread_func = new auto( + std::bind(std::forward(f), std::forward(args)...) + ); + + int ret = pthread_create( + &thread, + &attr, + [](void* arg) -> void* { + auto* func = static_cast(arg); + (*func)(); + delete func; + return nullptr; + }, + thread_func + ); + + pthread_attr_destroy(&attr); + + if (ret != 0) { + delete thread_func; + throw std::runtime_error("Failed to create thread"); + } + return thread; +} +#endif + void Sampler::sampling_thread(const uint64_t seq_num) { @@ -143,8 +184,15 @@ Sampler::start() // Launch the sampling thread. // Thread lifetime is bounded by the value of the sequence number. When it is changed from the value the thread was // launched with, the thread will exit. +#ifdef __linux__ + // We might as well get the default stack size and use that + rlimit stack_sz = {}; + getrlimit(RLIMIT_STACK, &stack_sz); + create_thread_with_stack(stack_sz.rlim_cur, &Sampler::sampling_thread, this, ++thread_seq_num); // leak the thread +#else std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); t.detach(); +#endif } void From 706c1e4ff4ae5eccfab494476d6bede17e674c66 Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Thu, 30 Jan 2025 16:44:03 +0000 Subject: [PATCH 09/17] Update --- .../profiling/stack_v2/include/sampler.hpp | 6 +- .../profiling/stack_v2/src/sampler.cpp | 68 +++++++++++-------- .../profiling/stack_v2/src/stack_v2.cpp | 6 +- 3 files changed, 45 insertions(+), 35 deletions(-) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp b/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp index 7050b6fcaa4..2493b35ca25 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp @@ -25,9 +25,6 @@ class Sampler // Parameters uint64_t echion_frame_cache_size = g_default_echion_frame_cache_size; - // Helper function; implementation of the echion sampling thread - void sampling_thread(const uint64_t seq_num); - // This is a singleton, so no public constructor Sampler(); @@ -37,7 +34,7 @@ class Sampler public: // Singleton instance static Sampler& get(); - void start(); + bool start(); void stop(); void register_thread(uint64_t id, uint64_t native_id, const char* name); void unregister_thread(uint64_t id); @@ -46,6 +43,7 @@ class Sampler PyObject* _asyncio_scheduled_tasks, PyObject* _asyncio_eager_tasks); void link_tasks(PyObject* parent, PyObject* child); + void sampling_thread(const uint64_t seq_num); // The Python side dynamically adjusts the sampling rate based on overhead, so we need to be able to update our // own intervals accordingly. Rather than a preemptive measure, we assume the rate is ~fairly stable and just diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp index c365c36e9dc..53a4492c3b2 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp @@ -10,44 +10,47 @@ using namespace Datadog; - // Helper class for spawning a std::thread with control over its default stack size #ifdef __linux__ -#include #include -template -pthread_t create_thread_with_stack(size_t stack_size, Function&& f, Args&&... args) { +#include + +struct ThreadArgs +{ + Sampler* sampler; + uint64_t seq_num; +}; + +void* +call_sampling_thread(void* args) +{ + ThreadArgs* thread_args = static_cast(args); + Sampler* sampler = thread_args->sampler; + sampler->sampling_thread(thread_args->seq_num); + return nullptr; +} + +pthread_t +create_thread_with_stack(size_t stack_size, Sampler* sampler, uint64_t seq_num) +{ pthread_attr_t attr; - pthread_attr_init(&attr); + if (pthread_attr_init(&attr) != 0) { + return 0; + } if (stack_size > 0) { pthread_attr_setstacksize(&attr, stack_size); } - // DAS: I think std::bind here is necessary? - pthread_t thread; - auto* thread_func = new auto( - std::bind(std::forward(f), std::forward(args)...) - ); - - int ret = pthread_create( - &thread, - &attr, - [](void* arg) -> void* { - auto* func = static_cast(arg); - (*func)(); - delete func; - return nullptr; - }, - thread_func - ); + pthread_t thread_id; + ThreadArgs thread_args = { sampler, seq_num }; + int ret = pthread_create(&thread_id, &attr, call_sampling_thread, &thread_args); pthread_attr_destroy(&attr); if (ret != 0) { - delete thread_func; - throw std::runtime_error("Failed to create thread"); + return 0; } - return thread; + return thread_id; } #endif @@ -175,7 +178,7 @@ Sampler::unregister_thread(uint64_t id) thread_info_map.erase(id); } -void +bool Sampler::start() { static std::once_flag once; @@ -188,11 +191,18 @@ Sampler::start() // We might as well get the default stack size and use that rlimit stack_sz = {}; getrlimit(RLIMIT_STACK, &stack_sz); - create_thread_with_stack(stack_sz.rlim_cur, &Sampler::sampling_thread, this, ++thread_seq_num); // leak the thread + if (create_thread_with_stack(stack_sz.rlim_cur, this, ++thread_seq_num) == 0) { + return false; + } #else - std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); - t.detach(); + try { + std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); + t.detach(); + } catch (const std::exception& e) { + return false; + } #endif + return true; } void diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp index c56b5524bcd..2e562943333 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp @@ -21,8 +21,10 @@ _stack_v2_start(PyObject* self, PyObject* args, PyObject* kwargs) } Sampler::get().set_interval(min_interval_s); - Sampler::get().start(); - Py_RETURN_NONE; + if (Sampler::get().start()) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; } // Bypasses the old-style cast warning with an unchecked helper function From acbcbdb221c824fb3be430b2f12b52742498f7b4 Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Wed, 29 Jan 2025 17:28:08 +0000 Subject: [PATCH 10/17] Make stackv2 sampler use pthreads --- .../profiling/stack_v2/src/sampler.cpp | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp index 7ad9ad692b2..c365c36e9dc 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp @@ -10,6 +10,47 @@ using namespace Datadog; + +// Helper class for spawning a std::thread with control over its default stack size +#ifdef __linux__ +#include +#include +template +pthread_t create_thread_with_stack(size_t stack_size, Function&& f, Args&&... args) { + pthread_attr_t attr; + pthread_attr_init(&attr); + if (stack_size > 0) { + pthread_attr_setstacksize(&attr, stack_size); + } + + // DAS: I think std::bind here is necessary? + pthread_t thread; + auto* thread_func = new auto( + std::bind(std::forward(f), std::forward(args)...) + ); + + int ret = pthread_create( + &thread, + &attr, + [](void* arg) -> void* { + auto* func = static_cast(arg); + (*func)(); + delete func; + return nullptr; + }, + thread_func + ); + + pthread_attr_destroy(&attr); + + if (ret != 0) { + delete thread_func; + throw std::runtime_error("Failed to create thread"); + } + return thread; +} +#endif + void Sampler::sampling_thread(const uint64_t seq_num) { @@ -143,8 +184,15 @@ Sampler::start() // Launch the sampling thread. // Thread lifetime is bounded by the value of the sequence number. When it is changed from the value the thread was // launched with, the thread will exit. +#ifdef __linux__ + // We might as well get the default stack size and use that + rlimit stack_sz = {}; + getrlimit(RLIMIT_STACK, &stack_sz); + create_thread_with_stack(stack_sz.rlim_cur, &Sampler::sampling_thread, this, ++thread_seq_num); // leak the thread +#else std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); t.detach(); +#endif } void From c18b1f6d80e98570ca0d1f10dab3087e3157b28a Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Thu, 30 Jan 2025 16:44:03 +0000 Subject: [PATCH 11/17] Update --- .../profiling/stack_v2/include/sampler.hpp | 6 +- .../profiling/stack_v2/src/sampler.cpp | 68 +++++++++++-------- .../profiling/stack_v2/src/stack_v2.cpp | 6 +- 3 files changed, 45 insertions(+), 35 deletions(-) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp b/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp index 7050b6fcaa4..2493b35ca25 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/include/sampler.hpp @@ -25,9 +25,6 @@ class Sampler // Parameters uint64_t echion_frame_cache_size = g_default_echion_frame_cache_size; - // Helper function; implementation of the echion sampling thread - void sampling_thread(const uint64_t seq_num); - // This is a singleton, so no public constructor Sampler(); @@ -37,7 +34,7 @@ class Sampler public: // Singleton instance static Sampler& get(); - void start(); + bool start(); void stop(); void register_thread(uint64_t id, uint64_t native_id, const char* name); void unregister_thread(uint64_t id); @@ -46,6 +43,7 @@ class Sampler PyObject* _asyncio_scheduled_tasks, PyObject* _asyncio_eager_tasks); void link_tasks(PyObject* parent, PyObject* child); + void sampling_thread(const uint64_t seq_num); // The Python side dynamically adjusts the sampling rate based on overhead, so we need to be able to update our // own intervals accordingly. Rather than a preemptive measure, we assume the rate is ~fairly stable and just diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp index c365c36e9dc..53a4492c3b2 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp @@ -10,44 +10,47 @@ using namespace Datadog; - // Helper class for spawning a std::thread with control over its default stack size #ifdef __linux__ -#include #include -template -pthread_t create_thread_with_stack(size_t stack_size, Function&& f, Args&&... args) { +#include + +struct ThreadArgs +{ + Sampler* sampler; + uint64_t seq_num; +}; + +void* +call_sampling_thread(void* args) +{ + ThreadArgs* thread_args = static_cast(args); + Sampler* sampler = thread_args->sampler; + sampler->sampling_thread(thread_args->seq_num); + return nullptr; +} + +pthread_t +create_thread_with_stack(size_t stack_size, Sampler* sampler, uint64_t seq_num) +{ pthread_attr_t attr; - pthread_attr_init(&attr); + if (pthread_attr_init(&attr) != 0) { + return 0; + } if (stack_size > 0) { pthread_attr_setstacksize(&attr, stack_size); } - // DAS: I think std::bind here is necessary? - pthread_t thread; - auto* thread_func = new auto( - std::bind(std::forward(f), std::forward(args)...) - ); - - int ret = pthread_create( - &thread, - &attr, - [](void* arg) -> void* { - auto* func = static_cast(arg); - (*func)(); - delete func; - return nullptr; - }, - thread_func - ); + pthread_t thread_id; + ThreadArgs thread_args = { sampler, seq_num }; + int ret = pthread_create(&thread_id, &attr, call_sampling_thread, &thread_args); pthread_attr_destroy(&attr); if (ret != 0) { - delete thread_func; - throw std::runtime_error("Failed to create thread"); + return 0; } - return thread; + return thread_id; } #endif @@ -175,7 +178,7 @@ Sampler::unregister_thread(uint64_t id) thread_info_map.erase(id); } -void +bool Sampler::start() { static std::once_flag once; @@ -188,11 +191,18 @@ Sampler::start() // We might as well get the default stack size and use that rlimit stack_sz = {}; getrlimit(RLIMIT_STACK, &stack_sz); - create_thread_with_stack(stack_sz.rlim_cur, &Sampler::sampling_thread, this, ++thread_seq_num); // leak the thread + if (create_thread_with_stack(stack_sz.rlim_cur, this, ++thread_seq_num) == 0) { + return false; + } #else - std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); - t.detach(); + try { + std::thread t(&Sampler::sampling_thread, this, ++thread_seq_num); + t.detach(); + } catch (const std::exception& e) { + return false; + } #endif + return true; } void diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp index c56b5524bcd..2e562943333 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/stack_v2.cpp @@ -21,8 +21,10 @@ _stack_v2_start(PyObject* self, PyObject* args, PyObject* kwargs) } Sampler::get().set_interval(min_interval_s); - Sampler::get().start(); - Py_RETURN_NONE; + if (Sampler::get().start()) { + Py_RETURN_TRUE; + } + Py_RETURN_FALSE; } // Bypasses the old-style cast warning with an unchecked helper function From c412655b2bce7ffc4c8ed59383e441869bfbb9fd Mon Sep 17 00:00:00 2001 From: Taegyun Kim Date: Thu, 30 Jan 2025 12:07:18 -0500 Subject: [PATCH 12/17] test on musl --- tests/smoke_test.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/tests/smoke_test.py b/tests/smoke_test.py index 56a2ab3a9c0..0281a15a58d 100644 --- a/tests/smoke_test.py +++ b/tests/smoke_test.py @@ -93,9 +93,6 @@ def emit(self, record): or (platform.system() == "Darwin" and platform.machine() == "x86_64") # echion only works with 3.8+ or sys.version_info < (3, 8, 0) - # echion crashes on musl linux with Python 3.12 for both x86_64 and - # aarch64 - or (platform.system() == "Linux" and sys.version_info[:2] == (3, 12) and platform.libc_ver()[0] != "glibc") ): orig_env = os.environ.copy() copied_env = copy.deepcopy(orig_env) From bd4d0be77044aeb93c17db01d08327bdb7a14e24 Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Thu, 30 Jan 2025 21:40:17 +0000 Subject: [PATCH 13/17] Fix --- .../datadog/profiling/stack_v2/src/sampler.cpp | 12 +++++++----- src/native/Cargo.lock | 4 ++-- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp index 53a4492c3b2..656e9c05c22 100644 --- a/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp +++ b/ddtrace/internal/datadog/profiling/stack_v2/src/sampler.cpp @@ -24,9 +24,10 @@ struct ThreadArgs void* call_sampling_thread(void* args) { - ThreadArgs* thread_args = static_cast(args); - Sampler* sampler = thread_args->sampler; - sampler->sampling_thread(thread_args->seq_num); + ThreadArgs thread_args = *static_cast(args); + delete static_cast(args); // no longer needed, dynamic alloc + Sampler* sampler = thread_args.sampler; + sampler->sampling_thread(thread_args.seq_num); return nullptr; } @@ -42,12 +43,13 @@ create_thread_with_stack(size_t stack_size, Sampler* sampler, uint64_t seq_num) } pthread_t thread_id; - ThreadArgs thread_args = { sampler, seq_num }; - int ret = pthread_create(&thread_id, &attr, call_sampling_thread, &thread_args); + ThreadArgs* thread_args = new ThreadArgs{ sampler, seq_num }; + int ret = pthread_create(&thread_id, &attr, call_sampling_thread, thread_args); pthread_attr_destroy(&attr); if (ret != 0) { + delete thread_args; // usually deleted in the thread, but need to clean it up here return 0; } return thread_id; diff --git a/src/native/Cargo.lock b/src/native/Cargo.lock index bbf189de0c2..a37cf6f4254 100644 --- a/src/native/Cargo.lock +++ b/src/native/Cargo.lock @@ -1,6 +1,6 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. -version = 3 +version = 4 [[package]] name = "anyhow" @@ -35,7 +35,7 @@ dependencies = [ ] [[package]] -name = "ddtrace-core" +name = "ddtrace-native" version = "0.1.0" dependencies = [ "datadog-ddsketch", From 7008e0efe264e38f58bcf1a4825861bd9907687b Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Fri, 31 Jan 2025 18:36:55 +0000 Subject: [PATCH 14/17] Add release note --- .../notes/fix-stackv2-musl-stack-size-7c265de9939ce2ce.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 docs/releasenotes/notes/fix-stackv2-musl-stack-size-7c265de9939ce2ce.yaml diff --git a/docs/releasenotes/notes/fix-stackv2-musl-stack-size-7c265de9939ce2ce.yaml b/docs/releasenotes/notes/fix-stackv2-musl-stack-size-7c265de9939ce2ce.yaml new file mode 100644 index 00000000000..cb226c73c34 --- /dev/null +++ b/docs/releasenotes/notes/fix-stackv2-musl-stack-size-7c265de9939ce2ce.yaml @@ -0,0 +1,6 @@ +--- +fixes: + - | + Fixes an issue where Profiling native threads would respect the musl libc + default stack size, which could cause stack overflows in certain + configurations. From 11134cfefd51098503a35783b74a33549589f7be Mon Sep 17 00:00:00 2001 From: sanchda <838104+sanchda@users.noreply.github.com> Date: Tue, 4 Feb 2025 18:01:22 +0000 Subject: [PATCH 15/17] fmt --- tests/smoke_test.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/smoke_test.py b/tests/smoke_test.py index a77bc463306..0281a15a58d 100644 --- a/tests/smoke_test.py +++ b/tests/smoke_test.py @@ -93,7 +93,6 @@ def emit(self, record): or (platform.system() == "Darwin" and platform.machine() == "x86_64") # echion only works with 3.8+ or sys.version_info < (3, 8, 0) - ): orig_env = os.environ.copy() copied_env = copy.deepcopy(orig_env) From 3ac70254751d343d4a4a518c7fc8ad85ff0bdb5c Mon Sep 17 00:00:00 2001 From: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> Date: Tue, 4 Feb 2025 13:12:04 -0500 Subject: [PATCH 16/17] chore(openai): remove remaining v0 snapshot files (#12218) Removes unused snapshots from now-removed v0 tests. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- ...t_integration_service_name[None-None].json | 77 ------------------- ...est_integration_service_name[None-v0].json | 77 ------------------- ...est_integration_service_name[None-v1].json | 77 ------------------- ..._integration_service_name[mysvc-None].json | 77 ------------------- ...st_integration_service_name[mysvc-v0].json | 77 ------------------- ...st_integration_service_name[mysvc-v1].json | 77 ------------------- 6 files changed, 462 deletions(-) delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-None].json delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v0].json delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v1].json delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-None].json delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v0].json delete mode 100644 tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v1].json diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-None].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-None].json deleted file mode 100644 index 65eec00d960..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-None].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "ddtrace_subprocess_dir", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7100000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "8c92d3e850d9413593bf481d805039d1" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20673 - }, - "duration": 21745000, - "start": 1701268849462298000 - }, - { - "name": "requests.request", - "service": "openai", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.base_service": "ddtrace_subprocess_dir", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1 - }, - "duration": 2999000, - "start": 1701268849479960000 - }]] diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v0].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v0].json deleted file mode 100644 index d6417fb5667..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v0].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "ddtrace_subprocess_dir", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7200000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "675032183b244929ba8c3a0a1c0021e5" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20696 - }, - "duration": 20412000, - "start": 1701268850764763000 - }, - { - "name": "requests.request", - "service": "openai", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.base_service": "ddtrace_subprocess_dir", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1 - }, - "duration": 3134000, - "start": 1701268850780901000 - }]] diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v1].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v1].json deleted file mode 100644 index 979ea768ef5..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[None-v1].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "ddtrace_subprocess_dir", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7400000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "1f3499a720954236be60cf0fece4246c" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20714 - }, - "duration": 19970000, - "start": 1701268852029562000 - }, - { - "name": "http.client.request", - "service": "ddtrace_subprocess_dir", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.peer.service.source": "out.host", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "peer.service": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1 - }, - "duration": 2897000, - "start": 1701268852045569000 - }]] diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-None].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-None].json deleted file mode 100644 index a80c1218caf..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-None].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "mysvc", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7500000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "1244eea37568412fb5bdedf9c37ed48a" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20736 - }, - "duration": 19953000, - "start": 1701268853284736000 - }, - { - "name": "requests.request", - "service": "openai", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.base_service": "mysvc", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1 - }, - "duration": 2837000, - "start": 1701268853300833000 - }]] diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v0].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v0].json deleted file mode 100644 index f3f9c57f768..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v0].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "mysvc", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7600000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "12b4a711854c44f681695957b545dcf5" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20750 - }, - "duration": 25352000, - "start": 1701268854568669000 - }, - { - "name": "requests.request", - "service": "openai", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.base_service": "mysvc", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1 - }, - "duration": 3922000, - "start": 1701268854588758000 - }]] diff --git a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v1].json b/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v1].json deleted file mode 100644 index 0696ae54454..00000000000 --- a/tests/snapshots/tests.contrib.openai.test_openai_v0.test_integration_service_name[mysvc-v1].json +++ /dev/null @@ -1,77 +0,0 @@ -[[ - { - "name": "openai.request", - "service": "mysvc", - "resource": "createCompletion", - "trace_id": 0, - "span_id": 1, - "parent_id": 0, - "type": "", - "error": 0, - "meta": { - "_dd.p.dm": "-0", - "_dd.p.tid": "65674d7700000000", - "component": "openai", - "language": "python", - "openai.api_base": "https://api.openai.com/v1", - "openai.api_type": "open_ai", - "openai.organization.name": "datadog-4", - "openai.request.client": "OpenAI", - "openai.request.endpoint": "/v1/completions", - "openai.request.max_tokens": "10", - "openai.request.method": "POST", - "openai.request.model": "ada", - "openai.request.n": "2", - "openai.request.prompt.0": "Hello world", - "openai.request.stop": ".", - "openai.request.temperature": "0.8", - "openai.response.choices.0.finish_reason": "length", - "openai.response.choices.0.text": ", relax!\u201d I said to my laptop", - "openai.response.choices.1.finish_reason": "stop", - "openai.response.choices.1.text": " (1", - "openai.response.created": "1681852797", - "openai.response.id": "cmpl-76n1xLvRKv3mfjx7hJ41UHrHy9ar6", - "openai.response.model": "ada", - "openai.user.api_key": "sk-...key>", - "runtime-id": "03e7664126ea4fe99e0aefec4efd003c" - }, - "metrics": { - "_dd.measured": 1, - "_dd.top_level": 1, - "_dd.tracer_kr": 1.0, - "_sampling_priority_v1": 1, - "openai.organization.ratelimit.requests.limit": 3000, - "openai.organization.ratelimit.requests.remaining": 2999, - "openai.organization.ratelimit.tokens.limit": 250000, - "openai.organization.ratelimit.tokens.remaining": 249979, - "process_id": 20772 - }, - "duration": 19966000, - "start": 1701268855885252000 - }, - { - "name": "http.client.request", - "service": "mysvc", - "resource": "POST /v1/completions", - "trace_id": 0, - "span_id": 2, - "parent_id": 1, - "type": "http", - "error": 0, - "meta": { - "_dd.peer.service.source": "out.host", - "component": "requests", - "http.method": "POST", - "http.status_code": "200", - "http.url": "https://api.openai.com/v1/completions", - "http.useragent": "OpenAI/v1 PythonBindings/0.27.2", - "out.host": "api.openai.com", - "peer.service": "api.openai.com", - "span.kind": "client" - }, - "metrics": { - "_dd.measured": 1 - }, - "duration": 2849000, - "start": 1701268855901267000 - }]] From 12458463905e0dae248e74ad9496c25ebd39f53f Mon Sep 17 00:00:00 2001 From: Munir Abdinur Date: Tue, 4 Feb 2025 13:56:01 -0500 Subject: [PATCH 17/17] chore(tracing): remove deprecated properties and parameters (#12199) - Introduces the following breaking changes to tracing components: - Removes deprecated parameters from ``Tracer.configure(...)`` method and removes the ``Tracer.sampler`` attribute. - Drops support for multiple tracer instances, ``ddtrace.trace.Tracer`` can not be reinitialized. - Removes the deprecated ``Span.sampled`` property - Drops support for configuring sampling rules using functions and regex in the ``ddtrace.tracer.sampler.rules[].choose_matcher(...)`` method and removes the ``timestamp_ns`` parameter from ``ddtrace.internal.rate_limiter.RateLimiter.is_allowed()``. - Drops support for configuring ``DD_TRACE_METHODS`` with the '[]' notation. Ensure DD_TRACE_METHODS use the ':' notation instead". - Removes the deprecated ``ddtracer`` parameter from ``ddtrace.opentracer.tracer.Tracer()``. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --- benchmarks/bm/utils.py | 2 +- benchmarks/rate_limiter/scenario.py | 4 +- ddtrace/_trace/sampling_rule.py | 14 +- ddtrace/_trace/span.py | 25 --- ddtrace/_trace/tracer.py | 174 ++---------------- ddtrace/internal/rate_limiter.py | 15 +- ddtrace/internal/tracemethods.py | 98 +--------- ddtrace/opentracer/tracer.py | 17 +- ...ve-tracing-attrs-3-0-5743fa668289d5bc.yaml | 15 ++ tests/integration/test_tracemethods.py | 45 +---- .../opentracer/core/test_dd_compatibility.py | 8 - tests/opentracer/core/test_tracer.py | 8 - tests/opentracer/utils.py | 2 +- tests/tracer/test_sampler.py | 89 +-------- 14 files changed, 53 insertions(+), 463 deletions(-) create mode 100644 releasenotes/notes/remove-tracing-attrs-3-0-5743fa668289d5bc.yaml diff --git a/benchmarks/bm/utils.py b/benchmarks/bm/utils.py index dd7b4991c57..13e99e8be74 100644 --- a/benchmarks/bm/utils.py +++ b/benchmarks/bm/utils.py @@ -65,7 +65,7 @@ def process_trace(self, trace): def drop_traces(tracer): - tracer.configure(settings={"FILTERS": [_DropTraces()]}) + tracer.configure(trace_processors=[_DropTraces()]) def drop_telemetry_events(): diff --git a/benchmarks/rate_limiter/scenario.py b/benchmarks/rate_limiter/scenario.py index 5210647ef89..3388af1cfb8 100644 --- a/benchmarks/rate_limiter/scenario.py +++ b/benchmarks/rate_limiter/scenario.py @@ -23,8 +23,8 @@ def _(loops): windows = [start + (i * self.time_window) for i in range(self.num_windows)] per_window = math.floor(loops / self.num_windows) - for window in windows: + for _ in windows: for _ in range(per_window): - rate_limiter.is_allowed(window) + rate_limiter.is_allowed() yield _ diff --git a/ddtrace/_trace/sampling_rule.py b/ddtrace/_trace/sampling_rule.py index 532a0b71f51..482a95d403a 100644 --- a/ddtrace/_trace/sampling_rule.py +++ b/ddtrace/_trace/sampling_rule.py @@ -8,8 +8,6 @@ from ddtrace.internal.glob_matching import GlobMatcher from ddtrace.internal.logger import get_logger from ddtrace.internal.utils.cache import cachedmethod -from ddtrace.internal.utils.deprecations import DDTraceDeprecationWarning -from ddtrace.vendor.debtcollector import deprecate if TYPE_CHECKING: # pragma: no cover @@ -210,14 +208,12 @@ def choose_matcher(self, prop): # We currently support the ability to pass in a function, a regular expression, or a string # If a string is passed in we create a GlobMatcher to handle the matching if callable(prop) or isinstance(prop, pattern_type): - # deprecated: passing a function or a regular expression' - deprecate( - "Using methods or regular expressions for SamplingRule matching is deprecated. ", - message="Please move to passing in a string for Glob matching.", - removal_version="3.0.0", - category=DDTraceDeprecationWarning, + log.error( + "Using methods or regular expressions for SamplingRule matching is not supported: %s ." + "Please move to passing in a string for Glob matching.", + str(prop), ) - return prop + return "None" # Name and Resource will never be None, but service can be, since we str() # whatever we pass into the GlobMatcher, we can just use its matching elif prop is None: diff --git a/ddtrace/_trace/span.py b/ddtrace/_trace/span.py index 446239a8091..c6eb4d4b72a 100644 --- a/ddtrace/_trace/span.py +++ b/ddtrace/_trace/span.py @@ -52,8 +52,6 @@ from ddtrace.internal.logger import get_logger from ddtrace.internal.sampling import SamplingMechanism from ddtrace.internal.sampling import set_sampling_decision_maker -from ddtrace.internal.utils.deprecations import DDTraceDeprecationWarning -from ddtrace.vendor.debtcollector import deprecate _NUMERIC_TAGS = (_ANALYTICS_SAMPLE_RATE_KEY,) @@ -279,29 +277,6 @@ def duration(self) -> Optional[float]: def duration(self, value: float) -> None: self.duration_ns = int(value * 1e9) - @property - def sampled(self) -> Optional[bool]: - deprecate( - "span.sampled is deprecated and will be removed in a future version of the tracer.", - message="""span.sampled references the state of span.context.sampling_priority. - Please use span.context.sampling_priority instead to check if a span is sampled.""", - category=DDTraceDeprecationWarning, - ) - if self.context.sampling_priority is None: - # this maintains original span.sampled behavior, where all spans would start - # with span.sampled = True until sampling runs - return True - return self.context.sampling_priority > 0 - - @sampled.setter - def sampled(self, value: bool) -> None: - deprecate( - "span.sampled is deprecated and will be removed in a future version of the tracer.", - message="""span.sampled has a no-op setter. - Please use span.set_tag('manual.keep'/'manual.drop') to keep or drop spans.""", - category=DDTraceDeprecationWarning, - ) - def finish(self, finish_time: Optional[float] = None) -> None: """Mark the end time of the span and submit it to the tracer. If the span has already been finished don't do anything. diff --git a/ddtrace/_trace/tracer.py b/ddtrace/_trace/tracer.py index 87f312bb18c..7030ec823d6 100644 --- a/ddtrace/_trace/tracer.py +++ b/ddtrace/_trace/tracer.py @@ -24,6 +24,7 @@ from ddtrace._trace.processor import TraceProcessor from ddtrace._trace.processor import TraceSamplingProcessor from ddtrace._trace.processor import TraceTagsProcessor +from ddtrace._trace.provider import BaseContextProvider from ddtrace._trace.provider import DefaultContextProvider from ddtrace._trace.sampler import BasePrioritySampler from ddtrace._trace.sampler import BaseSampler @@ -200,7 +201,7 @@ def __init__( self, url: Optional[str] = None, dogstatsd_url: Optional[str] = None, - context_provider: Optional[DefaultContextProvider] = None, + context_provider: Optional[BaseContextProvider] = None, ) -> None: """ Create a new ``Tracer`` instance. A global tracer is already initialized @@ -328,28 +329,6 @@ def sample(self, span): else: log.error("No sampler available to sample span") - @property - def sampler(self): - deprecate( - "tracer.sampler is deprecated and will be removed.", - message="To manually sample call tracer.sample(span) instead.", - category=DDTraceDeprecationWarning, - ) - return self._sampler - - @sampler.setter - def sampler(self, value): - deprecate( - "Setting a custom sampler is deprecated and will be removed.", - message="""Please use DD_TRACE_SAMPLING_RULES to configure the sampler instead: - https://ddtrace.readthedocs.io/en/stable/configuration.html#DD_TRACE_SAMPLING_RULES""", - category=DDTraceDeprecationWarning, - ) - if asm_config._apm_opt_out: - log.warning("Cannot set a custom sampler with Standalone ASM mode") - return - self._sampler = value - def on_start_span(self, func: Callable) -> Callable: """Register a function to execute when a span start. @@ -441,21 +420,7 @@ def get_log_correlation_context(self, active: Optional[Union[Context, Span]] = N def configure( self, - enabled: Optional[bool] = None, - hostname: Optional[str] = None, - port: Optional[int] = None, - uds_path: Optional[str] = None, - https: Optional[bool] = None, - sampler: Optional[BaseSampler] = None, - context_provider: Optional[DefaultContextProvider] = None, - wrap_executor: Optional[Callable] = None, - priority_sampling: Optional[bool] = None, - settings: Optional[Dict[str, Any]] = None, - dogstatsd_url: Optional[str] = None, - writer: Optional[TraceWriter] = None, - partial_flush_enabled: Optional[bool] = None, - partial_flush_min_spans: Optional[int] = None, - api_version: Optional[str] = None, + context_provider: Optional[BaseContextProvider] = None, compute_stats_enabled: Optional[bool] = None, appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, @@ -472,58 +437,14 @@ def configure( :param bool appsec_standalone_enabled: When tracing is disabled ensures ASM support is still enabled. :param List[TraceProcessor] trace_processors: This parameter sets TraceProcessor (ex: TraceFilters). Trace processors are used to modify and filter traces based on certain criteria. - - :param bool enabled: If True, finished traces will be submitted to the API, else they'll be dropped. - This parameter is deprecated and will be removed. - :param str hostname: Hostname running the Trace Agent. This parameter is deprecated and will be removed. - :param int port: Port of the Trace Agent. This parameter is deprecated and will be removed. - :param str uds_path: The Unix Domain Socket path of the agent. This parameter is deprecated and will be removed. - :param bool https: Whether to use HTTPS or HTTP. This parameter is deprecated and will be removed. - :param object sampler: A custom Sampler instance, locally deciding to totally drop the trace or not. - This parameter is deprecated and will be removed. - :param object wrap_executor: callable that is used when a function is decorated with - ``Tracer.wrap()``. This is an advanced option that usually doesn't need to be changed - from the default value. This parameter is deprecated and will be removed. - :param priority_sampling: This parameter is deprecated and will be removed in a future version. - :param bool settings: This parameter is deprecated and will be removed. - :param str dogstatsd_url: URL for UDP or Unix socket connection to DogStatsD - This parameter is deprecated and will be removed. - :param TraceWriter writer: This parameter is deprecated and will be removed. - :param bool partial_flush_enabled: This parameter is deprecated and will be removed. - :param bool partial_flush_min_spans: This parameter is deprecated and will be removed. - :param str api_version: This parameter is deprecated and will be removed. - :param bool compute_stats_enabled: This parameter is deprecated and will be removed. """ - if settings is not None: - deprecate( - "Support for ``tracer.configure(...)`` with the settings parameter is deprecated", - message="Please use the trace_processors parameter instead of settings['FILTERS'].", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) - trace_processors = (trace_processors or []) + (settings.get("FILTERS") or []) - return self._configure( - enabled, - hostname, - port, - uds_path, - https, - sampler, - context_provider, - wrap_executor, - priority_sampling, - trace_processors, - dogstatsd_url, - writer, - partial_flush_enabled, - partial_flush_min_spans, - api_version, - compute_stats_enabled, - appsec_enabled, - iast_enabled, - appsec_standalone_enabled, - True, + context_provider=context_provider, + trace_processors=trace_processors, + compute_stats_enabled=compute_stats_enabled, + appsec_enabled=appsec_enabled, + iast_enabled=iast_enabled, + appsec_standalone_enabled=appsec_standalone_enabled, ) def _configure( @@ -534,7 +455,7 @@ def _configure( uds_path: Optional[str] = None, https: Optional[bool] = None, sampler: Optional[BaseSampler] = None, - context_provider: Optional[DefaultContextProvider] = None, + context_provider: Optional[BaseContextProvider] = None, wrap_executor: Optional[Callable] = None, priority_sampling: Optional[bool] = None, trace_processors: Optional[List[TraceProcessor]] = None, @@ -547,48 +468,18 @@ def _configure( appsec_enabled: Optional[bool] = None, iast_enabled: Optional[bool] = None, appsec_standalone_enabled: Optional[bool] = None, - log_deprecations: bool = False, ) -> None: if enabled is not None: self.enabled = enabled - if log_deprecations: - deprecate( - "Enabling/Disabling tracing after application start is deprecated", - message="Please use DD_TRACE_ENABLED instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) - - if priority_sampling is not None and log_deprecations: - deprecate( - "Disabling priority sampling is deprecated", - message="Calling `tracer.configure(priority_sampling=....) has no effect", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) if trace_processors is not None: self._user_trace_processors = trace_processors if partial_flush_enabled is not None: self._partial_flush_enabled = partial_flush_enabled - if log_deprecations: - deprecate( - "Configuring partial flushing after application start is deprecated", - message="Please use DD_TRACE_PARTIAL_FLUSH_ENABLED to enable/disable the partial flushing instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) if partial_flush_min_spans is not None: self._partial_flush_min_spans = partial_flush_min_spans - if log_deprecations: - deprecate( - "Configuring partial flushing after application start is deprecated", - message="Please use DD_TRACE_PARTIAL_FLUSH_MIN_SPANS to set the flushing threshold instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) if appsec_enabled is not None: asm_config._asm_enabled = appsec_enabled @@ -620,33 +511,11 @@ def _configure( if sampler is not None: self._sampler = sampler self._user_sampler = self._sampler - if log_deprecations: - deprecate( - "Configuring custom samplers is deprecated", - message="Please use DD_TRACE_SAMPLING_RULES to configure the sample rates instead", - category=DDTraceDeprecationWarning, - removal_version="3.0.0", - ) if dogstatsd_url is not None: - if log_deprecations: - deprecate( - "Configuring dogstatsd_url after application start is deprecated", - message="Please use DD_DOGSTATSD_URL instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) self._dogstatsd_url = dogstatsd_url if any(x is not None for x in [hostname, port, uds_path, https]): - if log_deprecations: - deprecate( - "Configuring tracer agent connection after application start is deprecated", - message="Please use DD_TRACE_AGENT_URL instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) - # If any of the parts of the URL have updated, merge them with # the previous writer values. prev_url_parsed = compat.parse.urlparse(self._agent_url) @@ -670,13 +539,6 @@ def _configure( new_url = None if compute_stats_enabled is not None: - if log_deprecations: - deprecate( - "Configuring tracer stats computation after application start is deprecated", - message="Please use DD_TRACE_STATS_COMPUTATION_ENABLED instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) self._compute_stats = compute_stats_enabled try: @@ -685,14 +547,6 @@ def _configure( # It's possible the writer never got started pass - if api_version is not None and log_deprecations: - deprecate( - "Configuring Tracer API version after application start is deprecated", - message="Please use DD_TRACE_API_VERSION instead.", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) - if writer is not None: self._writer = writer elif any(x is not None for x in [new_url, api_version, sampler, dogstatsd_url, appsec_enabled]): @@ -754,12 +608,6 @@ def _configure( if wrap_executor is not None: self._wrap_executor = wrap_executor - if log_deprecations: - deprecate( - "Support for tracer.configure(...) with the wrap_executor parameter is deprecated", - version="3.0.0", - category=DDTraceDeprecationWarning, - ) self._generate_diagnostic_logs() @@ -1344,7 +1192,7 @@ def _handle_sampler_update(self, cfg: Config) -> None: and self._user_sampler ): # if we get empty configs from rc for both sample rate and rules, we should revert to the user sampler - self.sampler = self._user_sampler + self._sampler = self._user_sampler return if cfg._get_source("_trace_sample_rate") != "remote_config" and self._user_sampler: diff --git a/ddtrace/internal/rate_limiter.py b/ddtrace/internal/rate_limiter.py index 0a97a6a7abc..9b514e5ff32 100644 --- a/ddtrace/internal/rate_limiter.py +++ b/ddtrace/internal/rate_limiter.py @@ -9,9 +9,6 @@ from typing import Callable # noqa:F401 from typing import Optional # noqa:F401 -from ddtrace.internal.utils.deprecations import DDTraceDeprecationWarning -from ddtrace.vendor.debtcollector import deprecate - class RateLimiter(object): """ @@ -57,26 +54,18 @@ def __init__(self, rate_limit: int, time_window: float = 1e9): self._lock = threading.Lock() - def is_allowed(self, timestamp_ns: Optional[int] = None) -> bool: + def is_allowed(self) -> bool: """ Check whether the current request is allowed or not This method will also reduce the number of available tokens by 1 - :param int timestamp_ns: timestamp in nanoseconds for the current request. :returns: Whether the current request is allowed or not :rtype: :obj:`bool` """ - if timestamp_ns is not None: - deprecate( - "The `timestamp_ns` parameter is deprecated and will be removed in a future version." - "Ratelimiter will use the current time.", - category=DDTraceDeprecationWarning, - ) - # rate limits are tested and mocked in pytest so we need to compute the timestamp here # (or move the unit tests to rust) - timestamp_ns = timestamp_ns or time.monotonic_ns() + timestamp_ns = time.monotonic_ns() allowed = self._is_allowed(timestamp_ns) # Update counts used to determine effective rate self._update_rate_counts(allowed, timestamp_ns) diff --git a/ddtrace/internal/tracemethods.py b/ddtrace/internal/tracemethods.py index 5328797c09f..456cca597e1 100644 --- a/ddtrace/internal/tracemethods.py +++ b/ddtrace/internal/tracemethods.py @@ -4,8 +4,6 @@ import wrapt from ddtrace.internal.logger import get_logger -from ddtrace.internal.utils.deprecations import DDTraceDeprecationWarning -from ddtrace.vendor.debtcollector import deprecate log = get_logger(__name__) @@ -65,102 +63,10 @@ def _parse_trace_methods(raw_dd_trace_methods: str) -> List[Tuple[str, str]]: return dd_trace_methods -def _parse_legacy_trace_methods(raw_dd_trace_methods: str) -> List[str]: - """ - Return a list of method names to trace based on the specification of - DD_TRACE_METHODS. - - Note that support for wildcard methods with [*] is not implemented. - - This square bracket notation will be deprecated in favor of the new ':' notation - TODO: This method can be deleted once the legacy syntax is officially deprecated - """ - if not raw_dd_trace_methods: - return [] - dd_trace_methods = [] - for qualified_methods in raw_dd_trace_methods.split(";"): - # Validate that methods are specified - if "[" not in qualified_methods or "]" not in qualified_methods: - log.warning( - ( - "Invalid DD_TRACE_METHODS: %s. " - "Methods must be specified in square brackets following the fully qualified module or class name." - ), - qualified_methods, - ) - return [] - - # Store the prefix of the qualified method name (eg. for "foo.bar.baz[qux,quux]", this is "foo.bar.baz") - qualified_method_prefix = qualified_methods.split("[")[0] - - if qualified_method_prefix == "__main__": - # __main__ cannot be used since the __main__ that exists now is not the same as the __main__ that the user - # application will have. __main__ when sitecustomize module is run is the builtin __main__. - log.warning( - "Invalid DD_TRACE_METHODS: %s. Methods cannot be traced on the __main__ module.", qualified_methods - ) - return [] - - # Get the class or module name of the method (eg. for "foo.bar.baz[qux,quux]", this is "baz[qux,quux]") - class_or_module_with_methods = qualified_methods.split(".")[-1] - - # Strip off the leading 'moduleOrClass[' and trailing ']' - methods = class_or_module_with_methods.split("[")[1] - methods = methods[:-1] - - # Add the methods to the list of methods to trace - for method in methods.split(","): - if not str.isidentifier(method): - log.warning( - "Invalid method name: %r. %s", - method, - ( - "You might have a trailing comma." - if method == "" - else "Method names must be valid Python identifiers." - ), - ) - return [] - dd_trace_methods.append("%s.%s" % (qualified_method_prefix, method)) - return dd_trace_methods - - def _install_trace_methods(raw_dd_trace_methods: str) -> None: """Install tracing on the given methods.""" - if "[" in raw_dd_trace_methods: - deprecate( - "Using DD_TRACE_METHODS with the '[]' notation is deprecated", - message="Please use DD_TRACE_METHODS with the new ':' notation instead", - removal_version="3.0.0", - category=DDTraceDeprecationWarning, - ) - # Using legacy syntax - for qualified_method in _parse_legacy_trace_methods(raw_dd_trace_methods): - # We don't know if the method is a class method or a module method, so we need to assume it's a module - # and if the import fails then go a level up and try again. - base_module_guess = ".".join(qualified_method.split(".")[:-1]) - method_name = qualified_method.split(".")[-1] - module = None - - while base_module_guess: - try: - module = __import__(base_module_guess) - except ImportError: - # Add the class to the method name - method_name = "%s.%s" % (base_module_guess.split(".")[-1], method_name) - base_module_guess = ".".join(base_module_guess.split(".")[:-1]) - else: - break - - if module is None: - log.warning("Could not import module for %r", qualified_method) - continue - - trace_method(base_module_guess, method_name) - else: - # Using updated syntax, no need to try to import - for module_name, method_name in _parse_trace_methods(raw_dd_trace_methods): - trace_method(module_name, method_name) + for module_name, method_name in _parse_trace_methods(raw_dd_trace_methods): + trace_method(module_name, method_name) def trace_method(module, method_name): diff --git a/ddtrace/opentracer/tracer.py b/ddtrace/opentracer/tracer.py index ca10cb8125a..65d1b95b314 100644 --- a/ddtrace/opentracer/tracer.py +++ b/ddtrace/opentracer/tracer.py @@ -18,7 +18,6 @@ from ddtrace.trace import Context as DatadogContext # noqa:F401 from ddtrace.trace import Span as DatadogSpan from ddtrace.trace import Tracer as DatadogTracer -from ddtrace.vendor.debtcollector import deprecate from ..internal.logger import get_logger from .propagation import HTTPPropagator @@ -55,7 +54,7 @@ def __init__( service_name: Optional[str] = None, config: Optional[Dict[str, Any]] = None, scope_manager: Optional[ScopeManager] = None, - dd_tracer: Optional[DatadogTracer] = None, + _dd_tracer: Optional[DatadogTracer] = None, ) -> None: """Initialize a new Datadog opentracer. @@ -70,9 +69,6 @@ def __init__( here: https://github.com/opentracing/opentracing-python#scope-managers. If ``None`` is provided, defaults to :class:`opentracing.scope_managers.ThreadLocalScopeManager`. - :param dd_tracer: (optional) the Datadog tracer for this tracer to use. This - parameter is deprecated and will be removed in v3.0.0. The - to the global tracer (``ddtrace.tracer``) should always be used. """ # Merge the given config with the default into a new dict self._config = DEFAULT_CONFIG.copy() @@ -100,14 +96,7 @@ def __init__( self._scope_manager = scope_manager or ThreadLocalScopeManager() dd_context_provider = get_context_provider_for_scope_manager(self._scope_manager) - if dd_tracer is not None: - deprecate( - "The ``dd_tracer`` parameter is deprecated", - message="The global tracer (``ddtrace.tracer``) will be used instead.", - removal_version="3.0.0", - ) - - self._dd_tracer = dd_tracer or ddtrace.tracer + self._dd_tracer = _dd_tracer or ddtrace.tracer self._dd_tracer.set_tags(self._config.get(keys.GLOBAL_TAGS)) # type: ignore[arg-type] trace_processors = None if keys.SETTINGS in self._config: @@ -121,7 +110,7 @@ def __init__( trace_processors=trace_processors, priority_sampling=self._config.get(keys.PRIORITY_SAMPLING), uds_path=self._config.get(keys.UDS_PATH), - context_provider=dd_context_provider, # type: ignore[arg-type] + context_provider=dd_context_provider, ) self._propagators = { Format.HTTP_HEADERS: HTTPPropagator, diff --git a/releasenotes/notes/remove-tracing-attrs-3-0-5743fa668289d5bc.yaml b/releasenotes/notes/remove-tracing-attrs-3-0-5743fa668289d5bc.yaml new file mode 100644 index 00000000000..35ee9378801 --- /dev/null +++ b/releasenotes/notes/remove-tracing-attrs-3-0-5743fa668289d5bc.yaml @@ -0,0 +1,15 @@ +--- +upgrade: + - | + tracer: Removes deprecated parameters from ``Tracer.configure(...)`` method and removes the ``Tracer.sampler`` attribute. + - | + tracing: Drops support for multiple tracer instances, ``ddtrace.trace.Tracer`` can not be reinitialized. + - | + span: Removes the deprecated ``Span.sampled`` property + - | + sampling: Drops support for configuring sampling rules using functions and regex in the ``ddtrace.tracer.sampler.rules[].choose_matcher(...)`` method + and removes the ``timestamp_ns`` parameter from ``ddtrace.internal.rate_limiter.RateLimiter.is_allowed()``. + - | + configurations: Drops support for configuring ``DD_TRACE_METHODS`` with the '[]' notation. Ensure DD_TRACE_METHODS use the ':' notation instead". + - | + opentracing: Removes the deprecated ``ddtracer`` parameter from ``ddtrace.opentracer.tracer.Tracer()``. \ No newline at end of file diff --git a/tests/integration/test_tracemethods.py b/tests/integration/test_tracemethods.py index 15129c56161..7353c12182a 100644 --- a/tests/integration/test_tracemethods.py +++ b/tests/integration/test_tracemethods.py @@ -27,14 +27,10 @@ "mod.mod2.mod3:Class.test_method,Class.test_method2", [("mod.mod2.mod3", "Class.test_method"), ("mod.mod2.mod3", "Class.test_method2")], ), - ("module[method1, method2]", []), ("module", []), ("module.", []), ("module.method", []), - ("module.method[m1,m2,]", []), ("module.method;module.method", []), - ("module.method[m1];module.method[m1,m2,]", []), - ("module.method[[m1]", []), ], ) def test_trace_methods_parse(dd_trace_methods: str, expected_output: List[Tuple[str, str]]): @@ -43,37 +39,6 @@ def test_trace_methods_parse(dd_trace_methods: str, expected_output: List[Tuple[ assert _parse_trace_methods(dd_trace_methods) == expected_output -def test_legacy_trace_methods_parse(): - from ddtrace.internal.tracemethods import _parse_legacy_trace_methods - - assert _parse_legacy_trace_methods("") == [] - assert _parse_legacy_trace_methods("module[method1]") == ["module.method1"] - assert _parse_legacy_trace_methods("module[method1,method2]") == ["module.method1", "module.method2"] - assert _parse_legacy_trace_methods("module[method1,method2];mod2[m1,m2]") == [ - "module.method1", - "module.method2", - "mod2.m1", - "mod2.m2", - ] - assert _parse_legacy_trace_methods("mod.submod[m1,m2,m3]") == ["mod.submod.m1", "mod.submod.m2", "mod.submod.m3"] - assert _parse_legacy_trace_methods("mod.submod.subsubmod[m1,m2]") == [ - "mod.submod.subsubmod.m1", - "mod.submod.subsubmod.m2", - ] - assert _parse_legacy_trace_methods("mod.mod2.mod3.Class[test_method,test_method2]") == [ - "mod.mod2.mod3.Class.test_method", - "mod.mod2.mod3.Class.test_method2", - ] - assert _parse_legacy_trace_methods("module[method1, method2]") == [] - assert _parse_legacy_trace_methods("module") == [] - assert _parse_legacy_trace_methods("module.") == [] - assert _parse_legacy_trace_methods("module.method") == [] - assert _parse_legacy_trace_methods("module.method[m1,m2,]") == [] - assert _parse_legacy_trace_methods("module.method;module.method") == [] - assert _parse_legacy_trace_methods("module.method[m1];module.method[m1,m2,]") == [] - assert _parse_legacy_trace_methods("module.method[[m1]") == [] - - def _test_method(): pass @@ -105,9 +70,9 @@ def test_method(self): ddtrace_run=True, env=dict( DD_TRACE_METHODS=( - "tests.integration.test_tracemethods[_test_method,_test_method2];" - "tests.integration.test_tracemethods._Class[test_method,test_method2];" - "tests.integration.test_tracemethods._Class.NestedClass[test_method]" + "tests.integration.test_tracemethods:_test_method,_test_method2;" + "tests.integration.test_tracemethods:_Class.test_method,_Class.test_method2;" + "tests.integration.test_tracemethods:_Class.NestedClass.test_method" ) ), ) @@ -139,8 +104,8 @@ async def _async_test_method2(): def test_ddtrace_run_trace_methods_async(ddtrace_run_python_code_in_subprocess): env = os.environ.copy() env["DD_TRACE_METHODS"] = ( - "tests.integration.test_tracemethods[_async_test_method,_async_test_method2];" - "tests.integration.test_tracemethods._Class[async_test_method]" + "tests.integration.test_tracemethods:_async_test_method,_async_test_method2;" + "tests.integration.test_tracemethods:_Class.async_test_method" ) tests_dir = os.path.dirname(os.path.dirname(__file__)) env["PYTHONPATH"] = os.pathsep.join([tests_dir, env.get("PYTHONPATH", "")]) diff --git a/tests/opentracer/core/test_dd_compatibility.py b/tests/opentracer/core/test_dd_compatibility.py index 4ba14b0618f..c68b5ca6d6c 100644 --- a/tests/opentracer/core/test_dd_compatibility.py +++ b/tests/opentracer/core/test_dd_compatibility.py @@ -15,14 +15,6 @@ def test_ottracer_uses_global_ddtracer(self): tracer = ddtrace.opentracer.Tracer() assert tracer._dd_tracer is ddtrace.tracer - def test_custom_ddtracer(self): - """A user should be able to specify their own Datadog tracer instance if - they wish. - """ - custom_dd_tracer = ddtrace.trace.Tracer() - tracer = ddtrace.opentracer.Tracer(dd_tracer=custom_dd_tracer) - assert tracer._dd_tracer is custom_dd_tracer - def test_ot_dd_global_tracers(self, global_tracer): """Ensure our test function opentracer_init() prep""" ot_tracer = global_tracer diff --git a/tests/opentracer/core/test_tracer.py b/tests/opentracer/core/test_tracer.py index a0a18ff0dd8..f5534c8f1b0 100644 --- a/tests/opentracer/core/test_tracer.py +++ b/tests/opentracer/core/test_tracer.py @@ -15,8 +15,6 @@ from ddtrace.opentracer.span_context import SpanContext from ddtrace.propagation.http import HTTP_HEADER_TRACE_ID from ddtrace.settings import ConfigException -from ddtrace.trace import Tracer as DDTracer -from tests.utils import override_global_config class TestTracerConfig(object): @@ -69,12 +67,6 @@ def test_invalid_config_key(self): assert ["enabeld", "setttings"] in str(ce_info) # codespell:ignore assert tracer is not None - def test_ddtrace_fallback_config(self): - """Ensure datadog configuration is used by default.""" - with override_global_config(dict(_tracing_enabled=False)): - tracer = Tracer(dd_tracer=DDTracer()) - assert tracer._dd_tracer.enabled is False - def test_global_tags(self): """Global tags should be passed from the opentracer to the tracer.""" config = { diff --git a/tests/opentracer/utils.py b/tests/opentracer/utils.py index 6a34052a385..85b84865ad8 100644 --- a/tests/opentracer/utils.py +++ b/tests/opentracer/utils.py @@ -7,5 +7,5 @@ def init_tracer(service_name, dd_tracer, scope_manager=None): It accepts a Datadog tracer that should be the same one used for testing. """ - ot_tracer = Tracer(service_name, dd_tracer=dd_tracer, scope_manager=scope_manager) + ot_tracer = Tracer(service_name, scope_manager=scope_manager, _dd_tracer=dd_tracer) return ot_tracer diff --git a/tests/tracer/test_sampler.py b/tests/tracer/test_sampler.py index 813dc1be439..22620d8184b 100644 --- a/tests/tracer/test_sampler.py +++ b/tests/tracer/test_sampler.py @@ -1,6 +1,5 @@ from __future__ import division -import re import unittest import mock @@ -250,7 +249,7 @@ def test_sampling_rule_init_defaults(): def test_sampling_rule_init(): - a_regex = re.compile(r"\.request$") + a_regex = "*request" a_string = "my-service" rule = SamplingRule( @@ -261,7 +260,7 @@ def test_sampling_rule_init(): assert rule.sample_rate == 0.0, "SamplingRule should store the rate it's initialized with" assert rule.service.pattern == a_string, "SamplingRule should store the service it's initialized with" - assert rule.name == a_regex, "SamplingRule should store the name regex it's initialized with" + assert rule.name.pattern == a_regex, "SamplingRule should store the name regex it's initialized with" @pytest.mark.parametrize( @@ -272,38 +271,13 @@ def test_sampling_rule_init(): (SamplingRule(sample_rate=0.0), SamplingRule(sample_rate=0.0), True), (SamplingRule(sample_rate=0.5), SamplingRule(sample_rate=1.0), False), (SamplingRule(sample_rate=1.0, service="my-svc"), SamplingRule(sample_rate=1.0, service="my-svc"), True), - ( - SamplingRule(sample_rate=1.0, service=re.compile("my-svc")), - SamplingRule(sample_rate=1.0, service=re.compile("my-svc")), - True, - ), (SamplingRule(sample_rate=1.0, service="my-svc"), SamplingRule(sample_rate=1.0, service="other-svc"), False), (SamplingRule(sample_rate=1.0, service="my-svc"), SamplingRule(sample_rate=0.5, service="my-svc"), False), - ( - SamplingRule(sample_rate=1.0, service=re.compile("my-svc")), - SamplingRule(sample_rate=0.5, service=re.compile("my-svc")), - False, - ), - ( - SamplingRule(sample_rate=1.0, service=re.compile("my-svc")), - SamplingRule(sample_rate=1.0, service=re.compile("other")), - False, - ), ( SamplingRule(sample_rate=1.0, name="span.name"), SamplingRule(sample_rate=1.0, name="span.name"), True, ), - ( - SamplingRule(sample_rate=1.0, name=re.compile("span.name")), - SamplingRule(sample_rate=1.0, name=re.compile("span.name")), - True, - ), - ( - SamplingRule(sample_rate=1.0, name=re.compile("span.name")), - SamplingRule(sample_rate=1.0, name=re.compile("span.other")), - False, - ), ( SamplingRule(sample_rate=1.0, name="span.name"), SamplingRule(sample_rate=0.5, name="span.name"), @@ -316,16 +290,6 @@ def test_sampling_rule_init(): SamplingRule(sample_rate=1.0, service="my-svc", name="span.name"), True, ), - ( - SamplingRule(sample_rate=1.0, service="my-svc", name=re.compile("span.name")), - SamplingRule(sample_rate=1.0, service="my-svc", name=re.compile("span.name")), - True, - ), - ( - SamplingRule(sample_rate=1.0, service=re.compile("my-svc"), name=re.compile("span.name")), - SamplingRule(sample_rate=1.0, service=re.compile("my-svc"), name=re.compile("span.name")), - True, - ), ( SamplingRule(sample_rate=1.0, service="my-svc", name="span.name"), SamplingRule(sample_rate=0.5, service="my-svc", name="span.name"), @@ -491,15 +455,6 @@ def test_sampling_rule_init_via_env(): ("test.span", None, False), ("test.span", "test.span", True), ("test.span", "test_span", False), - ("test.span", re.compile(r"^test\.span$"), True), - ("test_span", re.compile(r"^test.span$"), True), - ("test.span", re.compile(r"^test_span$"), False), - ("test.span", re.compile(r"test"), True), - ("test.span", re.compile(r"test\.span|another\.span"), True), - ("another.span", re.compile(r"test\.span|another\.span"), True), - ("test.span", lambda name: "span" in name, True), - ("test.span", lambda name: "span" not in name, False), - ("test.span", lambda name: 1 / 0, False), ] ], ) @@ -518,20 +473,8 @@ def test_sampling_rule_matches_name(span, rule, span_expected_to_match_rule): ("my-service", None, False), (None, "tests.tracer", True), ("tests.tracer", "my-service", False), - ("tests.tracer", re.compile(r"my-service"), False), - ("tests.tracer", lambda service: "service" in service, False), ("my-service", "my-service", True), ("my-service", "my_service", False), - ("my-service", re.compile(r"^my-"), True), - ("my_service", re.compile(r"^my[_-]"), True), - ("my-service", re.compile(r"^my_"), False), - ("my-service", re.compile(r"my-service"), True), - ("my-service", re.compile(r"my"), True), - ("my-service", re.compile(r"my-service|another-service"), True), - ("another-service", re.compile(r"my-service|another-service"), True), - ("my-service", lambda service: "service" in service, True), - ("my-service", lambda service: "service" not in service, False), - ("my-service", lambda service: 1 / 0, False), ] ], ) @@ -553,7 +496,7 @@ def test_sampling_rule_matches_service(span, rule, span_expected_to_match_rule): SamplingRule( sample_rate=1, name="test.span", - service=re.compile(r"^my-"), + service="my-*", ), True, ), @@ -567,7 +510,7 @@ def test_sampling_rule_matches_service(span, rule, span_expected_to_match_rule): SamplingRule( sample_rate=0, name="test.span", - service=re.compile(r"^my-"), + service="my-*", ), True, ), @@ -580,7 +523,7 @@ def test_sampling_rule_matches_service(span, rule, span_expected_to_match_rule): SamplingRule( sample_rate=1, name="test_span", - service=re.compile(r"^my-"), + service="my-*", ), False, ), @@ -593,7 +536,7 @@ def test_sampling_rule_matches_service(span, rule, span_expected_to_match_rule): SamplingRule( sample_rate=1, name="test.span", - service=re.compile(r"^service-"), + service="service-", ), False, ), @@ -605,26 +548,6 @@ def test_sampling_rule_matches(span, rule, span_expected_to_match_rule): ) -def test_sampling_rule_matches_exception(): - def pattern(prop): - raise Exception("an error occurred") - - rule = SamplingRule(sample_rate=1.0, name=pattern) - span = create_span(name="test.span") - - with mock.patch("ddtrace._trace.sampling_rule.log") as mock_log: - assert ( - rule.matches(span) is False - ), "SamplingRule should not match when its name pattern function throws an exception" - mock_log.warning.assert_called_once_with( - "%r pattern %r failed with %r", - rule, - pattern, - "test.span", - exc_info=True, - ) - - @pytest.mark.subprocess( parametrize={"DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED": ["true", "false"]}, )