chore(symdb): add shallow mode by P403n1x87 · Pull Request #12957 · DataDog/dd-trace-py

P403n1x87 · 2025-03-28T18:21:36Z

We add the shallow mode to Symbol DB to upload only top-level module scopes when DI is not enabled.

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

We add the shallow mode to Symbol DB to upload only top-level module scopes when DI is not enabled.

github-actions · 2025-03-28T18:22:25Z

CODEOWNERS have been resolved as:

ddtrace/debugging/_debugger.py                                          @DataDog/debugger-python
ddtrace/internal/products.py                                            @DataDog/apm-core-python
ddtrace/internal/symbol_db/__init__.py                                  @DataDog/debugger-python
ddtrace/internal/symbol_db/remoteconfig.py                              @DataDog/debugger-python
ddtrace/internal/symbol_db/symbols.py                                   @DataDog/debugger-python

github-actions · 2025-03-28T18:39:57Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 243 ± 4 ms.

The average import time from base is: 241 ± 4 ms.

The import time difference between this PR and base is: 2.6 ± 0.2 ms.

Import time breakdown

The following import paths have appeared:

ddtrace.auto 24.778 ms (10.19%)

ddtrace.bootstrap.sitecustomize 24.778 ms (10.19%)

ddtrace.bootstrap.preload 24.778 ms (10.19%)

ddtrace.internal.remoteconfig.client 7.403 ms (3.04%)

ddtrace.internal.remoteconfig.constants 0.259 ms (0.11%)

ddtrace.internal.symbol_db.remoteconfig 5.089 ms (2.09%)

ddtrace.internal.symbol_db.symbols 3.829 ms (1.57%)

ddtrace.settings.symbol_db 0.429 ms (0.18%)

ddtrace.internal.utils.inspection 0.402 ms (0.17%)

ddtrace.internal.safety 0.338 ms (0.14%)

multiprocessing 3.887 ms (1.60%)

multiprocessing.context 3.563 ms (1.47%)

multiprocessing.reduction 2.342 ms (0.96%)

pickle 1.899 ms (0.78%)

_compat_pickle 0.458 ms (0.19%)

_pickle 0.411 ms (0.17%)

multiprocessing.process 0.472 ms (0.19%)

ddtrace.internal.remoteconfig._connectors 2.477 ms (1.02%)

ctypes 2.077 ms (0.85%)

_ctypes 0.579 ms (0.24%)

ctypes._endian 0.441 ms (0.18%)

multiprocessing.sharedctypes 1.900 ms (0.78%)

multiprocessing.heap 1.207 ms (0.50%)

multiprocessing.util 0.421 ms (0.17%)

mmap 0.407 ms (0.17%)

ddtrace.settings.dynamic_instrumentation 0.799 ms (0.33%)

ddtrace.internal.utils.config 0.251 ms (0.10%)

ddtrace.internal.remoteconfig._pubsub 0.668 ms (0.27%)

ddtrace.internal.remoteconfig._subscribers 0.349 ms (0.14%)

ddtrace.internal.remoteconfig.worker 0.590 ms (0.24%)

ddtrace.debugging._config 0.566 ms (0.23%)

ddtrace.settings.exception_replay 0.322 ms (0.13%)

ddtrace.settings.code_origin 0.402 ms (0.17%)

ddtrace.settings.errortracking 0.356 ms (0.15%)

ddtrace.internal.remoteconfig._publishers 0.340 ms (0.14%)

ddtrace.settings.live_debugging 0.301 ms (0.12%)

The following import paths have disappeared:

ddtrace.auto 24.314 ms (10.00%)

ddtrace.bootstrap.sitecustomize 24.314 ms (10.00%)

ddtrace.bootstrap.preload 24.314 ms (10.00%)

ddtrace.internal.products 24.314 ms (10.00%)

ddtrace.internal.remoteconfig.client 7.203 ms (2.96%)

ddtrace.internal.remoteconfig.constants 0.243 ms (0.10%)

ddtrace.internal.symbol_db.remoteconfig 5.007 ms (2.06%)

ddtrace.internal.symbol_db.symbols 3.787 ms (1.56%)

ddtrace.settings.symbol_db 0.412 ms (0.17%)

ddtrace.internal.utils.inspection 0.398 ms (0.16%)

ddtrace.internal.safety 0.339 ms (0.14%)

multiprocessing 3.782 ms (1.55%)

multiprocessing.context 3.459 ms (1.42%)

multiprocessing.reduction 2.278 ms (0.94%)

pickle 1.849 ms (0.76%)

_compat_pickle 0.443 ms (0.18%)

_pickle 0.403 ms (0.17%)

multiprocessing.process 0.458 ms (0.19%)

ddtrace.internal.remoteconfig._connectors 2.420 ms (1.00%)

ctypes 2.027 ms (0.83%)

_ctypes 0.645 ms (0.27%)

ctypes._endian 0.432 ms (0.18%)

multiprocessing.sharedctypes 1.848 ms (0.76%)

multiprocessing.heap 1.165 ms (0.48%)

multiprocessing.util 0.404 ms (0.17%)

mmap 0.398 ms (0.16%)

ddtrace.settings.dynamic_instrumentation 0.875 ms (0.36%)

ddtrace.internal.utils.config 0.245 ms (0.10%)

ddtrace.internal.remoteconfig._pubsub 0.661 ms (0.27%)

ddtrace.internal.remoteconfig._subscribers 0.342 ms (0.14%)

ddtrace.debugging._config 0.572 ms (0.24%)

ddtrace.settings.exception_replay 0.324 ms (0.13%)

ddtrace.internal.remoteconfig.worker 0.556 ms (0.23%)

ddtrace.settings.code_origin 0.401 ms (0.16%)

ddtrace.settings.errortracking 0.344 ms (0.14%)

ddtrace.internal.remoteconfig._publishers 0.337 ms (0.14%)

ddtrace.settings.live_debugging 0.307 ms (0.13%)

The following import paths have grown:

ddtrace.auto 14.106 ms (5.80%)

ddtrace.bootstrap.sitecustomize 14.088 ms (5.79%)

ddtrace.bootstrap.preload 13.655 ms (5.61%)

ddtrace.internal.runtime.runtime_metrics 1.046 ms (0.43%)

ddtrace.internal.runtime.metric_collectors 1.020 ms (0.42%)

ddtrace.internal.runtime.collector 0.028 ms (0.01%)

ddtrace.settings.profiling 0.293 ms (0.12%)

ddtrace.internal.datadog.profiling.ddup 0.085 ms (0.03%)

ddtrace.vendor.psutil 0.060 ms (0.02%)

pwd 0.060 ms (0.02%)

ddtrace.internal.flare.flare 0.087 ms (0.04%)

ddtrace.settings.crashtracker 0.042 ms (0.02%)

ddtrace._trace.trace_handlers 0.136 ms (0.06%)

ddtrace._trace._inferred_proxy 0.081 ms (0.03%)

ddtrace.propagation.http 0.028 ms (0.01%)

ddtrace.internal._tagset 0.028 ms (0.01%)

ddtrace._trace.utils 0.056 ms (0.02%)

shlex 0.098 ms (0.04%)

ddtrace.appsec._common_module_patches 0.095 ms (0.04%)

ddtrace.appsec._asm_request_context 0.095 ms (0.04%)

ddtrace.appsec._utils 0.095 ms (0.04%)

ddtrace 0.018 ms (0.01%)

ddtrace._monkey 0.018 ms (0.01%)

ddtrace.appsec._listeners 0.018 ms (0.01%)

ddtrace.appsec 0.018 ms (0.01%)

The following import paths have shrunk:

ddtrace.auto 15.240 ms (6.27%)

ddtrace.bootstrap.sitecustomize 14.581 ms (5.99%)

ddtrace.bootstrap.preload 14.418 ms (5.93%)

ddtrace.internal.products 13.361 ms (5.49%)

ddtrace.settings.profiling 0.797 ms (0.33%)

ddtrace.vendor.psutil 0.797 ms (0.33%)

ddtrace.vendor.psutil._pslinux 0.725 ms (0.30%)

ddtrace.appsec._remoteconfiguration 0.096 ms (0.04%)

ddtrace.appsec._capabilities 0.096 ms (0.04%)

ddtrace.settings.crashtracker 0.095 ms (0.04%)

ddtrace.internal.datadog.profiling.crashtracker 0.095 ms (0.04%)

ddtrace.internal.datadog.profiling.crashtracker._crashtracker 0.095 ms (0.04%)

ddtrace.internal.flare.flare 0.068 ms (0.03%)

ddtrace.internal.flare 0.068 ms (0.03%)

ddtrace.appsec._common_module_patches 0.083 ms (0.03%)

ddtrace.appsec._asm_request_context 0.083 ms (0.03%)

ddtrace._trace.trace_handlers 0.079 ms (0.03%)

ddtrace.contrib.internal.botocore.constants 0.079 ms (0.03%)

ddtrace.contrib.internal.botocore 0.079 ms (0.03%)

ddtrace 0.659 ms (0.27%)

pr-commenter · 2025-03-28T19:09:48Z

Benchmarks

Benchmark execution time: 2025-05-13 14:17:20

Comparing candidate commit 45d2fa2 in PR branch chore/symdb-shallow-mode with baseline commit ac3d939 in branch main.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 499 metrics, 9 unstable metrics.

scenario:iast_aspects-ospathnormcase_aspect

🟥 execution_time [+348.533ns; +442.872ns] or [+10.238%; +13.009%]

scenario:iast_aspects-replace_aspect

🟥 execution_time [+497.041ns; +619.161ns] or [+10.796%; +13.449%]

scenario:iast_aspects-rstrip_aspect

🟥 execution_time [+1.213µs; +1.338µs] or [+9.350%; +10.313%]

scenario:telemetryaddmetric-1-gauge-metric-1-times

🟥 execution_time [+249.850ns; +303.216ns] or [+11.759%; +14.271%]

ddtrace/internal/products.py

datadog-datadog-prod-us1 · 2025-05-13T08:40:07Z

Datadog Summary

✅ Code Quality ✅ Code Security ✅ Dependencies

Was this helpful? Give us feedback!

This fixes a bug introduced by #12957 where the shallow mode for SymDB was introduced. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

This fixes a bug introduced by #12957 where the shallow mode for SymDB was introduced. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) (cherry picked from commit 58bd69c)

…13528) Backport 58bd69c from #13522 to 3.8. This fixes a bug introduced by #12957 where the shallow mode for SymDB was introduced. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: Tyler Finethy <tylfin@gmail.com> Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com>

chore(symdb): add shallow mode

ae5f015

We add the shallow mode to Symbol DB to upload only top-level module scopes when DI is not enabled.

P403n1x87 added changelog/no-changelog A changelog entry is not required for this PR. Dynamic Instrumentation Dynamic Instrumentation/Live Debugger labels Mar 28, 2025

P403n1x87 and others added 3 commits April 16, 2025 15:10

Merge branch 'main' into chore/symdb-shallow-mode

605b7a2

Merge branch 'main' into chore/symdb-shallow-mode

3a21747

load products on protocol run

daded2a

P403n1x87 marked this pull request as ready for review May 6, 2025 12:35

P403n1x87 requested review from a team as code owners May 6, 2025 12:35

P403n1x87 requested a review from juanjux May 6, 2025 12:35

P403n1x87 enabled auto-merge (squash) May 6, 2025 12:35

ojung approved these changes May 6, 2025

View reviewed changes

P403n1x87 requested a review from tylfin May 6, 2025 14:11

emmettbutler approved these changes May 6, 2025

View reviewed changes

tylfin reviewed May 6, 2025

View reviewed changes

ddtrace/internal/products.py Show resolved Hide resolved

tylfin approved these changes May 6, 2025

View reviewed changes

P403n1x87 added 2 commits May 7, 2025 12:10

Merge branch 'main' into chore/symdb-shallow-mode

9b4145c

Merge branch 'main' into chore/symdb-shallow-mode

2949229

Merge branch 'main' into chore/symdb-shallow-mode

45d2fa2

P403n1x87 merged commit e6c8d4d into main May 13, 2025
713 of 714 checks passed

P403n1x87 deleted the chore/symdb-shallow-mode branch May 13, 2025 14:18

tylfin mentioned this pull request May 28, 2025

fix(symdb): switch _shallow to shallow in install #13522

Merged

2 tasks

github-actions bot mentioned this pull request May 28, 2025

fix(symdb): switch _shallow to shallow in install [backport 3.8] #13528

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

chore(symdb): add shallow mode#12957

chore(symdb): add shallow mode#12957
P403n1x87 merged 7 commits intomainfrom
chore/symdb-shallow-mode

P403n1x87 commented Mar 28, 2025 •

edited by tylfin

Loading

Uh oh!

github-actions bot commented Mar 28, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 28, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Mar 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented May 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

P403n1x87 commented Mar 28, 2025 • edited by tylfin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Reviewer Checklist

Uh oh!

github-actions bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bootstrap import analysis

Summary

Import time breakdown

Uh oh!

pr-commenter bot commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:iast_aspects-ospathnormcase_aspect

scenario:iast_aspects-replace_aspect

scenario:iast_aspects-rstrip_aspect

scenario:telemetryaddmetric-1-gauge-metric-1-times

Uh oh!

Uh oh!

datadog-datadog-prod-us1 bot commented May 13, 2025

Datadog Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

P403n1x87 commented Mar 28, 2025 •

edited by tylfin

Loading

github-actions bot commented Mar 28, 2025 •

edited

Loading

github-actions bot commented Mar 28, 2025 •

edited

Loading

pr-commenter bot commented Mar 28, 2025 •

edited

Loading