Remove expected traces count param from CI Vis snapshot tests #8311

nikita-tkachenko-datadog · 2025-01-30T10:57:50Z

What Does This Do

Removes expectedTraceCount parameter from CI Visibility instrumentation tests.
Instead of requiring the expected count to be specified manually, the tests will wait until the test session span is reported (this span is reported last, given that reporting spans in instrumentation tests is done sequentially, this should be reliable enough).

Motivation

Simplify writing tests.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1483

pr-commenter · 2025-01-30T11:37:08Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/ci-vis-snapshot-testing
git_commit_date	1738234130	1738234444
git_commit_sha	`1759b2d`	`606e233`
release_version	1.46.0-SNAPSHOT~1759b2df16	1.46.0-SNAPSHOT~606e2331de

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738236979	1738236979
ci_job_id	787081070	787081070
ci_pipeline_id	54426737	54426737
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-unakjcuf-project-304-concurrent-0-uajzcrp5 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-unakjcuf-project-304-concurrent-0-uajzcrp5 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1037677
Total [baseline] (10.448 s) : 0, 10447924
Agent [candidate] (1.042 s) : 0, 1041854
Total [candidate] (10.494 s) : 0, 10493973
section appsec
Agent [baseline] (1.181 s) : 0, 1181232
Total [baseline] (10.759 s) : 0, 10759488
Agent [candidate] (1.181 s) : 0, 1180971
Total [candidate] (10.78 s) : 0, 10780429
section iast
Agent [baseline] (1.177 s) : 0, 1176977
Total [baseline] (10.956 s) : 0, 10956073
Agent [candidate] (1.177 s) : 0, 1177276
Total [candidate] (11.029 s) : 0, 11028753
section profiling
Agent [baseline] (1.271 s) : 0, 1271000
Total [baseline] (10.909 s) : 0, 10908684
Agent [candidate] (1.266 s) : 0, 1265640
Total [candidate] (10.915 s) : 0, 10915123

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.038 s	-
Agent	appsec	1.181 s	143.555 ms (13.8%)
Agent	iast	1.177 s	139.3 ms (13.4%)
Agent	profiling	1.271 s	233.323 ms (22.5%)
Total	tracing	10.448 s	-
Total	appsec	10.759 s	311.565 ms (3.0%)
Total	iast	10.956 s	508.149 ms (4.9%)
Total	profiling	10.909 s	460.761 ms (4.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	appsec	1.181 s	139.117 ms (13.4%)
Agent	iast	1.177 s	135.422 ms (13.0%)
Agent	profiling	1.266 s	223.786 ms (21.5%)
Total	tracing	10.494 s	-
Total	appsec	10.78 s	286.456 ms (2.7%)
Total	iast	11.029 s	534.78 ms (5.1%)
Total	profiling	10.915 s	421.15 ms (4.0%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.115 ms) : 0, 713115
BytebuddyAgent [candidate] (717.064 ms) : 0, 717064
GlobalTracer [baseline] (239.772 ms) : 0, 239772
GlobalTracer [candidate] (240.902 ms) : 0, 240902
AppSec [baseline] (55.198 ms) : 0, 55198
AppSec [candidate] (55.571 ms) : 0, 55571
Remote Config [baseline] (707.143 µs) : 0, 707
Remote Config [candidate] (713.918 µs) : 0, 714
Telemetry [baseline] (13.661 ms) : 0, 13661
Telemetry [candidate] (12.308 ms) : 0, 12308
section appsec
BytebuddyAgent [baseline] (732.444 ms) : 0, 732444
BytebuddyAgent [candidate] (732.514 ms) : 0, 732514
GlobalTracer [baseline] (237.406 ms) : 0, 237406
GlobalTracer [candidate] (237.766 ms) : 0, 237766
IAST [baseline] (21.688 ms) : 0, 21688
IAST [candidate] (21.518 ms) : 0, 21518
AppSec [baseline] (175.953 ms) : 0, 175953
AppSec [candidate] (175.668 ms) : 0, 175668
Remote Config [baseline] (654.95 µs) : 0, 655
Remote Config [candidate] (666.927 µs) : 0, 667
Telemetry [baseline] (8.678 ms) : 0, 8678
Telemetry [candidate] (8.295 ms) : 0, 8295
section iast
BytebuddyAgent [baseline] (839.583 ms) : 0, 839583
BytebuddyAgent [candidate] (840.471 ms) : 0, 840471
GlobalTracer [baseline] (232.408 ms) : 0, 232408
GlobalTracer [candidate] (232.437 ms) : 0, 232437
IAST [baseline] (25.869 ms) : 0, 25869
IAST [candidate] (26.467 ms) : 0, 26467
AppSec [baseline] (54.173 ms) : 0, 54173
AppSec [candidate] (53.167 ms) : 0, 53167
Remote Config [baseline] (626.584 µs) : 0, 627
Remote Config [candidate] (627.325 µs) : 0, 627
Telemetry [baseline] (8.763 ms) : 0, 8763
Telemetry [candidate] (8.737 ms) : 0, 8737
section profiling
BytebuddyAgent [baseline] (713.776 ms) : 0, 713776
BytebuddyAgent [candidate] (712.408 ms) : 0, 712408
GlobalTracer [baseline] (352.516 ms) : 0, 352516
GlobalTracer [candidate] (350.968 ms) : 0, 350968
AppSec [baseline] (55.656 ms) : 0, 55656
AppSec [candidate] (54.206 ms) : 0, 54206
Remote Config [baseline] (706.132 µs) : 0, 706
Remote Config [candidate] (694.688 µs) : 0, 695
Telemetry [baseline] (8.941 ms) : 0, 8941
Telemetry [candidate] (8.885 ms) : 0, 8885
ProfilingAgent [baseline] (96.597 ms) : 0, 96597
ProfilingAgent [candidate] (95.733 ms) : 0, 95733
Profiling [baseline] (96.622 ms) : 0, 96622
Profiling [candidate] (95.757 ms) : 0, 95757

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.041 s) : 0, 1040841
Total [baseline] (8.636 s) : 0, 8635702
Agent [candidate] (1.041 s) : 0, 1040931
Total [candidate] (8.622 s) : 0, 8621673
section iast
Agent [baseline] (1.17 s) : 0, 1169748
Total [baseline] (9.214 s) : 0, 9214247
Agent [candidate] (1.175 s) : 0, 1174738
Total [candidate] (9.238 s) : 0, 9238160
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.166 s) : 0, 1166226
Total [baseline] (9.198 s) : 0, 9198433
Agent [candidate] (1.17 s) : 0, 1170077
Total [candidate] (9.268 s) : 0, 9267879
section iast_TELEMETRY_OFF
Agent [baseline] (1.163 s) : 0, 1162722
Total [baseline] (9.231 s) : 0, 9231500
Agent [candidate] (1.168 s) : 0, 1168243
Total [candidate] (9.21 s) : 0, 9209501

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	iast	1.17 s	128.907 ms (12.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.166 s	125.384 ms (12.0%)
Agent	iast_TELEMETRY_OFF	1.163 s	121.881 ms (11.7%)
Total	tracing	8.636 s	-
Total	iast	9.214 s	578.545 ms (6.7%)
Total	iast_HARDCODED_SECRET_DISABLED	9.198 s	562.731 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.231 s	595.797 ms (6.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	iast	1.175 s	133.807 ms (12.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.17 s	129.146 ms (12.4%)
Agent	iast_TELEMETRY_OFF	1.168 s	127.313 ms (12.2%)
Total	tracing	8.622 s	-
Total	iast	9.238 s	616.487 ms (7.2%)
Total	iast_HARDCODED_SECRET_DISABLED	9.268 s	646.206 ms (7.5%)
Total	iast_TELEMETRY_OFF	9.21 s	587.828 ms (6.8%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.148 ms) : 0, 716148
BytebuddyAgent [candidate] (714.767 ms) : 0, 714767
GlobalTracer [baseline] (240.324 ms) : 0, 240324
GlobalTracer [candidate] (239.712 ms) : 0, 239712
AppSec [baseline] (55.588 ms) : 0, 55588
AppSec [candidate] (56.118 ms) : 0, 56118
Remote Config [baseline] (711.912 µs) : 0, 712
Remote Config [candidate] (705.178 µs) : 0, 705
Telemetry [baseline] (12.849 ms) : 0, 12849
Telemetry [candidate] (14.459 ms) : 0, 14459
section iast
BytebuddyAgent [baseline] (834.245 ms) : 0, 834245
BytebuddyAgent [candidate] (840.081 ms) : 0, 840081
GlobalTracer [baseline] (230.938 ms) : 0, 230938
GlobalTracer [candidate] (230.452 ms) : 0, 230452
AppSec [baseline] (54.955 ms) : 0, 54955
AppSec [candidate] (53.826 ms) : 0, 53826
Remote Config [baseline] (607.245 µs) : 0, 607
Remote Config [candidate] (609.908 µs) : 0, 610
Telemetry [baseline] (8.744 ms) : 0, 8744
Telemetry [candidate] (8.77 ms) : 0, 8770
IAST [baseline] (24.929 ms) : 0, 24929
IAST [candidate] (25.648 ms) : 0, 25648
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (831.475 ms) : 0, 831475
BytebuddyAgent [candidate] (834.571 ms) : 0, 834571
GlobalTracer [baseline] (230.512 ms) : 0, 230512
GlobalTracer [candidate] (230.811 ms) : 0, 230811
AppSec [baseline] (54.841 ms) : 0, 54841
AppSec [candidate] (53.52 ms) : 0, 53520
Remote Config [baseline] (628.202 µs) : 0, 628
Remote Config [candidate] (632.895 µs) : 0, 633
Telemetry [baseline] (8.696 ms) : 0, 8696
Telemetry [candidate] (8.851 ms) : 0, 8851
IAST [baseline] (24.805 ms) : 0, 24805
IAST [candidate] (26.401 ms) : 0, 26401
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (829.157 ms) : 0, 829157
BytebuddyAgent [candidate] (833.613 ms) : 0, 833613
GlobalTracer [baseline] (230.21 ms) : 0, 230210
GlobalTracer [candidate] (231.456 ms) : 0, 231456
AppSec [baseline] (55.412 ms) : 0, 55412
AppSec [candidate] (54.368 ms) : 0, 54368
Remote Config [baseline] (607.661 µs) : 0, 608
Remote Config [candidate] (606.025 µs) : 0, 606
Telemetry [baseline] (8.528 ms) : 0, 8528
Telemetry [candidate] (8.563 ms) : 0, 8563
IAST [baseline] (23.587 ms) : 0, 23587
IAST [candidate] (24.391 ms) : 0, 24391

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-30T11:06:15	2025-01-30T11:13:19
git_branch	master	nikita-tkachenko/ci-vis-snapshot-testing
git_commit_date	1738234130	1738234444
git_commit_sha	`1759b2d`	`606e233`
release_version	1.46.0-SNAPSHOT~1759b2df16	1.46.0-SNAPSHOT~606e2331de
start_time	2025-01-30T11:06:01	2025-01-30T11:13:05

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738235956	1738235956
ci_job_id	787081071	787081071
ci_pipeline_id	54426737	54426737
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-unakjcuf-project-304-concurrent-1-fcb7cdl5 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-unakjcuf-project-304-concurrent-1-fcb7cdl5 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.345 ms) : 1325, 1364
.   : milestone, 1345,
appsec (1.756 ms) : 1732, 1779
.   : milestone, 1756,
appsec_no_iast (1.748 ms) : 1724, 1773
.   : milestone, 1748,
iast (1.511 ms) : 1487, 1535
.   : milestone, 1511,
profiling (1.512 ms) : 1489, 1534
.   : milestone, 1512,
tracing (1.502 ms) : 1479, 1526
.   : milestone, 1502,
section candidate
no_agent (1.365 ms) : 1345, 1386
.   : milestone, 1365,
appsec (1.745 ms) : 1721, 1770
.   : milestone, 1745,
appsec_no_iast (1.739 ms) : 1714, 1764
.   : milestone, 1739,
iast (1.507 ms) : 1482, 1532
.   : milestone, 1507,
profiling (1.496 ms) : 1474, 1519
.   : milestone, 1496,
tracing (1.492 ms) : 1467, 1516
.   : milestone, 1492,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.345 ms [1.325 ms, 1.364 ms]	-
appsec	1.756 ms [1.732 ms, 1.779 ms]	410.869 µs (30.6%)
appsec_no_iast	1.748 ms [1.724 ms, 1.773 ms]	403.762 µs (30.0%)
iast	1.511 ms [1.487 ms, 1.535 ms]	166.399 µs (12.4%)
profiling	1.512 ms [1.489 ms, 1.534 ms]	166.957 µs (12.4%)
tracing	1.502 ms [1.479 ms, 1.526 ms]	157.67 µs (11.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.365 ms [1.345 ms, 1.386 ms]	-
appsec	1.745 ms [1.721 ms, 1.77 ms]	380.27 µs (27.9%)
appsec_no_iast	1.739 ms [1.714 ms, 1.764 ms]	373.496 µs (27.4%)
iast	1.507 ms [1.482 ms, 1.532 ms]	141.525 µs (10.4%)
profiling	1.496 ms [1.474 ms, 1.519 ms]	131.256 µs (9.6%)
tracing	1.492 ms [1.467 ms, 1.516 ms]	126.467 µs (9.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16
    dateFormat X
    axisFormat %s
section baseline
no_agent (384.633 µs) : 364, 405
.   : milestone, 385,
iast (504.563 µs) : 483, 526
.   : milestone, 505,
iast_FULL (742.263 µs) : 720, 764
.   : milestone, 742,
iast_GLOBAL (565.86 µs) : 541, 591
.   : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (512.433 µs) : 489, 536
.   : milestone, 512,
iast_INACTIVE (456.528 µs) : 435, 478
.   : milestone, 457,
iast_TELEMETRY_OFF (499.03 µs) : 476, 522
.   : milestone, 499,
tracing (455.056 µs) : 431, 479
.   : milestone, 455,
section candidate
no_agent (376.413 µs) : 357, 396
.   : milestone, 376,
iast (507.794 µs) : 486, 530
.   : milestone, 508,
iast_FULL (744.404 µs) : 721, 768
.   : milestone, 744,
iast_GLOBAL (556.387 µs) : 534, 579
.   : milestone, 556,
iast_HARDCODED_SECRET_DISABLED (508.237 µs) : 485, 531
.   : milestone, 508,
iast_INACTIVE (460.158 µs) : 438, 482
.   : milestone, 460,
iast_TELEMETRY_OFF (489.261 µs) : 466, 512
.   : milestone, 489,
tracing (458.854 µs) : 437, 481
.   : milestone, 459,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	384.633 µs [364.493 µs, 404.772 µs]	-
iast	504.563 µs [482.953 µs, 526.172 µs]	119.93 µs (31.2%)
iast_FULL	742.263 µs [720.496 µs, 764.03 µs]	357.63 µs (93.0%)
iast_GLOBAL	565.86 µs [541.042 µs, 590.677 µs]	181.227 µs (47.1%)
iast_HARDCODED_SECRET_DISABLED	512.433 µs [489.094 µs, 535.772 µs]	127.8 µs (33.2%)
iast_INACTIVE	456.528 µs [434.931 µs, 478.126 µs]	71.896 µs (18.7%)
iast_TELEMETRY_OFF	499.03 µs [475.608 µs, 522.452 µs]	114.398 µs (29.7%)
tracing	455.056 µs [430.772 µs, 479.34 µs]	70.424 µs (18.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	376.413 µs [356.785 µs, 396.041 µs]	-
iast	507.794 µs [486.051 µs, 529.538 µs]	131.381 µs (34.9%)
iast_FULL	744.404 µs [721.028 µs, 767.781 µs]	367.991 µs (97.8%)
iast_GLOBAL	556.387 µs [534.108 µs, 578.665 µs]	179.974 µs (47.8%)
iast_HARDCODED_SECRET_DISABLED	508.237 µs [485.034 µs, 531.44 µs]	131.824 µs (35.0%)
iast_INACTIVE	460.158 µs [438.416 µs, 481.901 µs]	83.745 µs (22.2%)
iast_TELEMETRY_OFF	489.261 µs [466.241 µs, 512.282 µs]	112.848 µs (30.0%)
tracing	458.854 µs [437.193 µs, 480.514 µs]	82.441 µs (21.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/ci-vis-snapshot-testing
git_commit_date	1738234130	1738234444
git_commit_sha	`1759b2d`	`606e233`
release_version	1.46.0-SNAPSHOT~1759b2df16	1.46.0-SNAPSHOT~606e2331de

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1738236502	1738236502
ci_job_id	787081072	787081072
ci_pipeline_id	54426737	54426737
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-pgoe76dg-project-304-concurrent-1-u23r59lg 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-pgoe76dg-project-304-concurrent-1-u23r59lg 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.038 s) : 15038000, 15038000
.   : milestone, 15038000,
appsec (14.859 s) : 14859000, 14859000
.   : milestone, 14859000,
iast (18.658 s) : 18658000, 18658000
.   : milestone, 18658000,
iast_GLOBAL (17.97 s) : 17970000, 17970000
.   : milestone, 17970000,
profiling (15.624 s) : 15624000, 15624000
.   : milestone, 15624000,
tracing (15.138 s) : 15138000, 15138000
.   : milestone, 15138000,
section candidate
no_agent (14.848 s) : 14848000, 14848000
.   : milestone, 14848000,
appsec (14.936 s) : 14936000, 14936000
.   : milestone, 14936000,
iast (18.335 s) : 18335000, 18335000
.   : milestone, 18335000,
iast_GLOBAL (18.271 s) : 18271000, 18271000
.   : milestone, 18271000,
profiling (15.152 s) : 15152000, 15152000
.   : milestone, 15152000,
tracing (14.952 s) : 14952000, 14952000
.   : milestone, 14952000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.038 s [15.038 s, 15.038 s]	-
appsec	14.859 s [14.859 s, 14.859 s]	-179.0 ms (-1.2%)
iast	18.658 s [18.658 s, 18.658 s]	3.62 s (24.1%)
iast_GLOBAL	17.97 s [17.97 s, 17.97 s]	2.932 s (19.5%)
profiling	15.624 s [15.624 s, 15.624 s]	586.0 ms (3.9%)
tracing	15.138 s [15.138 s, 15.138 s]	100.0 ms (0.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.848 s [14.848 s, 14.848 s]	-
appsec	14.936 s [14.936 s, 14.936 s]	88.0 ms (0.6%)
iast	18.335 s [18.335 s, 18.335 s]	3.487 s (23.5%)
iast_GLOBAL	18.271 s [18.271 s, 18.271 s]	3.423 s (23.1%)
profiling	15.152 s [15.152 s, 15.152 s]	304.0 ms (2.0%)
tracing	14.952 s [14.952 s, 14.952 s]	104.0 ms (0.7%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~606e2331de, baseline=1.46.0-SNAPSHOT~1759b2df16
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (2.367 ms) : 2324, 2410
.   : milestone, 2367,
iast (2.117 ms) : 2063, 2172
.   : milestone, 2117,
iast_GLOBAL (2.156 ms) : 2101, 2212
.   : milestone, 2156,
profiling (1.997 ms) : 1952, 2042
.   : milestone, 1997,
tracing (1.949 ms) : 1907, 1991
.   : milestone, 1949,
section candidate
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (2.365 ms) : 2322, 2408
.   : milestone, 2365,
iast (2.105 ms) : 2050, 2160
.   : milestone, 2105,
iast_GLOBAL (2.153 ms) : 2098, 2208
.   : milestone, 2153,
profiling (1.985 ms) : 1941, 2028
.   : milestone, 1985,
tracing (1.947 ms) : 1905, 1989
.   : milestone, 1947,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.462 ms, 1.485 ms]	-
appsec	2.367 ms [2.324 ms, 2.41 ms]	893.853 µs (60.7%)
iast	2.117 ms [2.063 ms, 2.172 ms]	643.855 µs (43.7%)
iast_GLOBAL	2.156 ms [2.101 ms, 2.212 ms]	682.808 µs (46.3%)
profiling	1.997 ms [1.952 ms, 2.042 ms]	523.501 µs (35.5%)
tracing	1.949 ms [1.907 ms, 1.991 ms]	475.94 µs (32.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	2.365 ms [2.322 ms, 2.408 ms]	888.762 µs (60.2%)
iast	2.105 ms [2.05 ms, 2.16 ms]	628.63 µs (42.6%)
iast_GLOBAL	2.153 ms [2.098 ms, 2.208 ms]	676.578 µs (45.8%)
profiling	1.985 ms [1.941 ms, 2.028 ms]	508.13 µs (34.4%)
tracing	1.947 ms [1.905 ms, 1.989 ms]	470.394 µs (31.9%)

Remove expected traces count param from CI Vis snapshot tests

606e233

nikita-tkachenko-datadog added comp: testing Testing tag: no release notes Changes to exclude from release notes type: refactoring comp: ci visibility Continuous Integration Visibility labels Jan 30, 2025

nikita-tkachenko-datadog marked this pull request as ready for review January 30, 2025 10:58

nikita-tkachenko-datadog requested a review from a team as a code owner January 30, 2025 10:58

nikita-tkachenko-datadog requested review from juan-fernandez and daniel-mohedano January 30, 2025 10:58

daniel-mohedano approved these changes Jan 30, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit a6b67c3 into master Jan 30, 2025
200 of 201 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/ci-vis-snapshot-testing branch January 30, 2025 11:40

github-actions bot added this to the 1.46.0 milestone Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove expected traces count param from CI Vis snapshot tests #8311

Remove expected traces count param from CI Vis snapshot tests #8311

nikita-tkachenko-datadog commented Jan 30, 2025 •

edited

Loading

pr-commenter bot commented Jan 30, 2025

Remove expected traces count param from CI Vis snapshot tests #8311

Remove expected traces count param from CI Vis snapshot tests #8311

Conversation

nikita-tkachenko-datadog commented Jan 30, 2025 • edited Loading

What Does This Do

Motivation

Contributor Checklist

pr-commenter bot commented Jan 30, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 30, 2025 •

edited

Loading