Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix couchbase flaky test (cleanup phase) #7598

Merged
merged 1 commit into from
Sep 11, 2024
Merged

Fix couchbase flaky test (cleanup phase) #7598

merged 1 commit into from
Sep 11, 2024

Conversation

smola
Copy link
Member

@smola smola commented Sep 10, 2024

What Does This Do

Mute tracing on cleanup, instead of waiting for a cleanup span. This test is flaky when it does not receive the cleanup span in time (under heavy load).

Also, do not shallow exceptions on cleanup. If we ever get them, we can address them, or add a more specific catch.

Motivation

Additional Notes

Trick copied from @amarziali at #7597
The same pattern could be used in quite a few more (potentially flaky) tests.

Contributor Checklist

@smola
Fix couchbase flaky test (cleanup phase)
a049705 
Mute tracing on cleanup, instead of waiting for a cleanup span. This
test is flaky when it does not receive the cleanup span in time (under
heavy load).

Also, do not shallow exceptions on cleanup. If we ever get them, we can
address them, or add a more specific catch.
@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes tag: flaky test Flaky tests labels Sep 10, 2024
@pr-commenter
Copy link

pr-commenter bot commented Sep 10, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/flaky-tests-couchbase
git_commit_date 1725979768 1725982331
git_commit_sha 04db2f0 a049705
release_version 1.40.0-SNAPSHOT~04db2f0bfb 1.40.0-SNAPSHOT~a049705733
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725984899 1725984899
ci_job_id 634047890 634047890
ci_pipeline_id 43993869 43993869
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.082 s) : 0, 1081732
Total [baseline] (8.589 s) : 0, 8589251
Agent [candidate] (1.082 s) : 0, 1081569
Total [candidate] (8.571 s) : 0, 8570687
section iast
Agent [baseline] (1.191 s) : 0, 1191187
Total [baseline] (8.991 s) : 0, 8991312
Agent [candidate] (1.19 s) : 0, 1190415
Total [candidate] (8.998 s) : 0, 8998270
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.193 s) : 0, 1192810
Total [baseline] (8.965 s) : 0, 8965112
Agent [candidate] (1.191 s) : 0, 1190976
Total [candidate] (8.973 s) : 0, 8972588
section iast_TELEMETRY_OFF
Agent [baseline] (1.187 s) : 0, 1187488
Total [baseline] (8.97 s) : 0, 8969874
Agent [candidate] (1.189 s) : 0, 1188750
Total [candidate] (8.972 s) : 0, 8972195
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.082 s -
Agent iast 1.191 s 109.455 ms (10.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.193 s 111.078 ms (10.3%)
Agent iast_TELEMETRY_OFF 1.187 s 105.756 ms (9.8%)
Total tracing 8.589 s -
Total iast 8.991 s 402.062 ms (4.7%)
Total iast_HARDCODED_SECRET_DISABLED 8.965 s 375.861 ms (4.4%)
Total iast_TELEMETRY_OFF 8.97 s 380.624 ms (4.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.082 s -
Agent iast 1.19 s 108.846 ms (10.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.191 s 109.407 ms (10.1%)
Agent iast_TELEMETRY_OFF 1.189 s 107.181 ms (9.9%)
Total tracing 8.571 s -
Total iast 8.998 s 427.583 ms (5.0%)
Total iast_HARDCODED_SECRET_DISABLED 8.973 s 401.9 ms (4.7%)
Total iast_TELEMETRY_OFF 8.972 s 401.508 ms (4.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (691.569 ms) : 0, 691569
BytebuddyAgent [candidate] (690.715 ms) : 0, 690715
GlobalTracer [baseline] (315.209 ms) : 0, 315209
GlobalTracer [candidate] (315.774 ms) : 0, 315774
AppSec [baseline] (52.54 ms) : 0, 52540
AppSec [candidate] (52.756 ms) : 0, 52756
Remote Config [baseline] (720.582 µs) : 0, 721
Remote Config [candidate] (726.798 µs) : 0, 727
Telemetry [baseline] (7.765 ms) : 0, 7765
Telemetry [candidate] (7.652 ms) : 0, 7652
section iast
BytebuddyAgent [baseline] (792.308 ms) : 0, 792308
BytebuddyAgent [candidate] (791.928 ms) : 0, 791928
GlobalTracer [baseline] (299.493 ms) : 0, 299493
GlobalTracer [candidate] (300.13 ms) : 0, 300130
AppSec [baseline] (50.407 ms) : 0, 50407
AppSec [candidate] (53.641 ms) : 0, 53641
IAST [baseline] (27.147 ms) : 0, 27147
IAST [candidate] (22.981 ms) : 0, 22981
Remote Config [baseline] (684.974 µs) : 0, 685
Remote Config [candidate] (599.432 µs) : 0, 599
Telemetry [baseline] (7.363 ms) : 0, 7363
Telemetry [candidate] (7.377 ms) : 0, 7377
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (793.451 ms) : 0, 793451
BytebuddyAgent [candidate] (792.56 ms) : 0, 792560
GlobalTracer [baseline] (300.247 ms) : 0, 300247
GlobalTracer [candidate] (299.769 ms) : 0, 299769
AppSec [baseline] (53.533 ms) : 0, 53533
AppSec [candidate] (54.257 ms) : 0, 54257
IAST [baseline] (22.953 ms) : 0, 22953
IAST [candidate] (22.712 ms) : 0, 22712
Remote Config [baseline] (629.161 µs) : 0, 629
Remote Config [candidate] (607.321 µs) : 0, 607
Telemetry [baseline] (8.13 ms) : 0, 8130
Telemetry [candidate] (7.318 ms) : 0, 7318
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (788.932 ms) : 0, 788932
BytebuddyAgent [candidate] (790.058 ms) : 0, 790058
GlobalTracer [baseline] (300.009 ms) : 0, 300009
GlobalTracer [candidate] (300.121 ms) : 0, 300121
AppSec [baseline] (56.852 ms) : 0, 56852
AppSec [candidate] (54.465 ms) : 0, 54465
IAST [baseline] (19.963 ms) : 0, 19963
IAST [candidate] (22.451 ms) : 0, 22451
Remote Config [baseline] (605.675 µs) : 0, 606
Remote Config [candidate] (599.197 µs) : 0, 599
Telemetry [baseline] (7.358 ms) : 0, 7358
Telemetry [candidate] (7.258 ms) : 0, 7258
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1066387
Total [baseline] (10.389 s) : 0, 10389285
Agent [candidate] (1.074 s) : 0, 1074412
Total [candidate] (10.372 s) : 0, 10371813
section appsec
Agent [baseline] (1.203 s) : 0, 1202659
Total [baseline] (10.613 s) : 0, 10612679
Agent [candidate] (1.202 s) : 0, 1202354
Total [candidate] (10.64 s) : 0, 10640330
section iast
Agent [baseline] (1.201 s) : 0, 1200556
Total [baseline] (10.846 s) : 0, 10845676
Agent [candidate] (1.196 s) : 0, 1195585
Total [candidate] (10.856 s) : 0, 10855980
section profiling
Agent [baseline] (1.275 s) : 0, 1274530
Total [baseline] (10.749 s) : 0, 10748936
Agent [candidate] (1.273 s) : 0, 1273051
Total [candidate] (10.739 s) : 0, 10738762
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.066 s -
Agent appsec 1.203 s 136.271 ms (12.8%)
Agent iast 1.201 s 134.168 ms (12.6%)
Agent profiling 1.275 s 208.142 ms (19.5%)
Total tracing 10.389 s -
Total appsec 10.613 s 223.394 ms (2.2%)
Total iast 10.846 s 456.391 ms (4.4%)
Total profiling 10.749 s 359.651 ms (3.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent appsec 1.202 s 127.942 ms (11.9%)
Agent iast 1.196 s 121.174 ms (11.3%)
Agent profiling 1.273 s 198.639 ms (18.5%)
Total tracing 10.372 s -
Total appsec 10.64 s 268.518 ms (2.6%)
Total iast 10.856 s 484.167 ms (4.7%)
Total profiling 10.739 s 366.949 ms (3.5%) 
gantt
    title petclinic - break down per module: candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.084 ms) : 0, 681084
BytebuddyAgent [candidate] (686.145 ms) : 0, 686145
GlobalTracer [baseline] (311.441 ms) : 0, 311441
GlobalTracer [candidate] (313.834 ms) : 0, 313834
AppSec [baseline] (51.811 ms) : 0, 51811
AppSec [candidate] (52.263 ms) : 0, 52263
Remote Config [baseline] (707.328 µs) : 0, 707
Remote Config [candidate] (721.889 µs) : 0, 722
Telemetry [baseline] (7.632 ms) : 0, 7632
Telemetry [candidate] (7.597 ms) : 0, 7597
section appsec
BytebuddyAgent [baseline] (703.507 ms) : 0, 703507
BytebuddyAgent [candidate] (702.337 ms) : 0, 702337
GlobalTracer [baseline] (304.789 ms) : 0, 304789
GlobalTracer [candidate] (305.141 ms) : 0, 305141
AppSec [baseline] (160.904 ms) : 0, 160904
AppSec [candidate] (161.619 ms) : 0, 161619
IAST [baseline] (18.589 ms) : 0, 18589
IAST [candidate] (20.072 ms) : 0, 20072
Remote Config [baseline] (652.119 µs) : 0, 652
Remote Config [candidate] (650.186 µs) : 0, 650
Telemetry [baseline] (9.722 ms) : 0, 9722
Telemetry [candidate] (8.795 ms) : 0, 8795
section iast
BytebuddyAgent [baseline] (799.488 ms) : 0, 799488
BytebuddyAgent [candidate] (795.426 ms) : 0, 795426
GlobalTracer [baseline] (301.93 ms) : 0, 301930
GlobalTracer [candidate] (301.057 ms) : 0, 301057
AppSec [baseline] (53.547 ms) : 0, 53547
AppSec [candidate] (53.591 ms) : 0, 53591
IAST [baseline] (23.814 ms) : 0, 23814
IAST [candidate] (23.707 ms) : 0, 23707
Remote Config [baseline] (624.007 µs) : 0, 624
Remote Config [candidate] (620.689 µs) : 0, 621
Telemetry [baseline] (7.32 ms) : 0, 7320
Telemetry [candidate] (7.398 ms) : 0, 7398
section profiling
BytebuddyAgent [baseline] (680.521 ms) : 0, 680521
BytebuddyAgent [candidate] (680.027 ms) : 0, 680027
GlobalTracer [baseline] (396.579 ms) : 0, 396579
GlobalTracer [candidate] (395.988 ms) : 0, 395988
AppSec [baseline] (53.093 ms) : 0, 53093
AppSec [candidate] (52.758 ms) : 0, 52758
Remote Config [baseline] (725.474 µs) : 0, 725
Remote Config [candidate] (707.72 µs) : 0, 708
Telemetry [baseline] (7.487 ms) : 0, 7487
Telemetry [candidate] (7.475 ms) : 0, 7475
ProfilingAgent [baseline] (97.62 ms) : 0, 97620
ProfilingAgent [candidate] (97.564 ms) : 0, 97564
Profiling [baseline] (97.644 ms) : 0, 97644
Profiling [candidate] (97.588 ms) : 0, 97588
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-09-10T15:45:24 2024-09-10T15:52:18
git_branch master smola/flaky-tests-couchbase
git_commit_date 1725979768 1725982331
git_commit_sha 04db2f0 a049705
release_version 1.40.0-SNAPSHOT~04db2f0bfb 1.40.0-SNAPSHOT~a049705733
start_time 2024-09-10T15:45:11 2024-09-10T15:52:04
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725983884 1725983884
ci_job_id 634047892 634047892
ci_pipeline_id 43993869 43993869
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.365 ms) : 1345, 1386
.   : milestone, 1365,
appsec (1.709 ms) : 1684, 1734
.   : milestone, 1709,
appsec_no_iast (1.75 ms) : 1726, 1775
.   : milestone, 1750,
iast (1.475 ms) : 1453, 1497
.   : milestone, 1475,
profiling (1.489 ms) : 1464, 1514
.   : milestone, 1489,
tracing (1.495 ms) : 1471, 1519
.   : milestone, 1495,
section candidate
no_agent (1.333 ms) : 1314, 1353
.   : milestone, 1333,
appsec (1.73 ms) : 1705, 1754
.   : milestone, 1730,
appsec_no_iast (1.728 ms) : 1703, 1752
.   : milestone, 1728,
iast (1.481 ms) : 1459, 1503
.   : milestone, 1481,
profiling (1.487 ms) : 1463, 1510
.   : milestone, 1487,
tracing (1.478 ms) : 1454, 1502
.   : milestone, 1478,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.365 ms [1.345 ms, 1.386 ms] -
appsec 1.709 ms [1.684 ms, 1.734 ms] 343.54 µs (25.2%)
appsec_no_iast 1.75 ms [1.726 ms, 1.775 ms] 384.786 µs (28.2%)
iast 1.475 ms [1.453 ms, 1.497 ms] 109.537 µs (8.0%)
profiling 1.489 ms [1.464 ms, 1.514 ms] 123.838 µs (9.1%)
tracing 1.495 ms [1.471 ms, 1.519 ms] 129.876 µs (9.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.333 ms [1.314 ms, 1.353 ms] -
appsec 1.73 ms [1.705 ms, 1.754 ms] 396.18 µs (29.7%)
appsec_no_iast 1.728 ms [1.703 ms, 1.752 ms] 394.211 µs (29.6%)
iast 1.481 ms [1.459 ms, 1.503 ms] 147.306 µs (11.0%)
profiling 1.487 ms [1.463 ms, 1.51 ms] 153.07 µs (11.5%)
tracing 1.478 ms [1.454 ms, 1.502 ms] 144.792 µs (10.9%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb
    dateFormat X
    axisFormat %s
section baseline
no_agent (375.341 µs) : 355, 395
.   : milestone, 375,
iast (495.418 µs) : 473, 518
.   : milestone, 495,
iast_FULL (559.73 µs) : 538, 581
.   : milestone, 560,
iast_GLOBAL (520.966 µs) : 499, 543
.   : milestone, 521,
iast_HARDCODED_SECRET_DISABLED (496.697 µs) : 475, 519
.   : milestone, 497,
iast_INACTIVE (450.041 µs) : 430, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (494.053 µs) : 471, 518
.   : milestone, 494,
tracing (455.462 µs) : 435, 476
.   : milestone, 455,
section candidate
no_agent (378.123 µs) : 358, 398
.   : milestone, 378,
iast (493.173 µs) : 471, 516
.   : milestone, 493,
iast_FULL (559.204 µs) : 538, 580
.   : milestone, 559,
iast_GLOBAL (506.406 µs) : 486, 527
.   : milestone, 506,
iast_HARDCODED_SECRET_DISABLED (493.773 µs) : 472, 516
.   : milestone, 494,
iast_INACTIVE (456.705 µs) : 436, 478
.   : milestone, 457,
iast_TELEMETRY_OFF (482.174 µs) : 459, 505
.   : milestone, 482,
tracing (452.607 µs) : 432, 474
.   : milestone, 453,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 375.341 µs [355.473 µs, 395.209 µs] -
iast 495.418 µs [473.194 µs, 517.641 µs] 120.077 µs (32.0%)
iast_FULL 559.73 µs [538.397 µs, 581.063 µs] 184.389 µs (49.1%)
iast_GLOBAL 520.966 µs [499.272 µs, 542.659 µs] 145.625 µs (38.8%)
iast_HARDCODED_SECRET_DISABLED 496.697 µs [474.515 µs, 518.88 µs] 121.357 µs (32.3%)
iast_INACTIVE 450.041 µs [429.526 µs, 470.555 µs] 74.7 µs (19.9%)
iast_TELEMETRY_OFF 494.053 µs [470.565 µs, 517.541 µs] 118.712 µs (31.6%)
tracing 455.462 µs [434.874 µs, 476.051 µs] 80.122 µs (21.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.123 µs [358.411 µs, 397.835 µs] -
iast 493.173 µs [470.565 µs, 515.78 µs] 115.05 µs (30.4%)
iast_FULL 559.204 µs [537.942 µs, 580.465 µs] 181.081 µs (47.9%)
iast_GLOBAL 506.406 µs [485.547 µs, 527.264 µs] 128.283 µs (33.9%)
iast_HARDCODED_SECRET_DISABLED 493.773 µs [471.899 µs, 515.646 µs] 115.65 µs (30.6%)
iast_INACTIVE 456.705 µs [435.553 µs, 477.857 µs] 78.582 µs (20.8%)
iast_TELEMETRY_OFF 482.174 µs [459.148 µs, 505.199 µs] 104.051 µs (27.5%)
tracing 452.607 µs [431.591 µs, 473.623 µs] 74.485 µs (19.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/flaky-tests-couchbase
git_commit_date 1725979768 1725982331
git_commit_sha 04db2f0 a049705
release_version 1.40.0-SNAPSHOT~04db2f0bfb 1.40.0-SNAPSHOT~a049705733
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1725984424 1725984424
ci_job_id 634047894 634047894
ci_pipeline_id 43993869 43993869
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.321 s) : 15321000, 15321000
.   : milestone, 15321000,
appsec (15.209 s) : 15209000, 15209000
.   : milestone, 15209000,
iast (18.861 s) : 18861000, 18861000
.   : milestone, 18861000,
iast_GLOBAL (17.989 s) : 17989000, 17989000
.   : milestone, 17989000,
profiling (14.9 s) : 14900000, 14900000
.   : milestone, 14900000,
tracing (15.41 s) : 15410000, 15410000
.   : milestone, 15410000,
section candidate
no_agent (14.898 s) : 14898000, 14898000
.   : milestone, 14898000,
appsec (15.372 s) : 15372000, 15372000
.   : milestone, 15372000,
iast (18.933 s) : 18933000, 18933000
.   : milestone, 18933000,
iast_GLOBAL (18.09 s) : 18090000, 18090000
.   : milestone, 18090000,
profiling (14.876 s) : 14876000, 14876000
.   : milestone, 14876000,
tracing (15.162 s) : 15162000, 15162000
.   : milestone, 15162000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.321 s [15.321 s, 15.321 s] -
appsec 15.209 s [15.209 s, 15.209 s] -112.0 ms (-0.7%)
iast 18.861 s [18.861 s, 18.861 s] 3.54 s (23.1%)
iast_GLOBAL 17.989 s [17.989 s, 17.989 s] 2.668 s (17.4%)
profiling 14.9 s [14.9 s, 14.9 s] -421.0 ms (-2.7%)
tracing 15.41 s [15.41 s, 15.41 s] 89.0 ms (0.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.898 s [14.898 s, 14.898 s] -
appsec 15.372 s [15.372 s, 15.372 s] 474.0 ms (3.2%)
iast 18.933 s [18.933 s, 18.933 s] 4.035 s (27.1%)
iast_GLOBAL 18.09 s [18.09 s, 18.09 s] 3.192 s (21.4%)
profiling 14.876 s [14.876 s, 14.876 s] -22.0 ms (-0.1%)
tracing 15.162 s [15.162 s, 15.162 s] 264.0 ms (1.8%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~a049705733, baseline=1.40.0-SNAPSHOT~04db2f0bfb
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.457 ms) : 1446, 1469
.   : milestone, 1457,
appsec (2.278 ms) : 2238, 2319
.   : milestone, 2278,
iast (2.023 ms) : 1974, 2071
.   : milestone, 2023,
iast_GLOBAL (2.078 ms) : 2028, 2128
.   : milestone, 2078,
profiling (1.91 ms) : 1871, 1950
.   : milestone, 1910,
tracing (1.883 ms) : 1845, 1921
.   : milestone, 1883,
section candidate
no_agent (1.465 ms) : 1454, 1477
.   : milestone, 1465,
appsec (2.279 ms) : 2239, 2319
.   : milestone, 2279,
iast (2.042 ms) : 1992, 2092
.   : milestone, 2042,
iast_GLOBAL (2.08 ms) : 2030, 2130
.   : milestone, 2080,
profiling (1.917 ms) : 1877, 1956
.   : milestone, 1917,
tracing (1.883 ms) : 1845, 1921
.   : milestone, 1883,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.457 ms [1.446 ms, 1.469 ms] -
appsec 2.278 ms [2.238 ms, 2.319 ms] 820.749 µs (56.3%)
iast 2.023 ms [1.974 ms, 2.071 ms] 565.408 µs (38.8%)
iast_GLOBAL 2.078 ms [2.028 ms, 2.128 ms] 620.362 µs (42.6%)
profiling 1.91 ms [1.871 ms, 1.95 ms] 452.926 µs (31.1%)
tracing 1.883 ms [1.845 ms, 1.921 ms] 425.829 µs (29.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.465 ms [1.454 ms, 1.477 ms] -
appsec 2.279 ms [2.239 ms, 2.319 ms] 813.726 µs (55.5%)
iast 2.042 ms [1.992 ms, 2.092 ms] 577.228 µs (39.4%)
iast_GLOBAL 2.08 ms [2.03 ms, 2.13 ms] 615.255 µs (42.0%)
profiling 1.917 ms [1.877 ms, 1.956 ms] 451.577 µs (30.8%)
tracing 1.883 ms [1.845 ms, 1.921 ms] 418.203 µs (28.5%)

@smola smola marked this pull request as ready for review September 11, 2024 07:16
@smola smola requested a review from a team as a code owner September 11, 2024 07:16
@smola smola merged commit 9a35dac into master Sep 11, 2024
98 of 99 checks passed
@smola smola deleted the smola/flaky-tests-couchbase branch September 11, 2024 08:59
@github-actions github-actions bot added this to the 1.40.0 milestone Sep 11, 2024
@smola smola mentioned this pull request Sep 13, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amarziali amarziali amarziali approved these changes

Assignees
No one assigned
Labels
comp: testing Testing tag: flaky test Flaky tests tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.40.0
Development

Successfully merging this pull request may close these issues.
2 participants
@smola @amarziali