New API Security sampling algorithm #8178

ValentinZakharov · 2025-01-10T07:42:26Z

What Does This Do

Endpoint-based API Security sampling

API Security schema extraction had a fixed sampling rate. This led to an excessive amount of work spent on redundant schemas, while still not getting schemas for infrequent endpoints. This problem was aggravated by the interaction with APM sampling, which could decide to drop a trace after we inferred the schema.

The new algorithm implemented here works as follows:

Schema extraction is sampled in at most once every 30 seconds for each (route, http method, status code) combination.
Route is currently retrieved from the http.route tag, and if it is absent, schema extraction will not happen at all.
The sampling decision happens after APM sampling, so we will not try to sample in a trace if it was dropped by the APM sampler.

system-test upgrade for DataDog/system-tests#4195, is required

Late processing

APM sampling decision might not be present until a point later than the request end handlers. So in our usual request end handler for AppSec we cannot access it. To fit this requirement, we're moving schema extraction to the trace serialization thread (see TraceProcessingWorker), where the APM sampling decision is guaranteed to have happened. This is based on the API for trace post-processing introduced at #6800.

This also has the advantage of moving schema extraction out of the critical path to handle the request, and should have a positive impact in latency when API security is enabled.

However, it has two additional risks:

Excessively delay trace serialization. To mitigate that, there is a timeout of 1s (see dd.trace.post-processing.timeout) for trace post-processors. If that timeout ellapses, no more schema extraction will happen.
Performing schema extraction required an open AppSec request context. This would lead to excessive memory usage if we just kept all of them open until trace deserialization. To avoid that, we:
- At request end, perform a speculative check to see if this trace might potentially be sampled in later. If it is known beforehand that it will not be sampled in, then the request context is closed.
- We put a hard limit to the number of request contexts that can stay open concurrently past the end of the request. This is set to 4.

Other changes

The old sampling mechanism is fully removed:

Remote some leftovers from remote config code to handle sampling rate, which is gone.
The dd.api-security.request.sample.rate (system property) or DD_API_SECURITY_REQUEST_SAMPLE_RATE (env var) option is gone.
A new setting dd.api-security.sample.delay (system property) or DD_API_SECURITY_SAMPLE_DELAY defaulting to 30 is introduced, to set the interval in seconds for the new sampler. This should generally not be used by customers, but is used in testing scenarios.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: APPSEC-54874

pr-commenter · 2025-01-10T08:18:57Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	vzakharov/api_sec_sampling_new
git_commit_date	1742990727	1743064463
git_commit_sha	`3e2867a`	`98d957d`
release_version	1.48.0-SNAPSHOT~3e2867a84a	1.48.0-SNAPSHOT~98d957d662

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1743067580	1743067580
ci_job_id	867266664	867266664
ci_pipeline_id	60141941	60141941
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5gca-jya-project-304-concurrent-3-12g5wy1l 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5gca-jya-project-304-concurrent-3-12g5wy1l 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053293
Total [baseline] (8.711 s) : 0, 8710540
Agent [candidate] (1.058 s) : 0, 1057583
Total [candidate] (8.697 s) : 0, 8696586
section iast
Agent [baseline] (1.178 s) : 0, 1178181
Total [baseline] (9.264 s) : 0, 9263516
Agent [candidate] (1.182 s) : 0, 1181631
Total [candidate] (9.258 s) : 0, 9257892
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.176 s) : 0, 1176261
Total [baseline] (9.221 s) : 0, 9221195
Agent [candidate] (1.18 s) : 0, 1179956
Total [candidate] (9.232 s) : 0, 9232400
section iast_TELEMETRY_OFF
Agent [baseline] (1.173 s) : 0, 1172929
Total [baseline] (9.243 s) : 0, 9242701
Agent [candidate] (1.175 s) : 0, 1174748
Total [candidate] (9.242 s) : 0, 9241831

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	iast	1.178 s	124.887 ms (11.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.176 s	122.967 ms (11.7%)
Agent	iast_TELEMETRY_OFF	1.173 s	119.636 ms (11.4%)
Total	tracing	8.711 s	-
Total	iast	9.264 s	552.976 ms (6.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.221 s	510.655 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.243 s	532.161 ms (6.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	iast	1.182 s	124.048 ms (11.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.18 s	122.374 ms (11.6%)
Agent	iast_TELEMETRY_OFF	1.175 s	117.166 ms (11.1%)
Total	tracing	8.697 s	-
Total	iast	9.258 s	561.306 ms (6.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.232 s	535.815 ms (6.2%)
Total	iast_TELEMETRY_OFF	9.242 s	545.246 ms (6.3%)

gantt
    title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (720.67 ms) : 0, 720670
BytebuddyAgent [candidate] (726.694 ms) : 0, 726694
GlobalTracer [baseline] (240.493 ms) : 0, 240493
GlobalTracer [candidate] (242.118 ms) : 0, 242118
AppSec [baseline] (54.873 ms) : 0, 54873
AppSec [candidate] (55.139 ms) : 0, 55139
Debugger [baseline] (4.397 ms) : 0, 4397
Debugger [candidate] (4.44 ms) : 0, 4440
Remote Config [baseline] (716.178 µs) : 0, 716
Remote Config [candidate] (726.401 µs) : 0, 726
Telemetry [baseline] (16.089 ms) : 0, 16089
Telemetry [candidate] (12.336 ms) : 0, 12336
section iast
BytebuddyAgent [baseline] (838.616 ms) : 0, 838616
BytebuddyAgent [candidate] (841.825 ms) : 0, 841825
GlobalTracer [baseline] (230.587 ms) : 0, 230587
GlobalTracer [candidate] (230.982 ms) : 0, 230982
AppSec [baseline] (56.387 ms) : 0, 56387
AppSec [candidate] (56.272 ms) : 0, 56272
Debugger [baseline] (4.174 ms) : 0, 4174
Debugger [candidate] (4.135 ms) : 0, 4135
Remote Config [baseline] (608.574 µs) : 0, 609
Remote Config [candidate] (595.239 µs) : 0, 595
Telemetry [baseline] (8.886 ms) : 0, 8886
Telemetry [candidate] (8.795 ms) : 0, 8795
IAST [baseline] (22.907 ms) : 0, 22907
IAST [candidate] (23.005 ms) : 0, 23005
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.163 ms) : 0, 837163
BytebuddyAgent [candidate] (840.945 ms) : 0, 840945
GlobalTracer [baseline] (230.598 ms) : 0, 230598
GlobalTracer [candidate] (230.771 ms) : 0, 230771
AppSec [baseline] (56.178 ms) : 0, 56178
AppSec [candidate] (55.841 ms) : 0, 55841
Debugger [baseline] (4.139 ms) : 0, 4139
Debugger [candidate] (4.164 ms) : 0, 4164
Remote Config [baseline] (610.022 µs) : 0, 610
Remote Config [candidate] (592.316 µs) : 0, 592
Telemetry [baseline] (8.825 ms) : 0, 8825
Telemetry [candidate] (8.828 ms) : 0, 8828
IAST [baseline] (22.786 ms) : 0, 22786
IAST [candidate] (22.787 ms) : 0, 22787
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (835.393 ms) : 0, 835393
BytebuddyAgent [candidate] (836.452 ms) : 0, 836452
GlobalTracer [baseline] (230.22 ms) : 0, 230220
GlobalTracer [candidate] (230.407 ms) : 0, 230407
AppSec [baseline] (55.121 ms) : 0, 55121
AppSec [candidate] (56.103 ms) : 0, 56103
Debugger [baseline] (4.101 ms) : 0, 4101
Debugger [candidate] (4.138 ms) : 0, 4138
Remote Config [baseline] (607.127 µs) : 0, 607
Remote Config [candidate] (620.744 µs) : 0, 621
Telemetry [baseline] (8.59 ms) : 0, 8590
Telemetry [candidate] (8.723 ms) : 0, 8723
IAST [baseline] (22.915 ms) : 0, 22915
IAST [candidate] (22.321 ms) : 0, 22321

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1052034
Total [baseline] (10.523 s) : 0, 10523261
Agent [candidate] (1.049 s) : 0, 1048800
Total [candidate] (10.424 s) : 0, 10423561
section appsec
Agent [baseline] (1.198 s) : 0, 1197508
Total [baseline] (10.759 s) : 0, 10759489
Agent [candidate] (1.194 s) : 0, 1194170
Total [candidate] (10.852 s) : 0, 10852249
section iast
Agent [baseline] (1.178 s) : 0, 1178373
Total [baseline] (11.017 s) : 0, 11017471
Agent [candidate] (1.191 s) : 0, 1191107
Total [candidate] (11.018 s) : 0, 11017723
section profiling
Agent [baseline] (1.271 s) : 0, 1270696
Total [baseline] (10.893 s) : 0, 10892511
Agent [candidate] (1.276 s) : 0, 1276325
Total [candidate] (10.877 s) : 0, 10876883

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.052 s	-
Agent	appsec	1.198 s	145.474 ms (13.8%)
Agent	iast	1.178 s	126.339 ms (12.0%)
Agent	profiling	1.271 s	218.662 ms (20.8%)
Total	tracing	10.523 s	-
Total	appsec	10.759 s	236.228 ms (2.2%)
Total	iast	11.017 s	494.21 ms (4.7%)
Total	profiling	10.893 s	369.25 ms (3.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.194 s	145.37 ms (13.9%)
Agent	iast	1.191 s	142.306 ms (13.6%)
Agent	profiling	1.276 s	227.525 ms (21.7%)
Total	tracing	10.424 s	-
Total	appsec	10.852 s	428.688 ms (4.1%)
Total	iast	11.018 s	594.162 ms (5.7%)
Total	profiling	10.877 s	453.322 ms (4.3%)

gantt
    title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (724.355 ms) : 0, 724355
BytebuddyAgent [candidate] (720.659 ms) : 0, 720659
GlobalTracer [baseline] (241.188 ms) : 0, 241188
GlobalTracer [candidate] (240.177 ms) : 0, 240177
AppSec [baseline] (55.041 ms) : 0, 55041
AppSec [candidate] (55.361 ms) : 0, 55361
Debugger [baseline] (4.44 ms) : 0, 4440
Debugger [candidate] (5.171 ms) : 0, 5171
Remote Config [baseline] (728.987 µs) : 0, 729
Remote Config [candidate] (718.877 µs) : 0, 719
Telemetry [baseline] (10.125 ms) : 0, 10125
Telemetry [candidate] (10.66 ms) : 0, 10660
section appsec
BytebuddyAgent [baseline] (742.677 ms) : 0, 742677
BytebuddyAgent [candidate] (740.508 ms) : 0, 740508
GlobalTracer [baseline] (238.074 ms) : 0, 238074
GlobalTracer [candidate] (236.785 ms) : 0, 236785
AppSec [baseline] (176.505 ms) : 0, 176505
AppSec [candidate] (176.917 ms) : 0, 176917
Debugger [baseline] (4.32 ms) : 0, 4320
Debugger [candidate] (4.289 ms) : 0, 4289
Remote Config [baseline] (647.933 µs) : 0, 648
Remote Config [candidate] (643.847 µs) : 0, 644
Telemetry [baseline] (8.226 ms) : 0, 8226
Telemetry [candidate] (8.2 ms) : 0, 8200
IAST [baseline] (21.842 ms) : 0, 21842
IAST [candidate] (21.544 ms) : 0, 21544
section iast
BytebuddyAgent [baseline] (839.053 ms) : 0, 839053
BytebuddyAgent [candidate] (849.131 ms) : 0, 849131
GlobalTracer [baseline] (230.565 ms) : 0, 230565
GlobalTracer [candidate] (232.15 ms) : 0, 232150
AppSec [baseline] (56.158 ms) : 0, 56158
AppSec [candidate] (56.709 ms) : 0, 56709
Debugger [baseline] (4.16 ms) : 0, 4160
Debugger [candidate] (4.223 ms) : 0, 4223
Remote Config [baseline] (605.247 µs) : 0, 605
Remote Config [candidate] (613.26 µs) : 0, 613
Telemetry [baseline] (8.912 ms) : 0, 8912
Telemetry [candidate] (8.883 ms) : 0, 8883
IAST [baseline] (22.952 ms) : 0, 22952
IAST [candidate] (23.215 ms) : 0, 23215
section profiling
BytebuddyAgent [baseline] (709.273 ms) : 0, 709273
BytebuddyAgent [candidate] (712.486 ms) : 0, 712486
GlobalTracer [baseline] (350.355 ms) : 0, 350355
GlobalTracer [candidate] (352.076 ms) : 0, 352076
AppSec [baseline] (54.415 ms) : 0, 54415
AppSec [candidate] (53.796 ms) : 0, 53796
Debugger [baseline] (4.266 ms) : 0, 4266
Debugger [candidate] (4.289 ms) : 0, 4289
Remote Config [baseline] (694.293 µs) : 0, 694
Remote Config [candidate] (702.982 µs) : 0, 703
Telemetry [baseline] (8.911 ms) : 0, 8911
Telemetry [candidate] (8.947 ms) : 0, 8947
ProfilingAgent [baseline] (101.258 ms) : 0, 101258
ProfilingAgent [candidate] (102.427 ms) : 0, 102427
Profiling [baseline] (101.285 ms) : 0, 101285
Profiling [candidate] (102.453 ms) : 0, 102453

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-03-27T08:56:34	2025-03-27T09:04:21
git_branch	master	vzakharov/api_sec_sampling_new
git_commit_date	1742990727	1743064463
git_commit_sha	`3e2867a`	`98d957d`
release_version	1.48.0-SNAPSHOT~3e2867a84a	1.48.0-SNAPSHOT~98d957d662
start_time	2025-03-27T08:56:20	2025-03-27T09:04:06

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1743066659	1743066659
ci_job_id	867266665	867266665
ci_pipeline_id	60141941	60141941
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5gca-jya-project-304-concurrent-4-pi7br6z6 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5gca-jya-project-304-concurrent-4-pi7br6z6 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a
    dateFormat X
    axisFormat %s
section baseline
no_agent (386.33 µs) : 366, 406
.   : milestone, 386,
iast (512.573 µs) : 491, 535
.   : milestone, 513,
iast_FULL (733.905 µs) : 712, 756
.   : milestone, 734,
iast_GLOBAL (561.825 µs) : 540, 584
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (520.101 µs) : 498, 542
.   : milestone, 520,
iast_INACTIVE (467.271 µs) : 446, 489
.   : milestone, 467,
iast_TELEMETRY_OFF (504.273 µs) : 482, 526
.   : milestone, 504,
tracing (464.794 µs) : 444, 486
.   : milestone, 465,
section candidate
no_agent (388.877 µs) : 369, 409
.   : milestone, 389,
iast (517.644 µs) : 496, 540
.   : milestone, 518,
iast_FULL (734.471 µs) : 712, 756
.   : milestone, 734,
iast_GLOBAL (561.651 µs) : 540, 583
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (515.678 µs) : 494, 537
.   : milestone, 516,
iast_INACTIVE (468.176 µs) : 447, 490
.   : milestone, 468,
iast_TELEMETRY_OFF (503.655 µs) : 482, 526
.   : milestone, 504,
tracing (466.332 µs) : 444, 488
.   : milestone, 466,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.33 µs [366.334 µs, 406.325 µs]	-
iast	512.573 µs [490.645 µs, 534.502 µs]	126.244 µs (32.7%)
iast_FULL	733.905 µs [711.987 µs, 755.823 µs]	347.575 µs (90.0%)
iast_GLOBAL	561.825 µs [540.102 µs, 583.548 µs]	175.495 µs (45.4%)
iast_HARDCODED_SECRET_DISABLED	520.101 µs [498.059 µs, 542.144 µs]	133.772 µs (34.6%)
iast_INACTIVE	467.271 µs [445.846 µs, 488.695 µs]	80.941 µs (21.0%)
iast_TELEMETRY_OFF	504.273 µs [482.386 µs, 526.16 µs]	117.944 µs (30.5%)
tracing	464.794 µs [443.897 µs, 485.691 µs]	78.464 µs (20.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	388.877 µs [368.905 µs, 408.849 µs]	-
iast	517.644 µs [495.676 µs, 539.612 µs]	128.767 µs (33.1%)
iast_FULL	734.471 µs [712.486 µs, 756.456 µs]	345.594 µs (88.9%)
iast_GLOBAL	561.651 µs [539.845 µs, 583.458 µs]	172.774 µs (44.4%)
iast_HARDCODED_SECRET_DISABLED	515.678 µs [493.998 µs, 537.358 µs]	126.801 µs (32.6%)
iast_INACTIVE	468.176 µs [446.516 µs, 489.836 µs]	79.299 µs (20.4%)
iast_TELEMETRY_OFF	503.655 µs [481.657 µs, 525.653 µs]	114.778 µs (29.5%)
tracing	466.332 µs [444.367 µs, 488.298 µs]	77.455 µs (19.9%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.355 ms) : 1335, 1375
.   : milestone, 1355,
appsec (1.741 ms) : 1718, 1764
.   : milestone, 1741,
appsec_no_iast (1.759 ms) : 1736, 1783
.   : milestone, 1759,
code_origins (1.668 ms) : 1640, 1695
.   : milestone, 1668,
iast (1.517 ms) : 1492, 1541
.   : milestone, 1517,
profiling (1.516 ms) : 1492, 1540
.   : milestone, 1516,
tracing (1.495 ms) : 1469, 1520
.   : milestone, 1495,
section candidate
no_agent (1.368 ms) : 1349, 1388
.   : milestone, 1368,
appsec (1.724 ms) : 1701, 1748
.   : milestone, 1724,
appsec_no_iast (1.743 ms) : 1720, 1767
.   : milestone, 1743,
code_origins (1.666 ms) : 1638, 1694
.   : milestone, 1666,
iast (1.506 ms) : 1481, 1531
.   : milestone, 1506,
profiling (1.527 ms) : 1503, 1551
.   : milestone, 1527,
tracing (1.501 ms) : 1477, 1526
.   : milestone, 1501,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.355 ms [1.335 ms, 1.375 ms]	-
appsec	1.741 ms [1.718 ms, 1.764 ms]	385.629 µs (28.5%)
appsec_no_iast	1.759 ms [1.736 ms, 1.783 ms]	404.26 µs (29.8%)
code_origins	1.668 ms [1.64 ms, 1.695 ms]	312.473 µs (23.1%)
iast	1.517 ms [1.492 ms, 1.541 ms]	161.441 µs (11.9%)
profiling	1.516 ms [1.492 ms, 1.54 ms]	161.005 µs (11.9%)
tracing	1.495 ms [1.469 ms, 1.52 ms]	139.608 µs (10.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.368 ms [1.349 ms, 1.388 ms]	-
appsec	1.724 ms [1.701 ms, 1.748 ms]	355.941 µs (26.0%)
appsec_no_iast	1.743 ms [1.72 ms, 1.767 ms]	374.923 µs (27.4%)
code_origins	1.666 ms [1.638 ms, 1.694 ms]	297.536 µs (21.7%)
iast	1.506 ms [1.481 ms, 1.531 ms]	137.369 µs (10.0%)
profiling	1.527 ms [1.503 ms, 1.551 ms]	158.569 µs (11.6%)
tracing	1.501 ms [1.477 ms, 1.526 ms]	132.911 µs (9.7%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	vzakharov/api_sec_sampling_new
git_commit_date	1742990727	1743064463
git_commit_sha	`3e2867a`	`98d957d`
release_version	1.48.0-SNAPSHOT~3e2867a84a	1.48.0-SNAPSHOT~98d957d662

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1743067133	1743067133
ci_job_id	867266666	867266666
ci_pipeline_id	60141941	60141941
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-5gca-jya-project-304-concurrent-5-jmznmgxs 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-5gca-jya-project-304-concurrent-5-jmznmgxs 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.847 s) : 14847000, 14847000
.   : milestone, 14847000,
appsec (15.082 s) : 15082000, 15082000
.   : milestone, 15082000,
iast (18.317 s) : 18317000, 18317000
.   : milestone, 18317000,
iast_GLOBAL (18.096 s) : 18096000, 18096000
.   : milestone, 18096000,
profiling (15.226 s) : 15226000, 15226000
.   : milestone, 15226000,
tracing (14.896 s) : 14896000, 14896000
.   : milestone, 14896000,
section candidate
no_agent (15.285 s) : 15285000, 15285000
.   : milestone, 15285000,
appsec (14.988 s) : 14988000, 14988000
.   : milestone, 14988000,
iast (19.115 s) : 19115000, 19115000
.   : milestone, 19115000,
iast_GLOBAL (18.181 s) : 18181000, 18181000
.   : milestone, 18181000,
profiling (15.053 s) : 15053000, 15053000
.   : milestone, 15053000,
tracing (14.914 s) : 14914000, 14914000
.   : milestone, 14914000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.847 s [14.847 s, 14.847 s]	-
appsec	15.082 s [15.082 s, 15.082 s]	235.0 ms (1.6%)
iast	18.317 s [18.317 s, 18.317 s]	3.47 s (23.4%)
iast_GLOBAL	18.096 s [18.096 s, 18.096 s]	3.249 s (21.9%)
profiling	15.226 s [15.226 s, 15.226 s]	379.0 ms (2.6%)
tracing	14.896 s [14.896 s, 14.896 s]	49.0 ms (0.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.285 s [15.285 s, 15.285 s]	-
appsec	14.988 s [14.988 s, 14.988 s]	-297.0 ms (-1.9%)
iast	19.115 s [19.115 s, 19.115 s]	3.83 s (25.1%)
iast_GLOBAL	18.181 s [18.181 s, 18.181 s]	2.896 s (18.9%)
profiling	15.053 s [15.053 s, 15.053 s]	-232.0 ms (-1.5%)
tracing	14.914 s [14.914 s, 14.914 s]	-371.0 ms (-2.4%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~98d957d662, baseline=1.48.0-SNAPSHOT~3e2867a84a
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1487
.   : milestone, 1475,
appsec (2.334 ms) : 2290, 2378
.   : milestone, 2334,
iast (2.114 ms) : 2058, 2170
.   : milestone, 2114,
iast_GLOBAL (2.161 ms) : 2105, 2217
.   : milestone, 2161,
profiling (1.978 ms) : 1933, 2023
.   : milestone, 1978,
tracing (1.942 ms) : 1900, 1985
.   : milestone, 1942,
section candidate
no_agent (1.473 ms) : 1462, 1485
.   : milestone, 1473,
appsec (2.343 ms) : 2300, 2387
.   : milestone, 2343,
iast (2.123 ms) : 2067, 2178
.   : milestone, 2123,
iast_GLOBAL (2.161 ms) : 2105, 2217
.   : milestone, 2161,
profiling (1.974 ms) : 1930, 2018
.   : milestone, 1974,
tracing (1.943 ms) : 1901, 1986
.   : milestone, 1943,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.475 ms [1.463 ms, 1.487 ms]	-
appsec	2.334 ms [2.29 ms, 2.378 ms]	859.085 µs (58.2%)
iast	2.114 ms [2.058 ms, 2.17 ms]	639.079 µs (43.3%)
iast_GLOBAL	2.161 ms [2.105 ms, 2.217 ms]	685.877 µs (46.5%)
profiling	1.978 ms [1.933 ms, 2.023 ms]	502.859 µs (34.1%)
tracing	1.942 ms [1.9 ms, 1.985 ms]	467.386 µs (31.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.462 ms, 1.485 ms]	-
appsec	2.343 ms [2.3 ms, 2.387 ms]	869.787 µs (59.0%)
iast	2.123 ms [2.067 ms, 2.178 ms]	649.274 µs (44.1%)
iast_GLOBAL	2.161 ms [2.105 ms, 2.217 ms]	687.698 µs (46.7%)
profiling	1.974 ms [1.93 ms, 2.018 ms]	500.429 µs (34.0%)
tracing	1.943 ms [1.901 ms, 1.986 ms]	469.944 µs (31.9%)

github-actions · 2025-01-14T17:29:36Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

manuel-alvarez-alvarez · 2025-01-15T08:45:05Z

Does the motivation make sense for this PR?

Motivation
This pull request is driven by the need to implement changes based on a newly issued RFC that consolidates all previous specifications. The RFC introduces several minor modifications, including a crucial distinction between user IDs and user logins. By adhering to this updated specification, we aim to enhance our system's accuracy and consistency in tracking user lifecycle events.

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java

jbachorik · 2025-01-15T10:29:39Z

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java

+    long currentTime = System.currentTimeMillis();
+    long hash = computeApiHash(route, method, statusCode);
+
+    synchronized (apiAccessLog) {


I am a bit uneasy about introducing a global bottleneck at a place which might be called from many threads and rather frequently, IIUC.
I know that reducing contention here will require a more complex code because you would not be able to use the trick with the LinkedHashMap but if this is going to be called on a hot-path, the extra complexity might be unavoidable.

I wonder, do you need to keep exact capacity limit or it would be ok if the capacity is mostly obeyed (with some fluctuation around the target value due to concurrent updates/cleanups)? If the exact capacity is not required it should help with a simpler implementation.

Reworked implementation of ApiAccessTracker to avoid bottleneck

@jbachorik If you can have a look: now there's a concurrent hashmap for the access map, which is only written to from a single thread (post-processing, during serialization) with a 1s timeout (but usually should be ms). At request end, we do a preliminary sampling decision based on a lookup to that map, and use an atomic counter to sample at most 4 concurrently.

smola

I'm still half-way the review, but flushing my current comments since I have to pause.

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/PostProcessingCounter.java

dd-java-agent/appsec/src/main/java/com/datadog/appsec/gateway/GatewayBridge.java

...va-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiSecurityRequestSampler.java

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java

manuel-alvarez-alvarez

LGTM (great job!)

…ialization thread

ValentinZakharov self-assigned this Jan 10, 2025

ValentinZakharov added the comp: asm waf Application Security Management (WAF) label Jan 10, 2025

ValentinZakharov changed the title ~~Implemented new API Security Sampling~~ Implemented New API Security Sampling mechanism Jan 10, 2025

ValentinZakharov marked this pull request as ready for review January 14, 2025 17:29

ValentinZakharov requested a review from a team as a code owner January 14, 2025 17:29

ValentinZakharov requested review from smola and manuel-alvarez-alvarez January 14, 2025 17:29

manuel-alvarez-alvarez reviewed Jan 15, 2025

View reviewed changes

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java Outdated Show resolved Hide resolved

manuel-alvarez-alvarez reviewed Jan 15, 2025

View reviewed changes

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java Outdated Show resolved Hide resolved

manuel-alvarez-alvarez reviewed Jan 15, 2025

View reviewed changes

dd-java-agent/appsec/src/main/java/com/datadog/appsec/api/security/ApiAccessTracker.java Outdated Show resolved Hide resolved

manuel-alvarez-alvarez approved these changes Jan 15, 2025

View reviewed changes

jbachorik reviewed Jan 15, 2025

View reviewed changes

smola added comp: asm waf Application Security Management (WAF) and removed comp: asm waf Application Security Management (WAF) labels Jan 28, 2025

ValentinZakharov requested a review from a team as a code owner January 28, 2025 18:01

ValentinZakharov force-pushed the vzakharov/api_sec_sampling_new branch from 08d9b7f to bc70627 Compare January 28, 2025 18:32

ValentinZakharov force-pushed the vzakharov/api_sec_sampling_new branch from bc70627 to 6d5c67d Compare February 12, 2025 12:16

smola requested changes Feb 12, 2025

View reviewed changes

ValentinZakharov force-pushed the vzakharov/api_sec_sampling_new branch from 6d5c67d to ae09aa4 Compare February 12, 2025 18:22

smola marked this pull request as draft February 24, 2025 16:58

smola force-pushed the vzakharov/api_sec_sampling_new branch from ae09aa4 to de0e3ef Compare February 24, 2025 17:53

smola mentioned this pull request Feb 25, 2025

Extend NonBlockingSemaphore and make release for 1-permit case atomic #8446

Merged

smola unassigned ValentinZakharov Feb 25, 2025

smola force-pushed the vzakharov/api_sec_sampling_new branch 4 times, most recently from 4be5bfb to 92a91b3 Compare February 27, 2025 14:12

smola force-pushed the vzakharov/api_sec_sampling_new branch 2 times, most recently from b1824fa to 0327ec4 Compare March 19, 2025 11:34

smola approved these changes Mar 20, 2025

View reviewed changes

smola force-pushed the vzakharov/api_sec_sampling_new branch from 0327ec4 to 77e1a96 Compare March 20, 2025 09:23

smola requested a review from manuel-alvarez-alvarez March 20, 2025 14:28

sezen-datadog approved these changes Mar 21, 2025

View reviewed changes

manuel-alvarez-alvarez approved these changes Mar 25, 2025

View reviewed changes

smola force-pushed the vzakharov/api_sec_sampling_new branch from 77e1a96 to b41b321 Compare March 25, 2025 17:13

smola mentioned this pull request Mar 25, 2025

[java] Enable new API Security sampling tests DataDog/system-tests#4195

Merged

7 tasks

ValentinZakharov and others added 6 commits March 26, 2025 15:58

Added AppSec post-processing

fef2c64

Api Security schema computation moved to post-processing stage in ser…

21a2975

…ialization thread

Implemented time-based sampling per endpoint

0c2cec6

Added preSampler for short-circuit Api Security

30a1592

Added post-processing limit

6a71986

wip

64539ed

smola force-pushed the vzakharov/api_sec_sampling_new branch 2 times, most recently from c3a3ff5 to 4ea51f8 Compare March 26, 2025 16:28

smola added 9 commits March 27, 2025 09:34

ST commit

d4b08e3

wip

5e32bf8

wip

e9d1e75

wip

3551464

wip

21e1114

Remove excessive logs and tweak condition

7ecd889

typo

c429be8

fix test

5488b19

fix merge conflict

98d957d

smola force-pushed the vzakharov/api_sec_sampling_new branch from 4ea51f8 to 98d957d Compare March 27, 2025 08:34

smola merged commit 75e5963 into master Mar 27, 2025
248 of 267 checks passed

smola deleted the vzakharov/api_sec_sampling_new branch March 27, 2025 13:59

github-actions bot added this to the 1.48.0 milestone Mar 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New API Security sampling algorithm #8178

New API Security sampling algorithm #8178

ValentinZakharov commented Jan 10, 2025 •

edited by smola

Loading

pr-commenter bot commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

manuel-alvarez-alvarez commented Jan 15, 2025

jbachorik Jan 15, 2025

ValentinZakharov Jan 27, 2025 •

edited

Loading

smola Mar 6, 2025

smola left a comment

manuel-alvarez-alvarez left a comment

New API Security sampling algorithm #8178

New API Security sampling algorithm #8178

Conversation

ValentinZakharov commented Jan 10, 2025 • edited by smola Loading

What Does This Do

Endpoint-based API Security sampling

Late processing

Other changes

Contributor Checklist

pr-commenter bot commented Jan 10, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

github-actions bot commented Jan 14, 2025 • edited Loading

manuel-alvarez-alvarez commented Jan 15, 2025

jbachorik Jan 15, 2025

Choose a reason for hiding this comment

ValentinZakharov Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

smola Mar 6, 2025

Choose a reason for hiding this comment

smola left a comment

Choose a reason for hiding this comment

manuel-alvarez-alvarez left a comment

Choose a reason for hiding this comment

ValentinZakharov commented Jan 10, 2025 •

edited by smola

Loading

pr-commenter bot commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jan 14, 2025 •

edited

Loading

ValentinZakharov Jan 27, 2025 •

edited

Loading