Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update muzzleCheck for kafka #7912

Closed
wants to merge 2 commits into from
Closed

Update muzzleCheck for kafka #7912

wants to merge 2 commits into from

Conversation

nayeem-kamal
Copy link
Contributor

@nayeem-kamal nayeem-kamal commented Nov 7, 2024 •
edited
Loading

What Does This Do

Adds muzzleChecks to prevent double instrumenting of kafka.

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@nayeem-kamal nayeem-kamal added the inst: kafka Kafka instrumentation label Nov 7, 2024
@nayeem-kamal nayeem-kamal marked this pull request as ready for review November 7, 2024 22:05
@nayeem-kamal nayeem-kamal requested a review from a team as a code owner November 7, 2024 22:05
@nayeem-kamal nayeem-kamal enabled auto-merge (squash) November 7, 2024 22:05
piochelepiotr
piochelepiotr reviewed Nov 7, 2024
View reviewed changes
Copy link
Contributor

@piochelepiotr piochelepiotr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also update the 0.11 version of kafka-clients to have a max version instrumented?

I think with this change, versions 3.8.0+ could still be double instrumented

@pr-commenter
Copy link

pr-commenter bot commented Nov 7, 2024 •
edited
Loading

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-muzzle-fix2
git_commit_date 1731003381 1732284508
git_commit_sha 7680873 c62acdc
See matching parameters
Baseline Candidate
ci_job_date 1732285665 1732285665
ci_job_id 717095608 717095608
ci_pipeline_id 49730691 49730691
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce unsure
[+6403.034op/s; +81458.174op/s] or [+0.358%; +4.559%]
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Nov 7, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-muzzle-fix2
git_commit_date 1732280130 1732284508
git_commit_sha 9078a8c c62acdc
release_version 1.43.0-SNAPSHOT~9078a8c473 1.43.0-SNAPSHOT~c62acdc4f7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732287053 1732287053
ci_job_id 717095603 717095603
ci_pipeline_id 49730691 49730691
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 9 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.091 s) : 0, 1091140
Total [baseline] (8.662 s) : 0, 8661705
Agent [candidate] (1.099 s) : 0, 1098716
Total [candidate] (8.686 s) : 0, 8686043
section iast
Agent [baseline] (1.226 s) : 0, 1225722
Total [baseline] (9.254 s) : 0, 9253949
Agent [candidate] (1.215 s) : 0, 1215376
Total [candidate] (9.186 s) : 0, 9185879
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.216 s) : 0, 1215808
Total [baseline] (9.197 s) : 0, 9196541
Agent [candidate] (1.221 s) : 0, 1220844
Total [candidate] (9.18 s) : 0, 9179829
section iast_TELEMETRY_OFF
Agent [baseline] (1.221 s) : 0, 1220919
Total [baseline] (9.187 s) : 0, 9187355
Agent [candidate] (1.211 s) : 0, 1211170
Total [candidate] (9.19 s) : 0, 9189990
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.091 s -
Agent iast 1.226 s 134.582 ms (12.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.216 s 124.668 ms (11.4%)
Agent iast_TELEMETRY_OFF 1.221 s 129.779 ms (11.9%)
Total tracing 8.662 s -
Total iast 9.254 s 592.244 ms (6.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.197 s 534.836 ms (6.2%)
Total iast_TELEMETRY_OFF 9.187 s 525.65 ms (6.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.099 s -
Agent iast 1.215 s 116.66 ms (10.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.221 s 122.128 ms (11.1%)
Agent iast_TELEMETRY_OFF 1.211 s 112.454 ms (10.2%)
Total tracing 8.686 s -
Total iast 9.186 s 499.836 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.18 s 493.786 ms (5.7%)
Total iast_TELEMETRY_OFF 9.19 s 503.947 ms (5.8%) 
gantt
    title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.662 ms) : 0, 693662
BytebuddyAgent [candidate] (700.377 ms) : 0, 700377
GlobalTracer [baseline] (317.187 ms) : 0, 317187
GlobalTracer [candidate] (318.727 ms) : 0, 318727
AppSec [baseline] (54.506 ms) : 0, 54506
AppSec [candidate] (54.888 ms) : 0, 54888
Remote Config [baseline] (678.242 µs) : 0, 678
Remote Config [candidate] (693.925 µs) : 0, 694
Telemetry [baseline] (11.409 ms) : 0, 11409
Telemetry [candidate] (10.137 ms) : 0, 10137
section iast
BytebuddyAgent [baseline] (815.476 ms) : 0, 815476
BytebuddyAgent [candidate] (809.15 ms) : 0, 809150
GlobalTracer [baseline] (308.515 ms) : 0, 308515
GlobalTracer [candidate] (306.35 ms) : 0, 306350
AppSec [baseline] (56.874 ms) : 0, 56874
AppSec [candidate] (57.737 ms) : 0, 57737
IAST [baseline] (22.827 ms) : 0, 22827
IAST [candidate] (20.345 ms) : 0, 20345
Remote Config [baseline] (635.279 µs) : 0, 635
Remote Config [candidate] (613.994 µs) : 0, 614
Telemetry [baseline] (7.6 ms) : 0, 7600
Telemetry [candidate] (7.482 ms) : 0, 7482
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (808.948 ms) : 0, 808948
BytebuddyAgent [candidate] (813.543 ms) : 0, 813543
GlobalTracer [baseline] (306.585 ms) : 0, 306585
GlobalTracer [candidate] (306.503 ms) : 0, 306503
AppSec [baseline] (57.849 ms) : 0, 57849
AppSec [candidate] (58.109 ms) : 0, 58109
IAST [baseline] (20.668 ms) : 0, 20668
IAST [candidate] (20.865 ms) : 0, 20865
Remote Config [baseline] (612.784 µs) : 0, 613
Remote Config [candidate] (621.853 µs) : 0, 622
Telemetry [baseline] (7.461 ms) : 0, 7461
Telemetry [candidate] (7.432 ms) : 0, 7432
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (811.839 ms) : 0, 811839
BytebuddyAgent [candidate] (805.378 ms) : 0, 805378
GlobalTracer [baseline] (308.469 ms) : 0, 308469
GlobalTracer [candidate] (305.645 ms) : 0, 305645
AppSec [baseline] (58.175 ms) : 0, 58175
AppSec [candidate] (58.315 ms) : 0, 58315
IAST [baseline] (20.582 ms) : 0, 20582
IAST [candidate] (20.075 ms) : 0, 20075
Remote Config [baseline] (627.179 µs) : 0, 627
Remote Config [candidate] (612.862 µs) : 0, 613
Telemetry [baseline] (7.416 ms) : 0, 7416
Telemetry [candidate] (7.416 ms) : 0, 7416
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.088 s) : 0, 1088157
Total [baseline] (10.517 s) : 0, 10516945
Agent [candidate] (1.093 s) : 0, 1092506
Total [candidate] (10.487 s) : 0, 10487115
section appsec
Agent [baseline] (1.224 s) : 0, 1224478
Total [baseline] (10.703 s) : 0, 10702724
Agent [candidate] (1.221 s) : 0, 1220792
Total [candidate] (10.681 s) : 0, 10680584
section iast
Agent [baseline] (1.214 s) : 0, 1214080
Total [baseline] (10.938 s) : 0, 10937763
Agent [candidate] (1.216 s) : 0, 1215763
Total [candidate] (10.986 s) : 0, 10986382
section profiling
Agent [baseline] (1.286 s) : 0, 1285876
Total [baseline] (10.768 s) : 0, 10767725
Agent [candidate] (1.287 s) : 0, 1286854
Total [candidate] (10.823 s) : 0, 10823466
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.088 s -
Agent appsec 1.224 s 136.321 ms (12.5%)
Agent iast 1.214 s 125.923 ms (11.6%)
Agent profiling 1.286 s 197.719 ms (18.2%)
Total tracing 10.517 s -
Total appsec 10.703 s 185.778 ms (1.8%)
Total iast 10.938 s 420.817 ms (4.0%)
Total profiling 10.768 s 250.78 ms (2.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.093 s -
Agent appsec 1.221 s 128.286 ms (11.7%)
Agent iast 1.216 s 123.257 ms (11.3%)
Agent profiling 1.287 s 194.349 ms (17.8%)
Total tracing 10.487 s -
Total appsec 10.681 s 193.47 ms (1.8%)
Total iast 10.986 s 499.268 ms (4.8%)
Total profiling 10.823 s 336.352 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.919 ms) : 0, 690919
BytebuddyAgent [candidate] (694.229 ms) : 0, 694229
GlobalTracer [baseline] (316.147 ms) : 0, 316147
GlobalTracer [candidate] (318.083 ms) : 0, 318083
AppSec [baseline] (54.688 ms) : 0, 54688
AppSec [candidate] (54.449 ms) : 0, 54449
Remote Config [baseline] (683.328 µs) : 0, 683
Remote Config [candidate] (690.46 µs) : 0, 690
Telemetry [baseline] (12.047 ms) : 0, 12047
Telemetry [candidate] (11.401 ms) : 0, 11401
section appsec
BytebuddyAgent [baseline] (710.943 ms) : 0, 710943
BytebuddyAgent [candidate] (709.054 ms) : 0, 709054
GlobalTracer [baseline] (314.389 ms) : 0, 314389
GlobalTracer [candidate] (314.089 ms) : 0, 314089
AppSec [baseline] (167.419 ms) : 0, 167419
AppSec [candidate] (165.694 ms) : 0, 165694
Remote Config [baseline] (639.99 µs) : 0, 640
Remote Config [candidate] (643.037 µs) : 0, 643
Telemetry [baseline] (7.908 ms) : 0, 7908
Telemetry [candidate] (7.501 ms) : 0, 7501
IAST [baseline] (18.943 ms) : 0, 18943
IAST [candidate] (20.288 ms) : 0, 20288
section iast
BytebuddyAgent [baseline] (807.353 ms) : 0, 807353
BytebuddyAgent [candidate] (808.827 ms) : 0, 808827
GlobalTracer [baseline] (305.938 ms) : 0, 305938
GlobalTracer [candidate] (306.642 ms) : 0, 306642
AppSec [baseline] (58.101 ms) : 0, 58101
AppSec [candidate] (56.073 ms) : 0, 56073
Remote Config [baseline] (607.183 µs) : 0, 607
Remote Config [candidate] (611.857 µs) : 0, 612
Telemetry [baseline] (7.515 ms) : 0, 7515
Telemetry [candidate] (7.481 ms) : 0, 7481
IAST [baseline] (20.945 ms) : 0, 20945
IAST [candidate] (22.438 ms) : 0, 22438
section profiling
BytebuddyAgent [baseline] (685.816 ms) : 0, 685816
BytebuddyAgent [candidate] (686.51 ms) : 0, 686510
GlobalTracer [baseline] (400.212 ms) : 0, 400212
GlobalTracer [candidate] (400.069 ms) : 0, 400069
AppSec [baseline] (55.108 ms) : 0, 55108
AppSec [candidate] (54.948 ms) : 0, 54948
Remote Config [baseline] (673.12 µs) : 0, 673
Remote Config [candidate] (675.406 µs) : 0, 675
Telemetry [baseline] (13.429 ms) : 0, 13429
Telemetry [candidate] (10.143 ms) : 0, 10143
ProfilingAgent [baseline] (91.746 ms) : 0, 91746
ProfilingAgent [candidate] (95.58 ms) : 0, 95580
Profiling [baseline] (91.77 ms) : 0, 91770
Profiling [candidate] (95.604 ms) : 0, 95604
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-11-22T14:21:02 2024-11-22T14:28:00
git_branch master nayeem-kamal/kafka-muzzle-fix2
git_commit_date 1732280130 1732284508
git_commit_sha 9078a8c c62acdc
release_version 1.43.0-SNAPSHOT~9078a8c473 1.43.0-SNAPSHOT~c62acdc4f7
start_time 2024-11-22T14:20:49 2024-11-22T14:27:46
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1732286033 1732286033
ci_job_id 717095604 717095604
ci_pipeline_id 49730691 49730691
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.471 µs) : 352, 393
.   : milestone, 372,
iast (487.747 µs) : 466, 509
.   : milestone, 488,
iast_FULL (657.579 µs) : 636, 679
.   : milestone, 658,
iast_GLOBAL (517.443 µs) : 496, 539
.   : milestone, 517,
iast_HARDCODED_SECRET_DISABLED (492.915 µs) : 471, 514
.   : milestone, 493,
iast_INACTIVE (456.85 µs) : 435, 479
.   : milestone, 457,
iast_TELEMETRY_OFF (482.126 µs) : 460, 504
.   : milestone, 482,
tracing (450.009 µs) : 429, 471
.   : milestone, 450,
section candidate
no_agent (373.292 µs) : 353, 394
.   : milestone, 373,
iast (496.561 µs) : 475, 518
.   : milestone, 497,
iast_FULL (649.929 µs) : 629, 671
.   : milestone, 650,
iast_GLOBAL (515.155 µs) : 494, 536
.   : milestone, 515,
iast_HARDCODED_SECRET_DISABLED (493.348 µs) : 472, 515
.   : milestone, 493,
iast_INACTIVE (452.203 µs) : 431, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (482.587 µs) : 461, 504
.   : milestone, 483,
tracing (451.357 µs) : 430, 473
.   : milestone, 451,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.471 µs [351.623 µs, 393.32 µs] -
iast 487.747 µs [466.427 µs, 509.067 µs] 115.275 µs (30.9%)
iast_FULL 657.579 µs [636.293 µs, 678.866 µs] 285.108 µs (76.5%)
iast_GLOBAL 517.443 µs [496.162 µs, 538.724 µs] 144.972 µs (38.9%)
iast_HARDCODED_SECRET_DISABLED 492.915 µs [471.344 µs, 514.485 µs] 120.443 µs (32.3%)
iast_INACTIVE 456.85 µs [435.155 µs, 478.544 µs] 84.378 µs (22.7%)
iast_TELEMETRY_OFF 482.126 µs [460.22 µs, 504.031 µs] 109.654 µs (29.4%)
tracing 450.009 µs [428.796 µs, 471.221 µs] 77.537 µs (20.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 373.292 µs [353.01 µs, 393.575 µs] -
iast 496.561 µs [475.069 µs, 518.053 µs] 123.269 µs (33.0%)
iast_FULL 649.929 µs [628.535 µs, 671.322 µs] 276.636 µs (74.1%)
iast_GLOBAL 515.155 µs [493.917 µs, 536.393 µs] 141.863 µs (38.0%)
iast_HARDCODED_SECRET_DISABLED 493.348 µs [472.124 µs, 514.573 µs] 120.056 µs (32.2%)
iast_INACTIVE 452.203 µs [431.324 µs, 473.082 µs] 78.911 µs (21.1%)
iast_TELEMETRY_OFF 482.587 µs [460.737 µs, 504.437 µs] 109.295 µs (29.3%)
tracing 451.357 µs [430.062 µs, 472.653 µs] 78.065 µs (20.9%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.351 ms) : 1331, 1371
.   : milestone, 1351,
appsec (1.766 ms) : 1741, 1791
.   : milestone, 1766,
appsec_no_iast (1.751 ms) : 1726, 1776
.   : milestone, 1751,
iast (1.491 ms) : 1468, 1514
.   : milestone, 1491,
profiling (1.495 ms) : 1473, 1518
.   : milestone, 1495,
tracing (1.51 ms) : 1486, 1535
.   : milestone, 1510,
section candidate
no_agent (1.351 ms) : 1331, 1372
.   : milestone, 1351,
appsec (1.748 ms) : 1722, 1773
.   : milestone, 1748,
appsec_no_iast (1.759 ms) : 1735, 1783
.   : milestone, 1759,
iast (1.495 ms) : 1471, 1518
.   : milestone, 1495,
profiling (1.523 ms) : 1499, 1548
.   : milestone, 1523,
tracing (1.492 ms) : 1467, 1517
.   : milestone, 1492,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.351 ms [1.331 ms, 1.371 ms] -
appsec 1.766 ms [1.741 ms, 1.791 ms] 414.928 µs (30.7%)
appsec_no_iast 1.751 ms [1.726 ms, 1.776 ms] 399.573 µs (29.6%)
iast 1.491 ms [1.468 ms, 1.514 ms] 140.026 µs (10.4%)
profiling 1.495 ms [1.473 ms, 1.518 ms] 144.166 µs (10.7%)
tracing 1.51 ms [1.486 ms, 1.535 ms] 159.322 µs (11.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.351 ms [1.331 ms, 1.372 ms] -
appsec 1.748 ms [1.722 ms, 1.773 ms] 396.377 µs (29.3%)
appsec_no_iast 1.759 ms [1.735 ms, 1.783 ms] 407.812 µs (30.2%)
iast 1.495 ms [1.471 ms, 1.518 ms] 143.471 µs (10.6%)
profiling 1.523 ms [1.499 ms, 1.548 ms] 172.267 µs (12.7%)
tracing 1.492 ms [1.467 ms, 1.517 ms] 140.727 µs (10.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-muzzle-fix2
git_commit_date 1732280130 1732284508
git_commit_sha 9078a8c c62acdc
release_version 1.43.0-SNAPSHOT~9078a8c473 1.43.0-SNAPSHOT~c62acdc4f7
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1732286599 1732286599
ci_job_id 717095605 717095605
ci_pipeline_id 49730691 49730691
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1458, 1481
.   : milestone, 1470,
appsec (2.347 ms) : 2306, 2388
.   : milestone, 2347,
iast (2.083 ms) : 2031, 2136
.   : milestone, 2083,
iast_GLOBAL (2.125 ms) : 2072, 2177
.   : milestone, 2125,
profiling (1.946 ms) : 1904, 1987
.   : milestone, 1946,
tracing (1.941 ms) : 1901, 1982
.   : milestone, 1941,
section candidate
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.344 ms) : 2303, 2386
.   : milestone, 2344,
iast (2.092 ms) : 2039, 2144
.   : milestone, 2092,
iast_GLOBAL (2.135 ms) : 2082, 2188
.   : milestone, 2135,
profiling (1.955 ms) : 1911, 1998
.   : milestone, 1955,
tracing (1.933 ms) : 1893, 1973
.   : milestone, 1933,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.47 ms [1.458 ms, 1.481 ms] -
appsec 2.347 ms [2.306 ms, 2.388 ms] 876.806 µs (59.7%)
iast 2.083 ms [2.031 ms, 2.136 ms] 613.628 µs (41.7%)
iast_GLOBAL 2.125 ms [2.072 ms, 2.177 ms] 654.729 µs (44.5%)
profiling 1.946 ms [1.904 ms, 1.987 ms] 475.951 µs (32.4%)
tracing 1.941 ms [1.901 ms, 1.982 ms] 471.693 µs (32.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.461 ms, 1.484 ms] -
appsec 2.344 ms [2.303 ms, 2.386 ms] 871.867 µs (59.2%)
iast 2.092 ms [2.039 ms, 2.144 ms] 619.111 µs (42.0%)
iast_GLOBAL 2.135 ms [2.082 ms, 2.188 ms] 662.739 µs (45.0%)
profiling 1.955 ms [1.911 ms, 1.998 ms] 482.206 µs (32.7%)
tracing 1.933 ms [1.893 ms, 1.973 ms] 460.927 µs (31.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~c62acdc4f7, baseline=1.43.0-SNAPSHOT~9078a8c473
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.61 s) : 15610000, 15610000
.   : milestone, 15610000,
appsec (15.062 s) : 15062000, 15062000
.   : milestone, 15062000,
iast (18.753 s) : 18753000, 18753000
.   : milestone, 18753000,
iast_GLOBAL (17.965 s) : 17965000, 17965000
.   : milestone, 17965000,
profiling (14.961 s) : 14961000, 14961000
.   : milestone, 14961000,
tracing (14.95 s) : 14950000, 14950000
.   : milestone, 14950000,
section candidate
no_agent (15.43 s) : 15430000, 15430000
.   : milestone, 15430000,
appsec (15.109 s) : 15109000, 15109000
.   : milestone, 15109000,
iast (18.48 s) : 18480000, 18480000
.   : milestone, 18480000,
iast_GLOBAL (18.157 s) : 18157000, 18157000
.   : milestone, 18157000,
profiling (14.775 s) : 14775000, 14775000
.   : milestone, 14775000,
tracing (14.762 s) : 14762000, 14762000
.   : milestone, 14762000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.61 s [15.61 s, 15.61 s] -
appsec 15.062 s [15.062 s, 15.062 s] -548.0 ms (-3.5%)
iast 18.753 s [18.753 s, 18.753 s] 3.143 s (20.1%)
iast_GLOBAL 17.965 s [17.965 s, 17.965 s] 2.355 s (15.1%)
profiling 14.961 s [14.961 s, 14.961 s] -649.0 ms (-4.2%)
tracing 14.95 s [14.95 s, 14.95 s] -660.0 ms (-4.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.43 s [15.43 s, 15.43 s] -
appsec 15.109 s [15.109 s, 15.109 s] -321.0 ms (-2.1%)
iast 18.48 s [18.48 s, 18.48 s] 3.05 s (19.8%)
iast_GLOBAL 18.157 s [18.157 s, 18.157 s] 2.727 s (17.7%)
profiling 14.775 s [14.775 s, 14.775 s] -655.0 ms (-4.2%)
tracing 14.762 s [14.762 s, 14.762 s] -668.0 ms (-4.3%)

@pr-commenter
Copy link

pr-commenter bot commented Nov 7, 2024 •
edited
Loading

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nayeem-kamal/kafka-muzzle-fix2
git_commit_date 1731003381 1732284508
git_commit_sha 7680873 c62acdc
See matching parameters
Baseline Candidate
ci_job_date 1732285706 1732285706
ci_job_id 717095609 717095609
ci_pipeline_id 49730691 49730691
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume same

@PerfectSlayer PerfectSlayer added the tag: no release notes Changes to exclude from release notes label Nov 8, 2024
amarziali
amarziali requested changes Nov 12, 2024
View reviewed changes
Copy link
Collaborator

@amarziali amarziali left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not enough because 0.11+ also needs to be checked as uniquely applicable.

Also, instrumenting kafka 3.7 with the codebase currently in master fails because

java.lang.IllegalStateException: Cannot locate field named metadata for class org.apache.kafka.clients.consumer.KafkaConsumer
	at net.bytebuddy.asm.Advice$OffsetMapping$ForField$Unresolved.resolve(Advice.java:2556)
	at net.bytebuddy.asm.Advice$OffsetMapping$ForField.resolve(Advice.java:2481)
	at net.bytebuddy.asm.Advice$Dispatcher$Inlining$Resolved$ForMethodExit.doApply(Advice.java:9876)

I don't think this PR prevents it.
Muzzle (or narrower matchers) should be used on 0.11+ to prevent this kind of issues. Additionally this shows a potential range of versions we still not support properly

@amarziali
Copy link
Collaborator

#8006 seems have been opened for the same goal. Can this be closed?

auto-merge was automatically disabled November 25, 2024 14:48

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@piochelepiotr piochelepiotr piochelepiotr left review comments

@amarziali amarziali amarziali requested changes

Assignees
No one assigned
Labels
inst: kafka Kafka instrumentation tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@nayeem-kamal @amarziali @piochelepiotr @PerfectSlayer