feat(datadog): support forwarder_max_concurrent_requests

[aqian01]

What changed

Implements forwarder_max_concurrent_requests as the ADP forwarder endpoint concurrency setting for #1363.

• Reads forwarder_max_concurrent_requests into ForwarderConfiguration::endpoint_concurrency().
• Defaults endpoint concurrency to 10.
• Clamps 0 to 1 at accessor read time.
• Keeps the existing Tower .concurrency_limit(...) path and feeds it with the new setting.
• Removes forwarder_num_workers from ADP runtime config usage and marks it unsupported in docs/registry.
• Updates DogStatsD config docs and known-configs.json bookkeeping.

Why

ADP previously used forwarder_num_workers as the request concurrency setting, which did not match the Agent-facing config key for max concurrent outbound requests. This change makes forwarder_max_concurrent_requests the setting that controls ADP request concurrency.

Validation

Existing tests

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 6b23552 · Comparison: 7b60f6e · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.85 MiB (baseline) vs 37.85 MiB (comparison)
Size Change: -1.32 KiB (-0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
[Unmapped]	-2.35 KiB	1
anon.40110ee4bd9608a39c1dd63b7f5b13b9.156.llvm.2885460765385634042	+1.77 KiB	1
anon.b01ba6cee13ac936e5fe610d69a43414.156.llvm.3215639740667967730	-1.77 KiB	1
hashbrown	-1.67 KiB	44
anon.7eba1147330e6af375aeca2095e184d8.195.llvm.13430023222257517555	+1.35 KiB	1
anon.e3991f6ee449c3978b796a4b98c14eea.95.llvm.10534094001743460907	+1.35 KiB	1
anon.6f639be05e5476387366a40f60343d78.25.llvm.16756786403889059728	-1.35 KiB	1
anon.fceebb1b190d754f5128ba5f77e15ef9.137.llvm.2209156483589783128	-1.35 KiB	1
anon.0bde35f04fb057f7fc3941e3fc00018c.27.llvm.8326817328543904745	+1.33 KiB	1
anon.7eba1147330e6af375aeca2095e184d8.197.llvm.15780357041662117167	-1.33 KiB	1
anon.6f639be05e5476387366a40f60343d78.87.llvm.16756786403889059728	-1.32 KiB	1
anon.71548e082b2f5cfdba840d3b6a3db4a4.70.llvm.6881307420077850029	+1.32 KiB	1
anon.22e7a231d17f6601496c3bc90bbe626f.1275.llvm.2079349722348780431	+1.16 KiB	1
anon.d241dc6c2b4a0195067bac11bf6ecab6.1278.llvm.6907558562252841839	-1.16 KiB	1
anon.ab6251514ea18fe472d05926b62a34b0.25.llvm.8776330277072634164	+1.15 KiB	1
anon.7eba1147330e6af375aeca2095e184d8.1366.llvm.15780357041662117167	-1.15 KiB	1
anon.16bf0c39c1db2119a81e25dc03e1d63b.174.llvm.16466219259657594562	+958 B	1
anon.f32c1fae1ff6402ce558bcee3265801f.1061.llvm.467089701737411494	-956 B	1
anon.6f639be05e5476387366a40f60343d78.708.llvm.16756786403889059728	-913 B	1
anon.7c80d08f2094c3773fb79bab58b68f02.28.llvm.4491143641773496644	+912 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +2.95Ki  [NEW]     +61    core::ptr::drop_in_place<std::sync::poison::PoisonError<std::sync::poison::rwlock::RwLockReadGuard<quick_cache::shard::CacheShard<stringtheory::MetaString,core::option::Option<saluki_components::transforms::dogstatsd_mapper::CachedMapResult>,saluki_common::cache::weight::WrappedWeighter<saluki_common::cache::weight::ItemCountWeighter>,foldhash::quality::RandomState,saluki_common::cache::expiry::ExpiryCapableLifecycle<stringtheory::MetaString>,alloc::sync::Arc<quick_cache::sync_placeholder::Placeholder<core::option::Option<saluki_components::transforms::dogstatsd_mapper::CachedMapResult>>>>>>>::h0eb5c7a41a5d125f
  +0.1% +2.03Ki  -0.1% -1.54Ki    [4054 Others]
  [NEW] +1.77Ki  [NEW] +1.68Ki    anon.40110ee4bd9608a39c1dd63b7f5b13b9.156.llvm.2885460765385634042
  [NEW] +1.56Ki  [NEW] +1.39Ki    serde_path_to_error::de::_<impl serde_core::de::Visitor for serde_path_to_error::wrap::Wrap<X>>::visit_map::hade301bf60f8cd14
  [NEW] +1.35Ki  [NEW]     +22    anon.7eba1147330e6af375aeca2095e184d8.195.llvm.13430023222257517555
  [NEW] +1.35Ki  [NEW]     +22    anon.e3991f6ee449c3978b796a4b98c14eea.95.llvm.10534094001743460907
  [NEW] +1.33Ki  [NEW]     +92    anon.0bde35f04fb057f7fc3941e3fc00018c.27.llvm.8326817328543904745
  [NEW] +1.32Ki  [NEW]     +88    anon.71548e082b2f5cfdba840d3b6a3db4a4.70.llvm.6881307420077850029
  [NEW] +1.23Ki  [NEW]    +468    core::ptr::drop_in_place<alloc::boxed::Box<tokio::runtime::task::core::Cell<tokio::runtime::blocking::task::BlockingTask<tokio::fs::read_dir::read_dir<&std::path::PathBuf>::{{closure}}::{{closure}}>,tokio::runtime::blocking::schedule::BlockingSchedule>>>::h195cfec737ce94e9
  [NEW] +1.16Ki  [NEW]     +91    anon.22e7a231d17f6601496c3bc90bbe626f.1275.llvm.2079349722348780431
  [DEL] -1.15Ki  [DEL]     -87    anon.7eba1147330e6af375aeca2095e184d8.1366.llvm.15780357041662117167
  [DEL] -1.16Ki  [DEL]     -91    anon.d241dc6c2b4a0195067bac11bf6ecab6.1278.llvm.6907558562252841839
  [DEL] -1.23Ki  [DEL]    -468    core::ptr::drop_in_place<alloc::boxed::Box<tokio::runtime::task::core::Cell<tokio::runtime::blocking::task::BlockingTask<tokio::fs::read_dir::read_dir<&std::path::Path>::{{closure}}::{{closure}}>,tokio::runtime::blocking::schedule::BlockingSchedule>>>::h3c5ff075a48a8fd5
  [DEL] -1.32Ki  [DEL]     -88    anon.6f639be05e5476387366a40f60343d78.87.llvm.16756786403889059728
  [DEL] -1.33Ki  [DEL]     -92    anon.7eba1147330e6af375aeca2095e184d8.197.llvm.15780357041662117167
  [DEL] -1.35Ki  [DEL]     -22    anon.6f639be05e5476387366a40f60343d78.25.llvm.16756786403889059728
  [DEL] -1.35Ki  [DEL]     -22    anon.fceebb1b190d754f5128ba5f77e15ef9.137.llvm.2209156483589783128
 -70.3% -1.42Ki -76.4% -1.42Ki    _<serde_path_to_error::de::Deserializer<D> as serde_core::de::Deserializer>::deserialize_struct::h90408cb8e5e12a88
  [DEL] -1.77Ki  [DEL] -1.68Ki    anon.b01ba6cee13ac936e5fe610d69a43414.156.llvm.3215639740667967730
 -39.2% -2.35Ki  [ = ]       0    [Unmapped]
  [DEL] -2.95Ki  [DEL]     -61    core::ptr::drop_in_place<std::sync::poison::PoisonError<std::sync::poison::rwlock::RwLockReadGuard<figment::figment::Figment>>>::h4406de32d911b1ef
  -0.0% -1.32Ki  -0.0% -1.66Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: f09e7f73-afd6-4056-b339-d4744e7d2ba6
Baseline: 6b23552c · Comparison: 7b60f6e1 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	⚪ +6.47	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	⚪ +3.62	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ +2.33	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ +1.10	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ +1.03	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ +0.95	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ +0.64	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ -0.59	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ +0.48	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -0.41	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.34	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ +0.30	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.25	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.21	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ +0.20	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ -0.18	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ +0.14	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ +0.04	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ -0.02	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ +0.01	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ +0.00	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ +0.01	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ +0.04	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.08	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ -0.17	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.19	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.20	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ +0.54	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ -0.58	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ -1.23	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ -2.00	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ -2.11	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ -2.85	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 126 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 40.2 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 60.9 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 184 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.6 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[tobz]

Looking at the forwarder worker logic in DDA, it looks like the concurrency limit is set per worker, so I think we'd actually want to set our concurrency limit to forwarder_num_workers * forwarder_max_concurrent_requests to properly mimic the behavior.