Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions .claude/skills/create-config-field/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Use `AskUserQuestion` to collect the following. If `$ARGUMENTS` provides the con
5. **Default value**: What should the default be?
6. **Description**: Human-readable description of what this field controls.
7. **Scope**: Add inline in the subsystem function, or create a dedicated setup file (`pkg/config/setup/<feature>.go`) for a group of related fields?
8. **User-facing?**: Should it be documented in `pkg/config/config_template.yaml`?
8. **User-facing?**: Should the rendered example yaml (`datadog.yaml`, `system-probe.yaml`, ...) show this setting?

### Step 2: Register the config key

Expand All @@ -65,16 +65,21 @@ For a **group of related fields**, create `pkg/config/setup/<feature>.go` with a

For **serverless-compatible** core agent fields, register via the `serverlessConfigComponents` slice in `config.go` instead of directly in `InitConfig`.

### Step 3: (Optional) Document in the config template
### Step 3: Regenerate the schema

If user-facing, add documentation to `pkg/config/config_template.yaml` inside the appropriate conditional block (see Template conditional column in tables above). Read existing entries nearby for the exact format (`@param`, `@env` annotations).
The example yaml configs (`datadog.yaml`, `system-probe.yaml`, etc.) are rendered from the enriched schema under `pkg/config/schema/yaml/`. The schema is derived from the running agent, so build first, then regenerate:

```bash
dda inv agent.build --build-exclude=systemd
dda inv schema.generate --agent-bin=./bin/agent/agent
```

Commit the resulting changes under `pkg/config/schema/yaml/` alongside your Go code.

### Step 4: Verify

1. Build: `dda inv agent.build --build-exclude=systemd` (or `dda inv system-probe.build`)
2. Lint: `dda inv linter.go`
3. If the template was modified: `dda inv generate-config`
4. Report the results to the user.
1. Lint: `dda inv linter.go`
2. Report the results to the user.

## Key Methods Reference

Expand Down
2 changes: 0 additions & 2 deletions cmd/cluster-agent-cloudfoundry/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@

//go:build !windows && clusterchecks

//go:generate go run ../../pkg/config/render_config/render_config.go dcacf ../../pkg/config/config_template.yaml ../../cloudfoundry.yaml

//nolint:revive // TODO(PLINT) Fix revive linter
package main

Expand Down
2 changes: 0 additions & 2 deletions cmd/cluster-agent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@

//go:build !windows && kubeapiserver

//go:generate go run ../../pkg/config/render_config/render_config.go dca ../../pkg/config/config_template.yaml ../../Dockerfiles/cluster-agent/datadog-cluster.yaml

package main

import (
Expand Down
9 changes: 0 additions & 9 deletions cmd/dogstatsd/generate.go

This file was deleted.

1 change: 1 addition & 0 deletions cmd/dogstatsd/main_nix.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

//go:build !windows

// Main package for the dogstatsd binary
package main

import (
Expand Down
1 change: 1 addition & 0 deletions cmd/dogstatsd/main_windows.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
// This product includes software developed at Datadog (https://www.datadoghq.com/).
// Copyright 2016-present Datadog, Inc.

// Main package for the dogstatsd binary
package main

import (
Expand Down
2 changes: 1 addition & 1 deletion docs/public/guidelines/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ In order to ease/speed up our review, here are some items you can check/improve
- [X] The added code comes with tests.
- [X] The CI is green, all tests are passing (required or not).
- [X] All applicable labels are set on the PR (see [PR labels list](#pr-labels)).
- [X] If applicable, the [config template](https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml) has been updated.
- [X] If applicable, the [config schema](https://github.com/DataDog/datadog-agent/tree/main/pkg/config/schema/yaml) has been regenerated (`dda inv schema.generate --agent-bin=./bin/agent/agent`).
///

/// note
Expand Down
34 changes: 25 additions & 9 deletions pkg/config/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,29 +1,45 @@
"""Agent configuration files."""

load("//pkg/config/render_config:agent_config.bzl", "agent_config")
load("//pkg/config/schema:template.bzl", "schema_template_config")

package(default_visibility = ["//visibility:public"])

agent_config(
# Target OS for the rendered config. The legacy `render_config` Go binary
# used `runtime.GOOS` of the build host; the schema-template equivalent
# is a Bazel `select()` on `@platforms//os` so cross-platform builds
# (e.g. windows packages on a linux host) produce the right per-OS
# defaults.
_OS_TARGET = select({
"@platforms//os:linux": "linux",
"@platforms//os:windows": "windows",
"@platforms//os:macos": "darwin",
})

schema_template_config(
name = "agent_config",
srcs = ["//pkg/config/schema:core_schema_subfiles"],
out = "datadog.yaml",
build_type = "agent-py3",
template = "config_template.yaml",
os_target = _OS_TARGET,
top_schema = "//pkg/config/schema:yaml/core_schema.yaml",
)

agent_config(
schema_template_config(
name = "iot_agent_config",
srcs = ["//pkg/config/schema:core_schema_subfiles"],
out = "iot-agent.yaml",
build_type = "iot-agent",
template = "config_template.yaml",
os_target = _OS_TARGET,
top_schema = "//pkg/config/schema:yaml/core_schema.yaml",
)

agent_config(
schema_template_config(
name = "system_probe_config",
out = "system-probe.yaml",
build_type = "system-probe",
template = "system-probe_template.yaml",
os_target = _OS_TARGET,
top_schema = "//pkg/config/schema:yaml/system-probe_schema.yaml",
)

# TODO: Add targets for each package, as needed: dogstatsd, dca, dcacf
# That would get easier if we auto-pick the template based on the build type.
# TODO: Add targets for each package, as needed: dogstatsd, dca, dcacf.
# That would get easier if we auto-pick the schema based on the build type.
51 changes: 48 additions & 3 deletions pkg/config/schema/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,8 +1,50 @@
load("@bazel_lib//lib:run_binary.bzl", "run_binary")
load("@rules_go//go:def.bzl", "go_library", "go_test")
load("@rules_pkg//pkg:install.bzl", "pkg_install")
load("@rules_pkg//pkg:mappings.bzl", "pkg_files")
load("//bazel/rules:zstd.bzl", "zstd_compress")

# Top schemas exposed as labels for downstream rules (e.g. the
# `schema_template_config` rule in :template.bzl).
exports_files(
[
"yaml/core_schema.yaml",
"yaml/system-probe_schema.yaml",
],
visibility = ["//visibility:public"],
)

# Per-section sub-files referenced by yaml/core_schema.yaml via $ref.
# Listed explicitly rather than globbed (per QUESTIONS-2 Q12) so adding
# or removing a sub-file is a visible diff.
filegroup(
name = "core_schema_subfiles",
srcs = [
"yaml/admission_controller.yaml",
"yaml/agent_telemetry.yaml",
"yaml/apm_config.yaml",
"yaml/cluster_agent.yaml",
"yaml/cluster_checks.yaml",
"yaml/compliance_config.yaml",
"yaml/container_image.yaml",
"yaml/container_lifecycle.yaml",
"yaml/external_metrics_provider.yaml",
"yaml/gpu.yaml",
"yaml/internal_profiling.yaml",
"yaml/logs_config.yaml",
"yaml/multi_region_failover.yaml",
"yaml/orchestrator_explorer.yaml",
"yaml/otelcollector.yaml",
"yaml/private_action_runner.yaml",
"yaml/process_config.yaml",
"yaml/remote_configuration.yaml",
"yaml/runtime_security_config.yaml",
"yaml/sbom.yaml",
"yaml/snmp_listener.yaml",
],
visibility = ["//visibility:public"],
)

ZSTD_ARGS = [
"--no-check", # match DataDog/zstd Go library behavior: no XXH64 frame checksum
"-5", # match DataDog/zstd: DefaultCompression = 5
Expand All @@ -12,12 +54,15 @@ ZSTD_ARGS = [
# yaml/apm_config.yaml, ...) via $ref. Merge them into a single file before
# compression so the embedded Go artifact stays a single self-contained
# schema and the Go loader doesn't need to handle $ref resolution.
genrule(
run_binary(
name = "merged_core_schema",
srcs = glob(["yaml/*.yaml"]),
outs = ["core_schema.merged.yaml"],
cmd = "$(execpath //tasks:merge_schema) $(execpath yaml/core_schema.yaml) $@",
tools = ["//tasks:merge_schema"],
args = [
"$(execpath yaml/core_schema.yaml)",
"$@",
],
tool = "//tasks/schema:merge_schema",
)

zstd_compress(
Expand Down
67 changes: 67 additions & 0 deletions pkg/config/schema/template.bzl
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
"""Bazel rule for rendering a config example YAML from an enriched schema.

Inputs (per rule invocation):
- top_schema: the top-level schema file (core_schema.yaml or
system-probe_schema.yaml). For the core agent flavors this entry
references per-section sub-files via $ref, so those sub-files must
also be supplied via `srcs`.
- srcs: additional schema sub-files that the top_schema references.
- build_type: agent-py3, iot-agent, dogstatsd, dca, dcacf, or
system-probe.
- os_target: linux, windows, or darwin.
- out: the rendered example yaml.
"""

def _schema_template_config_impl(ctx):
args = ctx.actions.args()
args.add(ctx.file.top_schema.path)
args.add(ctx.attr.build_type)
args.add(ctx.attr.os_target)
args.add(ctx.outputs.out.path)

ctx.actions.run(
mnemonic = "SchemaTemplateConfig",
executable = ctx.executable._template,
arguments = [args],
inputs = [ctx.file.top_schema] + ctx.files.srcs,
outputs = [ctx.outputs.out],
progress_message = "Rendering %s (%s, %s)" % (
ctx.outputs.out.short_path,
ctx.attr.build_type,
ctx.attr.os_target,
),
)
return DefaultInfo(files = depset([ctx.outputs.out]))

schema_template_config = rule(
implementation = _schema_template_config_impl,
attrs = {
"build_type": attr.string(
doc = "One of agent-py3, iot-agent, dogstatsd, dca, dcacf, system-probe.",
mandatory = True,
),
"os_target": attr.string(
doc = "Target OS: one of linux, windows, darwin. Typically populated via `select()` on `@platforms//os`.",
mandatory = True,
),
"top_schema": attr.label(
doc = "The top-level enriched schema YAML file.",
allow_single_file = [".yaml"],
mandatory = True,
),
"srcs": attr.label_list(
doc = "Additional schema sub-files referenced by the top schema via $ref. Empty for system-probe (single-file schema).",
allow_files = [".yaml"],
default = [],
),
"out": attr.output(
doc = "Path of the rendered example yaml file.",
mandatory = True,
),
"_template": attr.label(
default = Label("//tasks/schema:schema_template"),
executable = True,
cfg = "exec",
),
},
)
13 changes: 13 additions & 0 deletions pkg/config/schema/yaml/admission_controller.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -618,6 +618,19 @@ properties:
type: string
default: datadog/ingress-nginx-injection
comment: ingress-nginx injection configuration
init_run_as_group:
node_type: setting
type: integer
default: 82
init_run_as_user:
node_type: setting
type: integer
default: 101
comment: |-
Non-root UID/GID for the injected init container. Defaults match the
stock datadog/ingress-nginx-injection image, which declares no USER and
would otherwise be rejected under runAsNonRoot. Set to a negative value
to leave the security context unset and honor a custom init_image's own USER.
module_mount_path:
node_type: setting
type: string
Expand Down
50 changes: 49 additions & 1 deletion pkg/config/schema/yaml/core_schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -810,7 +810,6 @@ properties:
default: true
visibility: public
description: Collect GPU related host tags
example: 'false'
tags:
- template_section:Common
- full-agent-only:true
Expand Down Expand Up @@ -1111,6 +1110,44 @@ properties:
example: '"http://127.0.0.1:8080"'
tags:
- template_section:Common
dual_ship:
node_type: setting
type: boolean
default: false
visibility: public
description: |-
When false (default), the Observability Pipelines Worker replaces the primary
Datadog logs intake as the sole destination. When true, logs are sent to BOTH
the primary Datadog intake and the Observability Pipelines Worker, which is
useful for evaluating OPW without interrupting the existing flow of telemetry
to Datadog.
tags:
- template_section:Common
comment: "dual_ship is logs-only: there is no equivalent dual-shipping\
\ code path for metrics, so\nthese keys live outside bindVectorOptions\
\ to avoid registering an unused metrics variant.\n\ndual_ship: when\
\ false (default), OPW replaces the primary Datadog endpoint and is\
\ the only\ndestination logs are shipped to. When true, Datadog remains\
\ the primary endpoint and OPW is\nadded as an additional endpoint \u2014\
\ intended for operators evaluating OPW without interrupting\nthe existing\
\ flow of telemetry to Datadog.\n\ndual_ship_reliable: when dual_ship=true,\
\ controls whether the OPW additional endpoint applies\nbackpressure\
\ to the main pipeline on failure (true) or is best-effort (false, the\
\ default).\nBest-effort is the safer default: an unreachable OPW must\
\ not block delivery to Datadog."
dual_ship_reliable:
node_type: setting
type: boolean
default: false
visibility: public
description: |-
Only meaningful when `dual_ship` is true. When false (default), the OPW
destination is treated as best-effort: failures to OPW do not apply
backpressure to the main pipeline, so an unhealthy OPW cannot stall delivery
to Datadog. Set to true to make the OPW destination reliable, meaning its
failures participate in flow control just like the primary Datadog endpoint.
tags:
- template_section:Common
traces:
node_type: section
type: object
Expand Down Expand Up @@ -9916,6 +9953,17 @@ properties:
node_type: section
type: object
properties:
dual_ship:
node_type: setting
type: boolean
default: false
comment: "Legacy vector.* aliases for dual_ship keys \u2014 users still\
\ on the legacy prefix must not have\ndual_ship=true silently dropped\
\ when the fallback in obsPipelineWorkerDualShip reads these keys."
dual_ship_reliable:
node_type: setting
type: boolean
default: false
enabled:
node_type: setting
type: boolean
Expand Down
6 changes: 6 additions & 0 deletions pkg/config/schema/yaml/external_metrics_provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ properties:
type: string
default: ''
comment: Override the Datadog APP Key for external metrics endpoint
autoscaler_autogen_label_selector:
node_type: setting
type: string
default: ''
comment: Label selector to filter which HPAs and WPAs the DCA generates DatadogMetrics
for (e.g. "app.kubernetes.io/managed-by!=keda-operator")
batch_window:
node_type: setting
type: integer
Expand Down
Loading
Loading