Skip to content

[TFgen] Implement tracking-field decoder + duplicate artifact name detection#3853

Merged
j10czar merged 5 commits into
generator-v2from
jason.tenczar/generator-v2/tracking-field-decode
Jun 4, 2026
Merged

[TFgen] Implement tracking-field decoder + duplicate artifact name detection#3853
j10czar merged 5 commits into
generator-v2from
jason.tenczar/generator-v2/tracking-field-decode

Conversation

@j10czar
Copy link
Copy Markdown

@j10czar j10czar commented Jun 2, 2026

Changes

Decodes the x-datadog-tf-generator vendor extension into typed metadata on
model.Operation.Tracking during LoadSpec, and enforces global uniqueness of the resulting
artifact_name. Both run inside a single LoadSpec call, which now returns a fully-populated
*model.Spec or an actionable error.

  • parser.DecodeTracking(op, path, method, extensionName) (*model.TrackingFieldMetadata, error) — parses the extension into a typed *model.TrackingFieldMetadata (artifact_kind,
    artifact_name, group, id_strategy, sensitive, skip) for downstream codegen,
    applying the id_strategy default (data.id). Before populating the struct it validates the
    raw value at runtime against the committed JSON Schema
    (internal/contracts/tracking-field.schema.json, embedded via a new contracts package), so
    callers only ever get well-formed metadata back. Returns (nil, nil) when the operation is
    out of scope (no extension, or skip: true), and a typed *TrackingError naming the OpenAPI
    location (method + path + operationId) when the extension is malformed (unknown
    artifact_kind, missing required field, unknown property, bad name pattern, …).
  • parser.CheckDuplicateArtifactNames(spec) — walks every Operation.Tracking and fails
    with a single aggregated *DuplicateArtifactNameError listing each artifact_name collision
    and all of its source locations, sorted by name then (path, method) for deterministic
    output.
  • Pipeline wiring — decode happens per-operation in the enumeration loop (storing the
    metadata on Operation.Tracking, fail-fast on a malformed extension); the duplicate check
    runs after the deterministic sort.
  • --tracking-field seam — adds a WithTrackingFieldName LoadSpec option and threads
    flags.trackingField through the generate command, mirroring the existing --max-depth
    wiring.
  • New dependencygithub.com/santhosh-tekuri/jsonschema/v6 (a JSON-Schema validator,
    not a second OAS parser).

Note: the model carries json: tags, so once the value is validated DecodeTracking
bridges YAML→JSON (json.Marshal/Unmarshal) to fill the typed struct, rather than decoding
the yaml.Node directly.

Test Plan

New coverage — decode (asserts the returned metadata): valid resource / valid data source
(kind, name, group, id_strategy populated), absent extension, skip:true, missing required
fields, unknown artifact_kind, unknown property (additionalProperties:false), bad
artifact_name pattern, applied id_strategy default, error-carries-OpenAPI-path, custom
extension name. Duplicates: none / single collision / all-sources-listed /
multiple-collisions-sorted / ignores-untracked / deterministic-across-input-order. LoadSpec
wiring fixtures: tracking populated (resource + data source + unflagged op), duplicate →
typed error naming both ops, malformed → typed error carrying the real OpenAPI path.

❯ make tfgen-test
cd .generator-v2 && go test ./internal/... ./cmd/tfgen/...
ok    generator/internal/cli        0.505s
?     generator/internal/contracts  [no test files]
ok    generator/internal/model      0.495s
ok    generator/internal/parser     0.561s
      generator/cmd/tfgen           [no test files]
tfgen tests passed

Follow-ups / action items

  • Remove the parser-side default extension name once the global CLI flags are wired into
    the cobra root (APIR-2923). DefaultTrackingFieldName and the empty-name fallback are a
    temporary stopgap so LoadSpec is usable standalone — the real default
    (x-datadog-tf-generator) already lives on the --tracking-field flag in root.go. Once the
    wiring guarantees the name is always supplied, delete:

    • the DefaultTrackingFieldName const — internal/parser/tracking.go
    • the if name != "" fallback in WithTrackingFieldName and the trackingFieldName: DefaultTrackingFieldName seed in LoadSpecinternal/parser/parser.go

    …so the extension name has a single source of truth (the CLI flag), instead of a default
    defined in two places.

  • Update OAS extension contract

⚠️ Note: artifact-name uniqueness is enforced per artifact_kind, not globally — a
resource and a data_source may share a name (matches Terraform's separate namespaces).
This intentionally refines the literal AC; the schema is updated to match, but the ticket AC +
APIR-2492 discovery doc still say "globally unique" and need an owner's confirmation.

Copy link
Copy Markdown
Author

j10czar commented Jun 2, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Supam Supam changed the base branch from thibault.viennot/generator-v2/parser to graphite-base/3853 June 3, 2026 08:46
@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from ede0af4 to e6a7a1c Compare June 3, 2026 09:56
@Supam Supam force-pushed the graphite-base/3853 branch from eb53a77 to 301490f Compare June 3, 2026 09:56
@Supam Supam changed the base branch from graphite-base/3853 to thibault.viennot/generator-v2/parser June 3, 2026 09:56
@Supam Supam force-pushed the thibault.viennot/generator-v2/parser branch from 301490f to 5eb0064 Compare June 3, 2026 12:23
@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from e6a7a1c to 085ec78 Compare June 3, 2026 12:23
@Supam Supam force-pushed the thibault.viennot/generator-v2/parser branch from 5eb0064 to bf3cc22 Compare June 3, 2026 12:30
@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from 085ec78 to bb3aa38 Compare June 3, 2026 12:30
@j10czar j10czar changed the title Implement tracking-field decoder + duplicate artifact name detection [TFgen] Implement tracking-field decoder + duplicate artifact name detection Jun 3, 2026
@j10czar j10czar marked this pull request as ready for review June 3, 2026 13:21
@j10czar j10czar requested a review from a team as a code owner June 3, 2026 13:21
@datadog-prod-us1-6

This comment has been minimized.

@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from bb3aa38 to 87082c5 Compare June 3, 2026 14:03
@Supam Supam force-pushed the thibault.viennot/generator-v2/parser branch from bf3cc22 to 10f43de Compare June 3, 2026 14:03
@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from 87082c5 to 5c7c167 Compare June 3, 2026 14:41
Comment thread .generator-v2/internal/parser/tracking.go Outdated
@j10czar
Copy link
Copy Markdown
Author

j10czar commented Jun 3, 2026

Need to add coverage to ignore collisions when both the resource and datasource have the same artifact name. Right now the CheckDuplicateArtifactNames function flags it. There are many instances where something that is a datasource and something that is a resource has the same artifact name ex. dashboard, monitors, slos etc.). Are we are adding some kind of uniqueness to artifact name in the future @Supam ?

"minLength": 1,
"maxLength": 64,
"description": "Terraform-facing artifact name without the 'datadog_' prefix. Must be lowercase snake_case. Must be unique across the entire spec."
"description": "Terraform-facing artifact name without the 'datadog_' prefix. Must be lowercase snake_case. Must be unique per artifact_kind — resources and data sources are separate Terraform namespaces, so a resource and a data source may share a name."
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Supam is this a valid statement? Should the OAS extension contract be updated?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct.

@j10czar j10czar force-pushed the thibault.viennot/generator-v2/parser branch from cd57259 to b98b820 Compare June 4, 2026 12:42
@j10czar j10czar force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from 5a70fec to 121dbbd Compare June 4, 2026 12:42
Copy link
Copy Markdown
Member

Supam commented Jun 4, 2026

There technically is already some differentiation for both datasources and resources, it is just not obvious at the OAS level.

In the OAS we will be asking for the artefact name, this name will be used in both datasources and resources.
Once we run the generator we will create datasources as datasource_<artefact-name> and resources as resource_<artefact-name>. This is how we differentiate between both.

@j10czar j10czar requested a review from Supam June 4, 2026 13:09
Base automatically changed from thibault.viennot/generator-v2/parser to generator-v2 June 4, 2026 15:28
@Supam Supam force-pushed the jason.tenczar/generator-v2/tracking-field-decode branch from 121dbbd to c4630e5 Compare June 4, 2026 15:30
@j10czar j10czar merged commit da502e3 into generator-v2 Jun 4, 2026
8 of 9 checks passed
@j10czar j10czar deleted the jason.tenczar/generator-v2/tracking-field-decode branch June 4, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants