fix(ingest/snowplow): use non-deprecated pipelines/v1 enrichments endpoint#17721
Open
treff7es wants to merge 6 commits into
Open
fix(ingest/snowplow): use non-deprecated pipelines/v1 enrichments endpoint#17721treff7es wants to merge 6 commits into
treff7es wants to merge 6 commits into
Conversation
…point
Snowplow deprecated the resources/v1/.../configuration/enrichments endpoint.
Switch get_enrichments() to GET pipelines/v1/{pipelineId}/enrichments and adapt
the new EnrichmentResource payload (name/enabled/schemaKey/userData) into the
internal model via Enrichment.from_resource(). userData is the parameter object
itself (not wrapped in a "parameters" key); vendor is derived from the Iglu
schemaKey since the new payload no longer returns it.
Because the new payload has no per-enrichment UUID, enrichment DataJob URNs are
now derived from the enrichment name instead of the UUID (breaking change noted
in updating-datahub.md). Validated end-to-end against the live BDP Console API.
github-actions
Bot
added
ingestion
Updated the documentation to reflect changes in the datahub-prefect package requirements, removal of the chart_pattern config field, legacy Docker Compose files, and Docker image tagging strategy.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
maggiehays
added
the
needs-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Snowplow connector fetched pipeline enrichments from the BDP Console API endpoint
GET organizations/{org}/resources/v1/pipelines/{pipelineId}/configuration/enrichments, which Snowplow has deprecated (confirmed viadeprecated: truein their OpenAPI spec). This switches to the supported endpointGET organizations/{org}/pipelines/v1/{pipelineId}/enrichments— the samepipelines/v1namespace already used byget_pipeline().Details
The replacement endpoint returns a different payload shape (
EnrichmentResource:name/enabled/schemaKey/userData) instead of the legacyEnrichmentDefinitionResource(id/filename/content/lastUpdate). To keep all downstream lineage/PII logic and existing tests unchanged, the new payload is adapted into the internal model via a newEnrichment.from_resource()anti-corruption mapping:userDatais the enrichment parameter object directly (e.g.{"geo": {...}},{"pii": [...], "strategy": {...}}) — it is not wrapped in aparameterskey. This matches the deprecated endpoint'scontent.data.parametersexactly.vendoris derived from the IgluschemaKey(iglu:<vendor>/<name>/...), since the new payload no longer returns it.Breaking change
Enrichment DataJob URNs change from UUID-based (
...,07409eac-...)) to name-based (...,campaign-attribution)). Existing enrichment DataJobs ingested under the old URNs will be orphaned and re-created under the new name-based URNs on the next run. No recipe changes required. Documented indocs/how/updating-datahub.md.Testing
test_snowplow_client.py,test_snowplow_client_extended.py) and the integration fixture + golden file to the new shape.Checklist
updating-datahub.md)