feat(ingestion/montecarlo): add Monte Carlo connector#17685
Conversation
Ingests Monte Carlo monitors, custom SQL rules, and alert/incident run events as DataHub Assertions so the Validation tab on a dataset reflects Monte Carlo's observability coverage and incident history. Key implementation details: - Monitors and custom SQL rules → CUSTOM AssertionInfo - Alerts / incidents → AssertionRunEvent (FAILURE) - MCON resolved to DataHub dataset URN via getTable + connection_to_platform_map - Results cached per MCON to minimise API calls - Results are tagged with native Monte Carlo type, resource ID and data-quality dimension via customProperties Adds: - Source: src/datahub/ingestion/source/montecarlo/ - Integration tests with golden-file fixtures - Unit tests for MCON resolution, assertion building, run events - Docs: docs/sources/montecarlo/ (Overview, Prerequisites, Capabilities, Limitations, Troubleshooting) - Platform bootstrap entry and logo - Connector registered in datahub.json registry Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Bundle ReportChanges will increase total bundle size by 48.2kB (0.21%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: datahub-react-web-esmAssets Changed:
Files in
|
Add Monte Carlo to the frontend connector registry (sources.json for both ingestV2 and the legacy ingest builder) so it appears in the "Create Source" gallery, and map its platform URN to the bundled logo in PLATFORM_URN_TO_LOGO so the icon renders in the gallery and source views. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…onnector # Conflicts: # metadata-ingestion/uv.lock
Summary
CUSTOMtype), with native Monte Carlo type/resource/dimension preserved incustomPropertiesAssertionRunEventfailures, making observability status visible on the dataset Validation tabgetTable+ user-suppliedconnection_to_platform_map, with per-MCON cachingChanges
src/datahub/ingestion/source/montecarlo/tests/integration/montecarlo/tests/unit/montecarlo/docs/sources/montecarlo/bootstrap_mcps/data-platforms.yaml,bootstrap_mcps.yamldatahub-web-react/src/images/montecarlologo.pngautogenerated/connector_registry/datahub.jsonsetup.py,constraints.txt,pyproject.toml,uv.lockTest plan
### Overview,### Prerequisites,### Capabilities,### Limitations,### Troubleshootingall present🤖 Generated with Claude Code