feat(messaging): add pgQueue as alternative messaging transport to Kafka

[david-leifker]

Introduce a PostgreSQL-based queue (pgQueue) as an alternative messaging transport, allowing DataHub to operate without Kafka/Elasticsearch dependencies for smaller deployments.

Key changes:

• pgQueue store with partitioned message tables, consumer offsets, and retention via pg_partman
• Messaging transport abstraction layer (MessagingTransport conditions, KafkaMessagingEnabled/Disabled) for seamless switching between Kafka and pgQueue
• Consumer/processor refactoring: split Kafka listeners from processors to enable pgQueue poll-based consumption
• SqlSetup framework for managing PostgreSQL schema upgrades
• Python pgQueue client, ingestion sink, and DataHub Actions event source
• Docker postgres image with pg_cron and pg_partman extensions

Design Documentation

pgQueue Design Documentation

Check List

• CI Smoke Tests (Regression w/Kafka) Run
• CI Smoke Tests (No Kafka) [PR label smoke:quickstartPgDebug enabled] Run

Related PRs:

• feat(docker): add published DataHub Postgres extensions image #17456
• feat(pgqueue): metadata-ingestion sink and actions pg_queue event source #17467
• feat(ci): drive unified smoke builds from smoke:<configKey> PR labels #17470
• chore(docker): pgQueue dev profiles and Gradle dependency alignment #17480
• feat(messaging): add transport abstractions and Kafka-only wiring #17481

[github-actions]

Linear: PFP-3906

[codecov]

Codecov Report

❌ Patch coverage is 78.10036% with 611 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ueue/postgres/EbeanPostgresMetadataQueueStore.java	83.09%	83 Missing and 37 partials ⚠️
...setup/postgres/pgqueue/PgQueueSqlSetupSupport.java	24.56%	41 Missing and 2 partials ⚠️
...b/upgrade/sqlsetup/postgres/PgQueueSchemaStep.java	67.20%	34 Missing and 7 partials ⚠️
...tadata-ingestion/src/datahub/pgqueue/repository.py	70.63%	37 Missing ⚠️
...m/linkedin/metadata/pgqueue/PgQueuePollWorker.java	77.70%	25 Missing and 10 partials ⚠️
...rade/sqlsetup/postgres/PgCronAdminConnections.java	12.82%	26 Missing and 8 partials ⚠️
...a/config/PgQueueMaePollerSourcesConfiguration.java	82.27%	21 Missing and 7 partials ⚠️
...postgres/migration/PostgresSqlMigrationRunner.java	68.67%	22 Missing and 4 partials ⚠️
...edin/metadata/trace/PgQueueMcpFailedTracePort.java	77.77%	16 Missing and 6 partials ⚠️
.../linkedin/metadata/event/PgQueueEventProducer.java	79.77%	16 Missing and 2 partials ⚠️
... and 47 more

📢 Thoughts on this report? Let us know!

[alwaysmeticulous]

✅ Meticulous spotted 0 visual differences across 1629 screens tested: view results.

Meticulous evaluated ~10 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 01f5bb4 feat(messaging): add pgQueue as alternative messaging transport to Kafka. This comment will update as new commits are pushed.}

[datahub-connector-tests]

Connector Tests Results

Connector tests failed for commit c9e0345

View full test logs →

To skip connector tests, add the skip-connector-tests label (org members only).

Autogenerated by the connector-tests CI pipeline.

[github-actions]

Linear: PFP-3932

[github-actions]

Your PR has been assigned to @david-leifker (david.leifker) for review (PFP-3906).

[david-leifker]

• Use a database schema migration tool or version the schema of the database
• Revisit tags/labels and impl agnostic metrics
• Is there a kafka API implementation which relies on a separate datastore
• pgmq - compare and contrast
• evaluate https://github.com/tansu-io/tansu