feat(ingest): Couchbase ingestion source

[mminichino]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[codecov]

Codecov Report

Attention: Patch coverage is 82.59304% with 145 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../ingestion/source/couchbase/couchbase_profiling.py	77.09%	41 Missing ⚠️
...ngestion/source/couchbase/couchbase_data_reader.py	0.00%	35 Missing ⚠️
...on/src/datahub/ingestion/source/couchbase/retry.py	69.84%	19 Missing ⚠️
...ub/ingestion/source/couchbase/couchbase_connect.py	87.68%	17 Missing ⚠️
...tahub/ingestion/source/couchbase/couchbase_main.py	89.47%	16 Missing ⚠️
.../ingestion/source/couchbase/couchbase_aggregate.py	82.92%	14 Missing ⚠️
.../ingestion/source/couchbase/couchbase_kv_schema.py	97.95%	2 Missing ⚠️
...estion/source/couchbase/couchbase_schema_reader.py	95.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
...b-react/src/app/ingest/source/builder/constants.ts	100.00% <100.00%> (ø)
...hub/ingestion/source/couchbase/couchbase_common.py	100.00% <100.00%> (ø)
...atahub/ingestion/source/couchbase/couchbase_sql.py	100.00% <100.00%> (ø)
...estion/source/couchbase/couchbase_schema_reader.py	95.00% <95.00%> (ø)
.../ingestion/source/couchbase/couchbase_kv_schema.py	97.95% <97.95%> (ø)
.../ingestion/source/couchbase/couchbase_aggregate.py	82.92% <82.92%> (ø)
...tahub/ingestion/source/couchbase/couchbase_main.py	89.47% <89.47%> (ø)
...ub/ingestion/source/couchbase/couchbase_connect.py	87.68% <87.68%> (ø)
...on/src/datahub/ingestion/source/couchbase/retry.py	69.84% <69.84%> (ø)
...ngestion/source/couchbase/couchbase_data_reader.py	0.00% <0.00%> (ø)
... and 1 more

... and 3 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5309ae0...2cc5ba5. Read the comment docs.

[hsheth2]

Left some initial comments on the code

It feels like some features here were added just because they exist for other sources e.g. profiling/classification/domain mapping. Are those strictly necessary for what we're trying to accomplish with this source? Are the implementations sufficiently performant to work at scale?

[hsheth2]

can we just use tenacity or another library for this?

[mminichino]

I suppose I could. Is this already imported? I have always used this code with Python and Couchbase so I know it is stable which is why I used it.

[hsheth2]

we already have a number of utilities for doing "schema inference" from objects - e.g.

datahub/metadata-ingestion/src/datahub/ingestion/source/schema_inference/object.py

Line 86 in 0361f24

def construct_schema(

Does this stuff still make sense?

[hsheth2]

in general, I prefer to have the UI forms get added in a separate PR.

Small PRs are easier to review - and in this case, I like to have the UI changes reviewed by someone who does more frontend dev than I do

[mminichino]

I can remove this.

[hsheth2]

this docker compose and entrypoint file seem more complex than I would have expected

Usually docker containers come with good defaults, which means our docker setups are very simple

[mminichino]

I was required to add these. Couchbase requires a lot of checkpoints to validate the software stack is ready to use when deployed and configured in a container. The provided port check is not sufficient. I can look at removing one check and waiting longer on the second, but some additional checking is needed before the cluster can be used.

[hsheth2]

looks like duplicate files here

[mminichino]

See previous comment on using Couchbase in a container with CI/CD. To exercise connectivity with a cluster, you need the same assets as with the integration test.

[hsheth2]

This is the first source that really uses async

In general, we should almost never be using "bare" asyncio. instead, the anyio library is preferred. We also should not really have references to the event loop all over the place - that tends to be an anti-pattern. Methods that need an event loop should be async. For cpu-bound operations, we can use the asyncer library to bridge between asyncio tasks and worker threads.

[mminichino]

The CB Python SDK is designed to work with either asyncio or Twisted. I can look into asyncer as opposed to calling asyncio methods to call async methods.

[mminichino]

Left some initial comments on the code

It feels like some features here were added just because they exist for other sources e.g. profiling/classification/domain mapping. Are those strictly necessary for what we're trying to accomplish with this source? Are the implementations sufficiently performant to work at scale?

The implementation was written to work at scale. Performance at scale is one of Couchbase's key characteristics. We should be able to support all the features and you can leverage the available settings such as sample size to limit the dataset to further accelerate the ingestion process.