Repository navigation
[Bug] dbt-athena is compiling non-specified sources #748
Description
Is this a new bug in dbt-athena?
- I believe this is a new bug in dbt-athena
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
I have two major databases in my dbt project sources - Athena and Trino and two targets in projects profiles as shown below
version: 2
sources:
- name: trino
database: trino
schema: trino_prod
tables:
- name: xxxxxx
- name: athena
database: AwsDataCatalog
schema: athena
tables:
- name: xxxxx
-
trino:
target: prod
outputs:
prod:
type: trino
host: xxx.xxx.xxx.xxx
user: xxx
password: xxx
catalog: xxx
port: 8080
schema: xxx
threads: 1
method: ldap
http_scheme: https
athena:
target: prod
outputs:
prod:
type: athena
region_name: us-east-1
schema: xxx
database: awsdatacatalog
aws_profile_name: xxx-xxx-xxx
work_group: xxx-xx-xxx
s3_staging_dir: s3://xxx
s3_data_dir: s3://xxxx/xxx
s3_data_naming: table
s3_tmp_table_dir: s3://xxxxxx
poll_interval: 10
I can use dbt docs generate --select "source:trino" --profile trino to limit generating documentation for trino but I can't do the same for dbt docs generate --select "source:athena" --profile athena as this leads to generating documentation for source:trino ultimately leading to failure
13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found.
13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxxxx_xxxx_xxxxxx was not found.
Expected Behavior
When I run dbt docs generate --select "source:athena" --profile athena I would like to generate the documentation for objects that depend on "source:athena"
Steps To Reproduce
- Create a sources.yml that has Athena and Trino sources
- Create a profiles.yml that has Athena and Trino targets
- Run
dbt docs generate --select "source:athena" --profile athenaand it tries to generate the metadata for databases accessible via Trino as well.
Relevant log output
13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found.
13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxxxx_xxxx_xxxxxx was not found.Environment
- OS: AL2
- Python: Python 3.11.11
- dbt-core: 1.7.15
- dbt-athena: 1.7.2
- dbt-trino: 1.7.1Additional Context
No response
Activity
Pass secrets to publish internal for AWS CLI (#748)
[-]dbt docs generate --select "source:athena" --profile athena fails with An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found.[/-][+][Bug] dbt-athena is compiling non-specified sources [/+]soham-dasgupta commentedon May 22, 2025
Is there an update on this issue?
This limits our capability to generate documentation for Athena sources where multiple adapters co exist within the same dbt project
Giqles commentedon Jul 29, 2025
I have a related issue.
dbt docs generatewith thedbt-athenaadapter seems to try and introspect tables that aren't specified as sources or created in targets by the dbt project, but are in the same (huge) Glue catalog (or at least source schema). And then fails when it encounters issues with permissions to other tables in the Glue catalog that aren't related to the project.As a result we also need to run with
dbt docs generate --empty-catalog.soham-dasgupta commentedon Jul 30, 2025
This is similar to the issue that I am facing. Its trying to compile non specified sources and thus failing.