Skip to content
This repository has been archived by the owner on Jul 21, 2023. It is now read-only.

DiscreETLy (tables) "uses" field can't reference tables of other DAGs #22

Open
MikolajBalcerek opened this issue Aug 2, 2019 · 1 comment
Open

DiscreETLy (tables) "uses" field can't reference tables of other DAGs #22

MikolajBalcerek opened this issue Aug 2, 2019 · 1 comment

Comments

@MikolajBalcerek
Copy link
Collaborator

MikolajBalcerek commented Aug 2, 2019 •
edited
Loading

Describe the bug
Tables extension doesn't properly support tables relying ("uses") on other tables from different DAG.
This causes issues on "Tables managed by ..." view, where no graph will be shown for the 'dag2' (see Steps to reproduce).

To Reproduce
Steps to reproduce the behavior:

  1. In /config/tables.yaml declare two tables, belonging to two different DAGs, where one is using another e.g.:

- name: table1
  db: db
  dag_id: dag1
  task_id: create_table1

- name: rollup_of_table1
  db: db
  uses: db.table1
  dag_id: dag2
  task_id: rollup_of_table
```1

2. Run DiscreETLy, go to /etl, select the graph view for dag2
3. No graph is shown

**Expected behavior**
The graph should be shown.
MikolajBalcerek added a commit that referenced this issue Aug 2, 2019
@MikolajBalcerek
Update the example and readme.md to reflect
452c3e8 
#22
MikolajBalcerek added a commit that referenced this issue Aug 2, 2019
@MikolajBalcerek
Document the current state of the /issues/22
6fbc8a5 
#22
@MikolajBalcerek MikolajBalcerek mentioned this issue Aug 2, 2019
Merged
MikolajBalcerek added a commit that referenced this issue Aug 2, 2019
@MikolajBalcerek
Issue 22 | Add workaround for table using table from another DAG (#23)
79da278 
* Fix typo in variable name

dag_progess -> dag_progress

* Add a workaround for table referencing another from different DAG

This replaces node's parent to link to root, instead of failing to lookup table from another DAG
Also references issue created on Github

* Document the current state of the /issues/22

#22
@MikolajBalcerek
Copy link
Collaborator Author

MikolajBalcerek commented Aug 2, 2019 •
edited
Loading

See #23 for an introduction of a workaround - allowing for declaration and specified above, but losing progress and dependency tracking on Tables Managed by..

More extensive rewrite would have to happen to properly support shared tables among different DAGs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MikolajBalcerek