Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add version to output #427

Merged
merged 31 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
1420383
add versino to unpack macros, recursive sql, and intermediate core mo…
dave-connors-3 Feb 12, 2024
ef1fe4e
update uniqueness test
dave-connors-3 Feb 12, 2024
8eb5889
add versions to int_model_5
dave-connors-3 Feb 12, 2024
935dd94
update direct join to source test
dave-connors-3 Feb 12, 2024
aff9157
update hard coded rule
dave-connors-3 Feb 12, 2024
dfdfd23
update fanout rule
dave-connors-3 Feb 12, 2024
aa106cc
update multiple sources joined
dave-connors-3 Feb 12, 2024
1d71283
update docs coverage percentage
dave-connors-3 Feb 12, 2024
098fc1e
update root models rule
dave-connors-3 Feb 12, 2024
9cbb9f7
interim refactor
dave-connors-3 Feb 12, 2024
6926da5
remove version and deprecation date from recursive sql
dave-connors-3 Feb 12, 2024
eef7435
amend djtc
dave-connors-3 Feb 12, 2024
bc6eaf7
amend hcr
dave-connors-3 Feb 12, 2024
03304f5
amend mf
dave-connors-3 Feb 12, 2024
a4cae24
amend msj
dave-connors-3 Feb 12, 2024
44bce77
update rjuc
dave-connors-3 Feb 12, 2024
a14f565
update seed for root models
dave-connors-3 Feb 12, 2024
cf6ffd7
update undocumented models seed
dave-connors-3 Feb 12, 2024
525a854
fix cvds seed
dave-connors-3 Feb 12, 2024
9043d2e
fix remaining seeds, remove version from test calc
dave-connors-3 Feb 12, 2024
f67ad76
revert model test summary test
dave-connors-3 Feb 12, 2024
52689ea
revert logic change to model test summary
dave-connors-3 Feb 12, 2024
1efde31
conditionally exclude check in duckdb due to bug with floats
dave-connors-3 Feb 15, 2024
9cde872
Merge branch 'main' into add-version-to-output
dave-connors-3 Feb 29, 2024
9cb9ece
Merge branch 'main' into add-version-to-output
dave-connors-3 Feb 29, 2024
3c82ad2
Merge branch 'main' into add-version-to-output
graciegoheen Mar 21, 2024
368f776
merge main
dave-connors-3 Mar 28, 2024
a357a64
adjust seeds
dave-connors-3 Mar 28, 2024
38a0417
update tests for rounding errors
dave-connors-3 Apr 1, 2024
6c2bfc1
Merge branch 'main' into add-version-to-output
dave-connors-3 Apr 4, 2024
ff36cf1
Merge branch 'main' into add-version-to-output
b-per Apr 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions integration_tests/models/marts/_int_model_5.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
models:
- name: int_model_5
latest_version: 2
versions:
- v: 1
deprecation_date: 2024-01-01
- v: 2
deprecation_date: 2029-01-01
- v: 3
1 change: 1 addition & 0 deletions integration_tests/models/marts/int_model_5_v1.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
select 1 as id
1 change: 1 addition & 0 deletions integration_tests/models/marts/int_model_5_v3.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
select 1 as id
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
parent,child,parent_and_child,is_loop_independent
stg_model_1,int_model_5,int_model_4,true
stg_model_1,int_model_5.v2,int_model_4,true
2 changes: 2 additions & 0 deletions integration_tests/seeds/dag/test_fct_root_models.csv
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
child
int_model_5.v1
int_model_5.v3
model_8
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
total_models,documented_models,documentation_coverage_pct,staging_documentation_coverage_pct,intermediate_documentation_coverage_pct,marts_documentation_coverage_pct,other_documentation_coverage_pct
16,4,25,20.00,0.00,0.00,75.00
18,4,22.22,20.00,0.00,0.00,75.00
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ model_8,marts
fct_model_9,marts
dim_model_7,marts
int_model_4,intermediate
int_model_5,intermediate
int_model_5.v1,intermediate
int_model_5.v2,intermediate
int_model_5.v3,intermediate
report_3,other
fct_model_10,marts
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@ stg_model_4,models/staging/stg_model_4.sql,models/staging/source_1/stg_model_4.s
stg_model_3,models/staging/source_1/stg_model_3.sql,models/staging/source_2/stg_model_3.sql
dim_model_7,models/marts/intermediate/dim_model_7.sql,models/.../marts/.../dim_model_7.sql
int_model_4,models/marts/int_model_4.sql,models/.../intermediate/.../int_model_4.sql
int_model_5,models/marts/int_model_5.sql,models/.../intermediate/.../int_model_5.sql
int_model_5.v1,models/marts/int_model_5_v1.sql,models/.../intermediate/.../int_model_5_v1.sql
int_model_5.v2,models/marts/int_model_5_v2.sql,models/.../intermediate/.../int_model_5_v2.sql
int_model_5.v3,models/marts/int_model_5_v3.sql,models/.../intermediate/.../int_model_5_v3.sql
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ dim_model_7,FALSE,0
fct_model_6,FALSE,0
model_8,FALSE,0
fct_model_9,FALSE,0
int_model_5,FALSE,0
int_model_5.v1,FALSE,0
int_model_5.v2,FALSE,0
int_model_5.v3,FALSE,0
report_1,FALSE,0
report_2,FALSE,0
report_3,FALSE,0
Expand Down
2 changes: 1 addition & 1 deletion integration_tests/seeds/tests/test_fct_test_coverage.csv
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
total_models,total_tests,tested_models,test_coverage_pct,staging_test_coverage_pct,intermediate_test_coverage_pct,marts_test_coverage_pct,other_test_coverage_pct,test_to_model_ratio
16,13,6,37.50,80.00,50.00,0.00,25.00,0.8125
18,13,6,33.33,80.00,25.00,0.00,25.00,0.7222
2 changes: 1 addition & 1 deletion integration_tests/seeds/tests/tests_seeds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ seeds:
- total_models
- total_tests
- tested_models
- test_coverage_pct
- "{{ 'test_coverage_pct' if not target.name in ['duckdb', 'databricks', 'trino'] else 'tested_models' }}"
- test_to_model_ratio
- staging_test_coverage_pct
- intermediate_test_coverage_pct
Expand Down
2 changes: 2 additions & 0 deletions macros/unpack/get_node_values.sql
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
wrap_string_with_quotes(node.group),
wrap_string_with_quotes(node.access),
wrap_string_with_quotes(node.latest_version),
wrap_string_with_quotes(node.version),
wrap_string_with_quotes(node.deprecation_date),
"cast(" ~ contract | trim ~ " as boolean)",
node.columns.values() | list | length,
node.columns.values() | list | selectattr('description') | list | length,
Expand Down
3 changes: 3 additions & 0 deletions models/marts/core/int_all_graph_resources.sql
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ unioned_with_calc as (
*,
case
when resource_type = 'source' then {{ dbt.concat(['source_name',"'.'",'name']) }}
when version is not null then {{ dbt.concat(['name',"'.v'",'version']) }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this vs. a new field called "version"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i add that on line 87!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ope - yes, thank you!

else name
end as resource_name,
case
Expand Down Expand Up @@ -83,6 +84,8 @@ joined as (
unioned_with_calc.access,
unioned_with_calc.access = 'public' as is_public,
unioned_with_calc.latest_version,
unioned_with_calc.version,
unioned_with_calc.deprecation_date,
unioned_with_calc.is_contract_enforced,
unioned_with_calc.total_defined_columns,
unioned_with_calc.total_described_columns,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ agg_test_relationships as (

final as (
select
all_graph_resources.resource_name,
all_graph_resources.resource_name,
all_graph_resources.resource_type,
all_graph_resources.model_type,
coalesce(agg_test_relationships.is_primary_key_tested, FALSE) as is_primary_key_tested,
Expand Down
2 changes: 2 additions & 0 deletions models/staging/graph/stg_nodes.sql
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ select
cast(null as {{ dbt.type_string() }}) as model_group,
cast(null as {{ dbt.type_string() }}) as access,
cast(null as {{ dbt.type_string() }}) as latest_version,
cast(null as {{ dbt.type_string() }}) as version,
cast(null as {{ dbt.type_string() }}) as deprecation_date,
cast(True as boolean) as is_contract_enforced,
cast(0 as {{ dbt.type_int() }}) as total_defined_columns,
cast(0 as {{ dbt.type_int() }}) as total_described_columns,
Expand Down