Skip to content

Commit bf2f45e

Browse files
authored
Merge branch 'current' into current
2 parents cbc72cb + 0f397d6 commit bf2f45e

File tree

22 files changed

+318
-151
lines changed

22 files changed

+318
-151
lines changed

website/blog/2022-04-19-dbt-cloud-postman-collection.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ is_featured: true
1919
2020
The dbt Cloud API has well-documented endpoints for creating, triggering and managing dbt Cloud jobs. But there are other endpoints that aren’t well documented yet, and they’re extremely useful for end-users. These endpoints exposed by the API enable organizations not only to orchestrate jobs, but to manage their dbt Cloud accounts programmatically. This creates some really interesting capabilities for organizations to scale their dbt Cloud implementations.
2121

22-
The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them.
22+
The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them.
2323

2424
<!--truncate-->
2525

@@ -45,7 +45,7 @@ Beyond the day-to-day process of managing their dbt Cloud accounts, many organiz
4545

4646
*Below this you’ll find a series of example requests - use these to guide you or [check out the Postman Collection](https://dbtlabs.postman.co/workspace/Team-Workspace~520c7ac4-3895-4779-8bc3-9a11b5287c1c/request/12491709-23cd2368-aa58-4c9a-8f2d-e8d56abb6b1dlinklink) to try it out yourself.*
4747

48-
## Appendix
48+
## Appendix
4949

5050
### Examples of how to use the Postman Collection
5151

@@ -55,7 +55,7 @@ Let’s run through some examples on how to make good use of this Postman Collec
5555

5656
One common question we hear from customers is “How can we migrate resources from one dbt Cloud project to another?” Often, they’ll create a development project, in which users have access to the UI and can manually make changes, and then migrate selected resources from the development project to a production project once things are ready.
5757

58-
There are several reasons one might want to do this, including:
58+
There are several reasons one might want to do this, including:
5959

6060
- Probably the most common is separating dev/test/prod environments across dbt Cloud projects to enable teams to build manually in a development project, and then automatically migrate those environments & jobs to a production project.
6161
- Building “starter projects” they can deploy as templates for new teams onboarding to dbt from a learning standpoint.
@@ -90,10 +90,10 @@ https://cloud.getdbt.com/api/v3/accounts/28885/projects/86704/environments/75286
9090

9191
#### Push the environment to the production project
9292

93-
We take the response from the GET request above, and then to the following:
93+
We take the response from the GET request above, and then to the following:
9494

9595
1. Adjust some of the variables for the new environment:
96-
- Change the the value of the “project_id” field from 86704 to 86711
96+
- Change the value of the “project_id” field from 86704 to 86711
9797
- Change the value of the “name” field from “dev-staging” to “production–api-generated”
9898
- Set the “custom_branch” field to “main”
9999

@@ -116,7 +116,7 @@ We take the response from the GET request above, and then to the following:
116116
}
117117
```
118118

119-
3. Note the environment ID returned in the response, as we’ll use to create a dbt Cloud job in the next step
119+
3. Note the environment ID returned in the response, as we’ll use to create a dbt Cloud job in the next step
120120

121121
#### Pull the job definition from the dev project
122122

website/blog/2022-05-17-stakeholder-friendly-model-names.md

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ In this article, we’ll take a deeper look at why model naming conventions are
2929

3030
>[Data folks], what we [create in the database]… echoes in eternity.” -Max(imus, Gladiator)
3131
32-
Analytics Engineers are often centrally located in the company, sandwiched between data analysts and data engineers. This means everything AEs create might be read and need to be understood by both an analytics or customer-facing team and by teams who spend most of their time in code and the database. Depending on the audience, the scope of access differs, which means the user experience and context changes. Let’s elaborate on what that experience might look like by breaking end-users into two buckets:
32+
Analytics Engineers are often centrally located in the company, sandwiched between data analysts and data engineers. This means everything AEs create might be read and need to be understood by both an analytics or customer-facing team and by teams who spend most of their time in code and the database. Depending on the audience, the scope of access differs, which means the user experience and context changes. Let’s elaborate on what that experience might look like by breaking end-users into two buckets:
3333

3434
- Analysts / BI users
3535
- Analytics engineers / Data engineers
@@ -49,21 +49,21 @@ Here we have drag and drop functionality and a skin over top of the underlying `
4949
**How model names can make this painful:**
5050
The end users might not even know what tables the data refers to, as potentially everything is joined by the system and they don’t need to write their own queries. If model names are chosen poorly, there is a good chance that the BI layer on top of the database tables has been renamed to something more useful for the analysts. This adds an extra step of mental complexity in tracing the <Term id="data-lineage">lineage</Term> from data model to BI.
5151

52-
#### Read only access to the dbt Cloud IDE docs
52+
#### Read only access to the dbt Cloud IDE docs
5353
If Analysts want more context via documentation, they may traverse back to the dbt layer and check out the data models in either the context of the Project or Database. In the Project view, they will see the data models in the folder hierarchy present in your project’s repository. In the Database view you will see the output of the data models as present in your database, ie. `database / schema / object`.
5454

5555
![A screenshot depicting the dbt Cloud IDE menu's Database view which shows you the output of your data models. Next to this view, is the Project view.](/img/blog/2022-05-17-stakeholder-friendly-model-names/project-view.png)
5656

5757
**How model names can make this painful:**
58-
For the Project view, generally abstracted department or organizational structures as folder names presupposes the reader/engineer knows what is contained within the folder beforehand or what that department actually does, or promotes haphazard clicking to open folders to see what is within. Organizing the final outputs by business unit or analytics function is great for end users but doesn't accurately represent all the sources and references that had to come together to build this output, as they often live in another folder.
58+
For the Project view, generally abstracted department or organizational structures as folder names presupposes the reader/engineer knows what is contained within the folder beforehand or what that department actually does, or promotes haphazard clicking to open folders to see what is within. Organizing the final outputs by business unit or analytics function is great for end users but doesn't accurately represent all the sources and references that had to come together to build this output, as they often live in another folder.
5959

6060
For the Database view, pray your team has been declaring a logical schema bucketing, or a logical model naming convention, otherwise you will have a long, alphabetized list of database objects to scroll through, where staging, intermediate, and final output models are all intermixed. Clicking into a data model and viewing the documentation is helpful, but you would need to check out the DAG to see where the model lives in the overall flow.
6161

6262
#### The full dropdown list in their data warehouse.
6363

6464
If they have access to Worksheets, SQL runner, or another way to write ad hoc sql queries, then they will have access to the data models as present in your database, ie. `database / schema / object`, but with less documentation attached, and more proclivity towards querying tables to check out their contents, which costs time and money.
6565

66-
![A screenshot of the the SQL Runner menu within Looker showcasing the dropdown list of all data models present in the database.](/img/blog/2022-05-17-stakeholder-friendly-model-names/data-warehouse-dropdown.png)
66+
![A screenshot of the SQL Runner menu within Looker showcasing the dropdown list of all data models present in the database.](/img/blog/2022-05-17-stakeholder-friendly-model-names/data-warehouse-dropdown.png)
6767

6868
**How model names can make this painful:**
6969
Without proper naming conventions, you will encounter `analytics.order`, `analytics.orders`, `analytics.orders_new` and not know which one is which, so you will open up a scratch statement tab and attempt to figure out which is correct:
@@ -73,9 +73,9 @@ Without proper naming conventions, you will encounter `analytics.order`, `analyt
7373
-- select * from analytics.orders limit 10
7474
select * from analytics.orders_new limit 10
7575
```
76-
Hopefully you get it right via sampling queries, or eventually find out there is a true source of truth defined in a totally separate area: `core.dim_orders`.
76+
Hopefully you get it right via sampling queries, or eventually find out there is a true source of truth defined in a totally separate area: `core.dim_orders`.
7777

78-
The problem here is the only information you can use to determine what data is within an object or the purpose of the object is within the schema and model name.
78+
The problem here is the only information you can use to determine what data is within an object or the purpose of the object is within the schema and model name.
7979

8080
### The engineer’s user experience
8181

@@ -98,7 +98,7 @@ There is not much worse than spending all week developing on a task, submitting
9898
This is largely the same as the Analyst experience above, except they created the data models or are aware of their etymologies. They are likely more comfortable writing ad hoc queries, but also have the ability to make changes, which adds a layer of thought processing when working.
9999

100100
**How model names can make this painful:**
101-
It takes time to become a subject matter expert in the database. You will need to know which schema a subject lives in, what tables are the source of truth and/or output models, versus experiments, outdated objects, or building blocks used along the way. Working within this context, engineers know the history and company lore behind why a table was named that way or how its purpose may differ slightly from its name, but they also have the ability to make changes.
101+
It takes time to become a subject matter expert in the database. You will need to know which schema a subject lives in, what tables are the source of truth and/or output models, versus experiments, outdated objects, or building blocks used along the way. Working within this context, engineers know the history and company lore behind why a table was named that way or how its purpose may differ slightly from its name, but they also have the ability to make changes.
102102

103103
Change management is hard; how many places would you need to update, rename, re-document, and retest to fix a poor naming choice from long ago? It is a daunting position, which can create internal strife when constrained for time over whether we should continually revamp and refactor for maintainability or focus on building new models in the same pattern as before.
104104

website/docs/docs/build/incremental-models.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,7 @@ To use incremental models, you also need to tell dbt:
3434

3535
The `is_incremental()` macro powers incremental materializations. It will return `True` if _all_ of the following conditions are met:
3636

37-
- The model must already exist in the database
38-
- The destination table already exists in the database
37+
- The model must already exist as a table in the database
3938
- The `full-refresh` flag _is not_ passed
4039
- The running model is configured with `materialized='incremental'`
4140

0 commit comments

Comments
 (0)