Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visual Editor quickstart guide #6770

Open
wants to merge 15 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
248 changes: 248 additions & 0 deletions website/docs/guides/visual-editor-qs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
---
title: "Quickstart for the dbt Cloud Visual Editor"
id: "visual-editor"
level: 'Beginner'
icon: 'dbt'
hide_table_of_contents: true
tags: ['Visual Editor','Analyist', 'dbt Cloud','model']
recently_updated: true
---

<div style={{maxWidth: '900px'}}>

## Introduction

:::note beta feature

The Visual Editor is currently in a limited beta. [Contact us](https://www.getdbt.com/contact/) if you're interested in joining it. Features currently in the beta are subject to change or removal.

:::

The dbt Cloud Visual Editor offers a quick and straightforward way for anyone to build analytics models, no background in analytics engineering required! In this guide, you will learn about:
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

- Accessing the Visual Editor and creating a new model
- How to navigate the interface
- How to build a model using operators
- How to commit your changes to Git
- Locating your Visual Editor model and data

## Prerequisites

To use the Visual Editor, you must meet the following prerequisites:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enterprise only as well!


- Your account must have the following configured:
- A data warehouse connection
- Integration with a Git provider
- Source models for the Visual Editor must have been run at least once
- You must have a `developer` license
- You must have credentials configured for your data warehouse and Git provider in the **Your profile** section of the **Account settings**.

:::note

The examples in this guide use the [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) GitHub repo sample project. You can use your data, but the Jaffle Shop offers a full-featured project useful for testing dbt features. Ask your dbt Cloud administrator about importing it to a project in your environment. They must also execute `dbt run` on the Jaffle Shop project before you begin, or you will be unable to reference the source models.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

:::

## Access the Visual Editor
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

To access the Visual Editor:

1. Click **Develop** from the main menu. If you do not see the **Develop** option, ensure you have selected a **Project** from the menu.
2. Click **Visual Editor**.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/ve-main-menu.png" width="90%" title="Visual Editor in the main menu."/>

3. From the right side, click **Get started** and then click **Create new model**.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/ve-create-new-model.png" width="90%" title="Create a new model from the Visual Editor landing page."/>

## Navigating the interface

The Visual editor comprises a series of menus activated by clicking icons surrounding the border of the larger canvas. With none of the menu items activated, the workspace looks like this:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: capitalize Editor


<Lightbox src="/img/docs/dbt-cloud/visual-editor/ve-screen.png" width="90%" title="The Visual Editor canvas. The number of items is defined in this section." />

Click on an icon to expand its section or execute an action depending on its purpose. The options are as follows:

1. The model's title. This defaults to "Untitled" but can be edited anytime by clicking on it.
2. The **Operators** menu contains the building blocks of [creating a model](#create-a-model) with the editor.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hiya @matthewshaver

It could just be me, but when I click this link, it doesn't take me to a page and when I right click to open the link in a new window, I'm directed to the same page.

Kind Regards
Natalie

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought that worked on guides. I'll remove

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
3. The **SQL code** area that displays the [SQL that compiles your model](#sql-code).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for this link

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
4. The **Runs and previews** that displays [run data and previews data](#runs-and-previews) for individual operators.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this one too

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
5. The **Commit history** display.
6. The **Explorer** view of your model.
7. The navigation tab that has icons for (from top to bottom):
- Zoom in
- Zoom out
- Center the model to fit the screen
- Auto-layout option for the individual operator tiles
8. The **Run** command executes `dbt run` for the model.
9. This button is, initially, a **Commit** command for your integrated Git provider. Changes will be made to "Open pull request" once changes are committed. This will not appear until a change is made requiring a commit.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

## Create a model

This section will walk you through creating a model with operators using sample data from the [Jaffle Shop](https://github.com/dbt-labs/jaffle-shop) project. With this guide, you will create a basic model that transforms two datasets to build a view of repeat customer purchases while you consider a loyalty program for your shop.

The operators are the heart of your model. They determine what data will be transformed and how. Click the **+** icon to open the operator menu.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/operators.png" width="90%" title="The operator’s menu on the side of the Visual Editor canvas." />

Read more about the [individual operators](/docs/cloud/visual-editor-interface#operators) to understand the basic purpose of each. Keep in mind that the model you're creating relies on existing models and that the term will primarily be used to reference the model operator in this section.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd clarify the second half of this sentence. Maybe:

"The dbt model created by the Visual Editor builds off of existing models - in these docs, the term "model" will primarily refer to the model operator found this operator drawer."


### Operator tiles

The operators are drag-and-drop from their menu to the canvas, and when they are dropped they will create a tile.

The tiles have the same basic setup with different fields depending on their function. All operators except for **Model** must be connected to another tile before configuring. Once configured, they’ll have the same basic layout.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/operator-tile.png" width="90%" title="An operator tile with configurations filled out." />

1. **The connectors:** Click-and-drag to the connector on another operator to link them.
2. **The title:** Click to change. The examples in this guide will remain default.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"In images in this guide, the operators names are kept as the defaults."

3. **Play icon and menu:** Preview the data at any point in its transformation by clicking the tiles play icon. The dropdown menu contains the option to **Delete** a tile.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: tile's

4. **Column icon:** The number next to it represents the number of columns in the data at that point in its transformation.

:::tip

Make operator tile titles unique compared to your column names to avoid confusion, and the same applies to any aliases you create.

:::

### Create your source models
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mixed feelings on calling these source models given that they're not dbt sources? is there another thing we could call them? maybe "create your first models"? especially since reading in sources is something that's coming to VE?

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

To get started:

1. Expand the **Operators** menu and drag the **Model** operator over to the canvas.
2. Click **Configure model** and then select the source `stg_models` from the dropdown.
3. Click the **Output all columns** option.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this section rocks!


<Lightbox src="/img/docs/dbt-cloud/visual-editor/one-model-operator.png" width="90%" title="A single model operator." />

You now have your first data source!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also here. maybe just "you now have your first model in visual editor" or something like that?

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

4. Drag a new **Model** operator to the canvas below the first and repeat the previous steps, but this time set the source model to `stg_order_items`.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/two-model-operators.png" width="90%" title="Two model operators in te canvas."/>
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

Now, you have two source data models and are ready to transform the data!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here w/source

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

:::tip

Don't see a source model you're looking for? Ask your dbt admins to ensure it's been run recently and hasn't gone stale.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here w/source. and is it worth saying "been run in production" recently? the reader might not know what that means but their dbt admin would know.

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

:::

### Create a join

1. From the **Operators** menu, drag the **Join** operator onto the canvas to the right of the source models.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/join-not-connected.png" width="90%" title="A join that has not been connected to the models" />

2. Click and drag a line from the **+** connector below the `L` on the join border to the **+** on the `stg_orders` model. Do the same for the `R` connector to the `stg_order_items` model.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/join-connected.png" width="90%" title="The join is connected to two model operators." />

3. In the **Join** tile, click **Configure join condition.**
4. Set the **Join type** to `Inner`.
5. In the pair of dropdowns, set both `stg_orders` and `stg_order_items` to `ORDER_ID`.
6. Click **Select and rename columns** and click **Configure columns**
select the following columns:
- From `stg_orders` click `ORDER_ID` and `CUSTOMER_ID`.
- From `stg_order` click `PRODUCT_ID`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be stg_order_items?

- Note: These will appear in the order they are clicked.
7. You've now built your join! Test it by clicking the **Play icon** in the top right corner of the join tile. Your data will populate in the **Runs and previews** pane.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/preview-join.png" width="90%" title="A completed join with the sample data." />

:::tip

Your work in the Visual Editor is automatically saved as you progress, so if you need a break, you can always come back to a session later. Just be sure to give it a unique title!

:::

## Enhance your model

You've got the basics going with your Visual Editor model! It has successfully joined two source models, but you need to further transform the data to get what you need. A list of customers who buy repeat items as you consider a loyalty club rewards program.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flagging another source mention here

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved

### Aggregate data

Multiple options for transforming your data include custom formulas, filters, and unions. Keep it simple and add an aggregation operator to tell you which customers buy the most repeat products.

1. Drag the **Aggregation** operator over to the right of the join.
2. Connect the aggregation operator to the join operator.
3. Click **Configure aggregation** in the **Aggregation tile**.
4. Click in the **Group by** field and first select `CUSTOMER_ID` then `PRODUCT_ID`.
5. Configure the next three fields with the following:
- **Function:** Count
- **Column:** PRODUCT_ID
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
- **Alias:** count_PRODUCT_ID
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<Lightbox src="/img/docs/dbt-cloud/visual-editor/aggregation.png" width="90%" title="The configured aggregation operator tile." />

6. Click the **Play icon** to preview the data. You're starting to see the results you're looking for, but the data is scattered. Let's clean it up a bit more.

:::tip

As your model grows, you can zoom in and out to view your needs. Click and hold in empty canvas space to drag your setup across the screen. Click the **Fit view** icon to see your entire model on the screen. Click the **Auto layout** icon to auto-arrange the tiles efficiently.

:::

### Add some order

There's a lot of data there. Dozens of customers are buying hundreds of products. You will sort it so that the customers are listed ascending by their CUSTOMER_ID number, with the most purchased products listed in descending order.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

1. Drag the **Order** operator over to the right of the **Aggregation** tile and connec them.
2. Click **Configure order**.
3. In the **Sort order** field click **Select column** and click `Aggregation1.CUSTOMER_ID` from the dropdown. Set it to `Asc`.
4. Click **Add sorting** and in the new **Select column** field select `Aggregation1.count_PRODUCT_ID`. Set it to `Desc`.
5. Press the **Play icon** to preview the new data.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/order.png" width="90%" title="The ordered data operator tile config and data preview." />

:::tip

Want to practice on your own? Try adding a **Filter** operator that removes items with less than 10 sales for any customer ID. Be sure to run the preview and verify the data is correct.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i love this!! 🔥

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, excellent practice item here!


:::

## Run and share your model

Now that you've built a model that results in the data you want, it's time to run it and push it to your Git repo.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would break this down and recap how preview/run/prod work at the top, since it's a new concept.

  1. preview does not affect state in your warehouse, so you can test and develop.
  2. when you're ready to use this mode in a downstream tool, you can run it to materialize it in the warehouse.
  3. Once the model is ready to make its way to production and be used by others or orchestrated, you can commit and open a PR.


### Run

To run your model, you only need to click the big **Run** button. With the Visual Editor, there is no command line and no need to memorize a list of commands; there is only **Run**. Click it to see the results populate in the **Runs and previews** pane.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome callout here


<Lightbox src="/img/docs/dbt-cloud/visual-editor/run-results.png" width="90%" title="The results of a successful run in the 'Runs and previews' pane." />

This will [materialize](/docs/build/materializations) the data as a `view` in your developer schema in the database. Once the model has been merged with your project and `dbt run` is executed in your Staging or Production environments, it will be materialized as a view in their related schemas.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/preview-data.png" width="90%" title="Preview of the transformed data in Snowflake." />

### Git commit

The models built in the Visual Editor are a part of your larger dbt project. They are stored in the `visual_editor` folder of your `/models` directory. This is all done automatically; you don't have to configure any paths or directories.

<Lightbox src="/img/docs/dbt-cloud/visual-editor/ve-model-folder.png" width="90%" title="Example of the Visual Editors model path in GitHub." />

However, it won't be created in your Git repo until you commit your first model. So, back in the model's view:

1. Click **Commit** in the top right.
- If you've already created a commit and wish to make more, click the arrow next to **Create a pull request** to see the **Commit** option.
2. Fill out the **Description** field with information about your model. If it's long, part of it will be included in the pull request title, and the rest will be in the body. That's okay! You can correct it during the PR creation process.
3. Click **Commmit**.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
4. The **Commit** button will change to **Create a pull request**. You can add more commits, but click the **Create a pull request** button for now. You will then be redirected to your Git provider in a new tab.

The following examples use GitHub as the provider:

<Lightbox src="/img/docs/dbt-cloud/visual-editor/demo-model-github.png" width="90%" title="Example of the screen you're taken to in GitHub when you create a pull request from the Visual Editor." />

5. Click **Create pull request** in the GitHub window.
6. Complete the **Add a title** and **Add a description** fields. If your description is split between both, copy all the contents to the description field and give it a shorter title.
matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
7. Click **Create pull request**.

You've just submitted your first model from the Visual Editor for review. Once approved and merged, the model will be included in your organization’s project and run whenever `dbt run` is executed in any environment it is in. You're now on your way to becoming an expert in data transformation!
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You've just submitted your first model from the Visual Editor for review. Once approved and merged, the model will be included in your organization’s project and run whenever `dbt run` is executed in any environment it is in. You're now on your way to becoming an expert in data transformation!
You've just submitted your first model from the Visual Editor for review. Once approved and merged, the model will be included in your organization’s project and run whenever `dbt run` is executed in any environment your model is in. You're now on your way to becoming an expert in data transformation!



matthewshaver marked this conversation as resolved.
Show resolved Hide resolved


</div>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading