Skip to content

Commit c88201e

Browse files
benpankowbenpankow
committed
[components][docs] Add components doc on moving definitions into componentw
1 parent b0d8dba commit c88201e

File tree

32 files changed

+371
-0
lines changed

32 files changed

+371
-0
lines changed

‎docs/docs/guides/preview/components/existing-code-location.md

+1
Original file line numberDiff line numberDiff line change
@@ -73,5 +73,6 @@ Now, your code location is ready to use components! `dg` can be used to scaffold
7373

7474
## Next steps
7575

76+
- [Migrate existing definitions to components](./migrating-definitions)
7677
- [Add a new component to your code location](./using-a-component)
7778
- [Create a new component type](./creating-a-component)

‎docs/docs/guides/preview/components/migrating-definitions.md

+58
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
title: 'Migrating existing Definitions to components'
3+
sidebar_position: 350
4+
---
5+
6+
:::note
7+
This guide covers migrating existing Python `Definitions` to components. This guide presupposes a components-enabled project. See the [getting started guide](./) or [Making an existing code location components-compatible](./existing-code-location) guide for more information.
8+
:::
9+
10+
When adding components to an existing Dagster code location, it is often useful to restructure your definitions into component folders, making it easier to eventually migrate them entirely to using components.
11+
12+
## Example project
13+
14+
Let's walk through an example of how to migrate existing definitions to components, with a project that has the following structure:
15+
16+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/1-tree.txt" />
17+
18+
The root `Definitions` object combines definitions from various nested modules:
19+
20+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/2-definitions-before.py" title="my_existing_project/definitions.py" />
21+
22+
Each of these modules consolidates its own `Definitions` object:
23+
24+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/3-inner-definitions-before.py" title="my_existing_project/elt/definitions.py" />
25+
26+
We'll migrate the `elt` module to a component.
27+
28+
## Create a Definitions component
29+
30+
We'll start by creating a `Definitions` component for the `elt` module:
31+
32+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/4-scaffold.txt" />
33+
34+
35+
This creates a new folder in `my_existing_project/components/elt-definitions`, with a `component.yaml` file. This component is rather simple, it just points to a file which contains a `Definitions` object.
36+
37+
Let's move the `elt` module's `definitions.py` file to the new component folder:
38+
39+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/6-mv.txt" />
40+
41+
Now, we can update the `component.yaml` file to point to the new `definitions.py` file:
42+
43+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/5-component-yaml.txt" title="my_existing_project/components/elt-definitions/component.yaml" />
44+
45+
Finally, we can update the root `definitions.py` file to no longer explicitly load the `elt` module's `Definitions`:
46+
47+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/7-definitions-after.py" title="my_existing_project/definitions.py" />
48+
49+
Now, our project structure looks like this:
50+
51+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/8-tree-after.txt" />
52+
53+
We can repeat the same process for our other modules.
54+
55+
## Next steps
56+
57+
- [Add a new component to your code location](./using-a-component)
58+
- [Create a new component type](./creating-a-component)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/1-tree.txt

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
tree
2+
3+
.
4+
├── README.md
5+
├── my_existing_project
6+
│   ├── __init__.py
7+
│   ├── analytics
8+
│   │   ├── __init__.py
9+
│   │   ├── assets.py
10+
│   │   └── definitions.py
11+
│   ├── components
12+
│   ├── definitions.py
13+
│   └── elt
14+
│   ├── __init__.py
15+
│   └── definitions.py
16+
├── my_existing_project_tests
17+
│   ├── __init__.py
18+
│   └── test_assets.py
19+
└── pyproject.toml
20+
21+
6 directories, 11 files

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/2-definitions-before.py

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
from pathlib import Path
2+
3+
import dagster_components as dg_components
4+
5+
import dagster as dg
6+
from my_existing_project.analytics import definitions as analytics_definitions
7+
from my_existing_project.elt import definitions as elt_definitions
8+
9+
defs = dg.Definitions.merge(
10+
dg.load_definitions_from_module(elt_definitions),
11+
dg.load_definitions_from_module(analytics_definitions),
12+
dg_components.build_component_defs(Path(__file__).parent / "components"),
13+
)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/3-inner-definitions-before.py

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
from dagster import asset
2+
from dagster._core.definitions.definitions_class import Definitions
3+
4+
5+
@asset
6+
def my_elt_asset(): ...
7+
8+
9+
defs = Definitions(assets=[my_elt_asset])

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/4-scaffold.txt

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
dg component scaffold 'definitions@dagster_components' elt-definitions
2+
3+
Creating a Dagster component instance folder at /.../my-existing-project/my_existing_project/components/elt-definitions.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/5-component-yaml.txt

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
type: definitions@dagster_components
2+
3+
params:
4+
definitions_path: definitions.py

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/6-mv.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
mv my_existing_project/elt/definitions.py my_existing_project/components/elt-definitions && rm -rf my_existing_project/elt

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/7-definitions-after.py

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
from pathlib import Path
2+
3+
import dagster_components as dg_components
4+
5+
import dagster as dg
6+
from my_existing_project.analytics import definitions as analytics_definitions
7+
8+
defs = dg.Definitions.merge(
9+
dg.load_definitions_from_module(analytics_definitions),
10+
dg_components.build_component_defs(Path(__file__).parent / "components"),
11+
)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/migrating-definitions/8-tree-after.txt

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
tree
2+
3+
.
4+
├── README.md
5+
├── my_existing_project
6+
│   ├── __init__.py
7+
│   ├── analytics
8+
│   │   ├── __init__.py
9+
│   │   ├── assets.py
10+
│   │   └── definitions.py
11+
│   ├── components
12+
│   │   └── elt-definitions
13+
│   │   ├── component.yaml
14+
│   │   └── definitions.py
15+
│   └── definitions.py
16+
├── my_existing_project_tests
17+
│   ├── __init__.py
18+
│   └── test_assets.py
19+
├── pyproject.toml
20+
└── uv.lock
21+
22+
6 directories, 12 files

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/__init__.py

Whitespace-only changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/README.md renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/README.md

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/__init__.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/my_existing_project/__init__.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/assets.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/my_existing_project/assets.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/definitions.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/my_existing_project/definitions.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project_tests/__init__.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/my_existing_project_tests/__init__.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project_tests/test_assets.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/my_existing_project_tests/test_assets.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/pyproject.toml renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/my-existing-project/pyproject.toml

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_existing_code_location.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_component_docs_existing_code_location/test_existing_code_location.py

File renamed without changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/__init__.py

Whitespace-only changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Sample existing project for testing docs for the "Making an existing code location components-compatible" guide.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/analytics/__init__.py

Whitespace-only changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/analytics/assets.py

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from dagster import asset
2+
3+
4+
@asset
5+
def my_asset():
6+
pass

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/analytics/definitions.py

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
from dagster import asset
2+
from dagster._core.definitions.definitions_class import Definitions
3+
4+
5+
@asset
6+
def my_analytics_asset(): ...
7+
8+
9+
defs = Definitions(assets=[my_analytics_asset])

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/definitions.py

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
from pathlib import Path
2+
3+
import dagster_components as dg_components
4+
5+
import dagster as dg
6+
from my_existing_project.analytics import definitions as analytics_definitions
7+
from my_existing_project.elt import definitions as elt_definitions
8+
9+
defs = dg.Definitions.merge(
10+
dg.load_definitions_from_module(elt_definitions),
11+
dg.load_definitions_from_module(analytics_definitions),
12+
dg_components.build_component_defs(Path(__file__).parent / "components"),
13+
)

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/elt/__init__.py

Whitespace-only changes.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project/elt/definitions.py

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
from dagster import asset
2+
from dagster._core.definitions.definitions_class import Definitions
3+
4+
5+
@asset
6+
def my_elt_asset(): ...
7+
8+
9+
defs = Definitions(assets=[my_elt_asset])

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project_tests/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/my_existing_project_tests/test_assets.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/my-existing-project/pyproject.toml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
[project]
2+
name = "my_existing_project"
3+
version = "0.1.0"
4+
description = "Add your description here"
5+
readme = "README.md"
6+
requires-python = ">=3.9,<3.13"
7+
dependencies = [
8+
"dagster",
9+
"dagster-components",
10+
]
11+
12+
[project.optional-dependencies]
13+
dev = [
14+
"dagster-webserver",
15+
"pytest>8",
16+
]
17+
18+
[build-system]
19+
requires = ["setuptools"]
20+
build-backend = "setuptools.build_meta"
21+
22+
[tool.dg]
23+
is_code_location = true
24+
25+
[tool.dagster]
26+
module_name = "my_existing_project.definitions"
27+
code_location_name = "my_existing_project"
28+
29+
[tool.setuptools.packages.find]
30+
exclude=["my_existing_project_tests"]

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_migrating_definitions/test_migrating_definitions.py

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
import os
2+
import re
3+
import subprocess
4+
from pathlib import Path
5+
from tempfile import TemporaryDirectory
6+
7+
import pytest
8+
9+
from dagster._utils.env import environ
10+
from docs_beta_snippets_tests.snippet_checks.guides.components.utils import (
11+
DAGSTER_ROOT,
12+
EDITABLE_DIR,
13+
MASK_EDITABLE_DAGSTER,
14+
MASK_JAFFLE_PLATFORM,
15+
MASK_SLING_DOWNLOAD_DUCKDB,
16+
MASK_SLING_PROMO,
17+
MASK_SLING_WARNING,
18+
MASK_TIME,
19+
)
20+
from docs_beta_snippets_tests.snippet_checks.utils import (
21+
_run_command,
22+
check_file,
23+
compare_tree_output,
24+
create_file,
25+
re_ignore_after,
26+
re_ignore_before,
27+
run_command_and_snippet_output,
28+
)
29+
30+
MASK_MY_EXISTING_PROJECT = (r" \/.*?\/my-existing-project", " /.../my-existing-project")
31+
32+
33+
COMPONENTS_SNIPPETS_DIR = (
34+
DAGSTER_ROOT
35+
/ "examples"
36+
/ "docs_beta_snippets"
37+
/ "docs_beta_snippets"
38+
/ "guides"
39+
/ "components"
40+
/ "migrating-definitions"
41+
)
42+
43+
44+
MY_EXISTING_PROJECT = Path(__file__).parent / "my-existing-project"
45+
46+
47+
def test_components_docs_migrating_definitions(update_snippets: bool) -> None:
48+
snip_no = 0
49+
50+
def next_snip_no():
51+
nonlocal snip_no
52+
snip_no += 1
53+
return snip_no
54+
55+
with (
56+
TemporaryDirectory() as tempdir,
57+
environ(
58+
{
59+
"COLUMNS": "90",
60+
"NO_COLOR": "1",
61+
"HOME": "/tmp",
62+
"DAGSTER_GIT_REPO_DIR": str(DAGSTER_ROOT),
63+
}
64+
),
65+
):
66+
os.chdir(tempdir)
67+
68+
_run_command(f"cp -r {MY_EXISTING_PROJECT} . && cd my-existing-project")
69+
_run_command(r"find . -type d -name __pycache__ -exec rm -r {} \+")
70+
_run_command(
71+
r"find . -type d -name my_existing_project.egg-info -exec rm -r {} \+"
72+
)
73+
74+
run_command_and_snippet_output(
75+
cmd="tree",
76+
snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-tree.txt",
77+
update_snippets=update_snippets,
78+
custom_comparison_fn=compare_tree_output,
79+
)
80+
81+
check_file(
82+
Path("my_existing_project") / "definitions.py",
83+
COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-definitions-before.py",
84+
update_snippets=update_snippets,
85+
)
86+
87+
check_file(
88+
Path("my_existing_project") / "elt" / "definitions.py",
89+
COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-inner-definitions-before.py",
90+
update_snippets=update_snippets,
91+
)
92+
93+
_run_command(cmd="uv venv")
94+
_run_command(cmd="uv sync")
95+
_run_command(
96+
f"uv add --editable '{EDITABLE_DIR / 'dagster-components'!s}' '{DAGSTER_ROOT / 'python_modules' / 'dagster'!s}' '{DAGSTER_ROOT / 'python_modules' / 'dagster-webserver'!s}'"
97+
)
98+
99+
run_command_and_snippet_output(
100+
cmd="dg component scaffold 'definitions@dagster_components' elt-definitions",
101+
snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-scaffold.txt",
102+
update_snippets=update_snippets,
103+
snippet_replace_regex=[MASK_MY_EXISTING_PROJECT],
104+
)
105+
106+
create_file(
107+
Path("my_existing_project")
108+
/ "components"
109+
/ "elt-definitions"
110+
/ "component.yaml",
111+
"""type: definitions@dagster_components
112+
113+
params:
114+
definitions_path: definitions.py
115+
""",
116+
COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-component-yaml.txt",
117+
)
118+
119+
run_command_and_snippet_output(
120+
cmd="mv my_existing_project/elt/definitions.py my_existing_project/components/elt-definitions && rm -rf my_existing_project/elt",
121+
snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-mv.txt",
122+
update_snippets=update_snippets,
123+
)
124+
125+
create_file(
126+
Path("my_existing_project") / "definitions.py",
127+
"""from pathlib import Path
128+
129+
import dagster_components as dg_components
130+
131+
import dagster as dg
132+
from my_existing_project.analytics import definitions as analytics_definitions
133+
134+
defs = dg.Definitions.merge(
135+
dg.load_definitions_from_module(analytics_definitions),
136+
dg_components.build_component_defs(Path(__file__).parent / "components"),
137+
)
138+
""",
139+
COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-definitions-after.py",
140+
)
141+
142+
_run_command(r"find . -type d -name __pycache__ -exec rm -r {} \+")
143+
_run_command(
144+
r"find . -type d -name my_existing_project.egg-info -exec rm -r {} \+"
145+
)
146+
147+
run_command_and_snippet_output(
148+
cmd="tree",
149+
snippet_path=COMPONENTS_SNIPPETS_DIR / f"{next_snip_no()}-tree-after.txt",
150+
update_snippets=update_snippets,
151+
custom_comparison_fn=compare_tree_output,
152+
)
153+
154+
# validate loads
155+
_run_command(
156+
"uv run dagster asset materialize --select '*' -m 'my_existing_project.definitions'"
157+
)

0 commit comments

Comments
 (0)