Skip to content

Commit 0f397d6

Browse files
authored
update dbt_is_deleted to string (#6749)
2 parents 51b063f + 2aa02f2 commit 0f397d6

File tree

2 files changed

+73
-76
lines changed

2 files changed

+73
-76
lines changed

website/docs/docs/build/snapshots.md

Lines changed: 61 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -437,103 +437,100 @@ Snapshot <Term id="table">tables</Term> will be created as a clone of your sourc
437437

438438
In dbt Core v1.9+ (or available sooner in [the "Latest" release track in dbt Cloud](/docs/dbt-versions/cloud-release-tracks)):
439439
- These column names can be customized to your team or organizational conventions using the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config.
440-
ess)
441440
- Use the [`dbt_valid_to_current` config](/reference/resource-configs/dbt_valid_to_current) to set a custom indicator for the value of `dbt_valid_to` in current snapshot records (like a future date such as `9999-12-31`). By default, this value is `NULL`. When set, dbt will use this specified value instead of `NULL` for `dbt_valid_to` for current records in the snapshot table.
442441
- Use the [`hard_deletes`](/reference/resource-configs/hard-deletes) config to track deleted records as new rows with the `dbt_is_deleted` meta field when using the `hard_deletes='new_record'` field.
443442

444443

445-
| Field | Meaning | Usage |
446-
| -------------- | ------- | ----- |
447-
| dbt_valid_from | The timestamp when this snapshot row was first inserted | This column can be used to order the different "versions" of a record. |
448-
| dbt_valid_to | The timestamp when this row became invalidated. <br /> For current records, this is `NULL` by default <VersionBlock firstVersion="1.9"> or the value specified in `dbt_valid_to_current`.</VersionBlock> | The most recent snapshot record will have `dbt_valid_to` set to `NULL` <VersionBlock firstVersion="1.9"> or the specified value. </VersionBlock> |
449-
| dbt_scd_id | A unique key generated for each snapshotted record. | This is used internally by dbt |
450-
| dbt_updated_at | The updated_at timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt |
451-
| dbt_is_deleted | A boolean value indicating if the record has been deleted. `True` if deleted, `False` otherwise. | Added when `hard_deletes='new_record'` is configured. This is used internally by dbt |
444+
| Field | <div style={{width:'250px'}}>Meaning</div> | Notes | Example|
445+
| -------------- | ------- | ----- | ------- |
446+
| `dbt_valid_from` | The timestamp when this snapshot row was first inserted and became valid. | This column can be used to order the different "versions" of a record. | `snapshot_meta_column_names: {dbt_valid_from: start_date}` |
447+
| `dbt_valid_to` | The timestamp when this row became invalidated. For current records, this is `NULL` by default or the value specified in `dbt_valid_to_current`. | The most recent snapshot record will have `dbt_valid_to` set to `NULL` or the specified value. | `snapshot_meta_column_names: {dbt_valid_to: end_date}` |
448+
| `dbt_scd_id` | A unique key generated for each snapshot row. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_scd_id: scd_id}` |
449+
| `dbt_updated_at` | The `updated_at` timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_updated_at: modified_date}` |
450+
| `dbt_is_deleted` | A string value indicating if the record has been deleted. (`True` if deleted, `False` if not deleted). |Added when `hard_deletes='new_record'` is configured. | `snapshot_meta_column_names: {dbt_is_deleted: is_deleted}` |
452451

453-
*The timestamps used for each column are subtly different depending on the strategy you use:
452+
All of these column names can be customized using the `snapshot_meta_column_names` config. Refer to this [example](/reference/resource-configs/snapshot_meta_column_names#example) for more details.
454453

455-
For the `timestamp` strategy, the configured `updated_at` column is used to populate the `dbt_valid_from`, `dbt_valid_to` and `dbt_updated_at` columns.
454+
*The timestamps used for each column are subtly different depending on the strategy you use:
456455

457-
<details>
458-
<summary> Details for the timestamp strategy </summary>
456+
- For the `timestamp` strategy, the configured `updated_at` column is used to populate the `dbt_valid_from`, `dbt_valid_to` and `dbt_updated_at` columns.
459457

460-
Snapshot query results at `2024-01-01 11:00`
458+
<Expandable alt_header="Sample results for the timestamp strategy">
461459

462-
| id | status | updated_at |
463-
| -- | ------- | ---------------- |
464-
| 1 | pending | 2024-01-01 10:47 |
460+
Snapshot query results at `2024-01-01 11:00`
465461

466-
Snapshot results (note that `11:00` is not used anywhere):
462+
| id | status | updated_at |
463+
| -- | ------- | ---------------- |
464+
| 1 | pending | 2024-01-01 10:47 |
467465

468-
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at |
469-
| -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- |
470-
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | | 2024-01-01 10:47 |
466+
Snapshot results (note that `11:00` is not used anywhere):
471467

472-
Query results at `2024-01-01 11:30`:
468+
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at |
469+
| -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- |
470+
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | | 2024-01-01 10:47 |
473471

474-
| id | status | updated_at |
475-
| -- | ------- | ---------------- |
476-
| 1 | shipped | 2024-01-01 11:05 |
472+
Query results at `2024-01-01 11:30`:
477473

478-
Snapshot results (note that `11:30` is not used anywhere):
474+
| id | status | updated_at |
475+
| -- | ------- | ---------------- |
476+
| 1 | shipped | 2024-01-01 11:05 |
479477

480-
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at |
481-
| -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- |
482-
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 |
483-
| 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | | 2024-01-01 11:05 |
478+
Snapshot results (note that `11:30` is not used anywhere):
484479

485-
Snapshot results with `hard_deletes='new_record'`:
480+
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at |
481+
| -- | ------- | ---------------- | ---------------- | ---------------- | ---------------- |
482+
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 |
483+
| 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | | 2024-01-01 11:05 |
486484

487-
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at | dbt_is_deleted |
488-
|----|---------|------------------|------------------|------------------|------------------|----------------|
489-
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | False |
490-
| 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | 2024-01-01 11:05 | False |
491-
| 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | | 2024-01-01 11:20 | True |
485+
Snapshot results with `hard_deletes='new_record'`:
492486

487+
| id | status | updated_at | dbt_valid_from | dbt_valid_to | dbt_updated_at | dbt_is_deleted |
488+
|----|---------|------------------|------------------|------------------|------------------|----------------|
489+
| 1 | pending | 2024-01-01 10:47 | 2024-01-01 10:47 | 2024-01-01 11:05 | 2024-01-01 10:47 | False |
490+
| 1 | shipped | 2024-01-01 11:05 | 2024-01-01 11:05 | 2024-01-01 11:20 | 2024-01-01 11:05 | False |
491+
| 1 | deleted | 2024-01-01 11:20 | 2024-01-01 11:20 | | 2024-01-01 11:20 | True |
493492

494-
</details>
495493

496-
<br/>
494+
</Expandable>
497495

498-
For the `check` strategy, the current timestamp is used to populate each column. If configured, the `check` strategy uses the `updated_at` column instead, as with the timestamp strategy.
496+
- For the `check` strategy, the current timestamp is used to populate each column. If configured, the `check` strategy uses the `updated_at` column instead, as with the timestamp strategy.
499497

500-
<details>
501-
<summary> Details for the check strategy </summary>
498+
<Expandable alt_header="Sample results for the check strategy">
502499

503-
Snapshot query results at `2024-01-01 11:00`
500+
Snapshot query results at `2024-01-01 11:00`
504501

505-
| id | status |
506-
| -- | ------- |
507-
| 1 | pending |
502+
| id | status |
503+
| -- | ------- |
504+
| 1 | pending |
508505

509-
Snapshot results:
506+
Snapshot results:
510507

511-
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at |
512-
| -- | ------- | ---------------- | ---------------- | ---------------- |
513-
| 1 | pending | 2024-01-01 11:00 | | 2024-01-01 11:00 |
508+
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at |
509+
| -- | ------- | ---------------- | ---------------- | ---------------- |
510+
| 1 | pending | 2024-01-01 11:00 | | 2024-01-01 11:00 |
514511

515-
Query results at `2024-01-01 11:30`:
512+
Query results at `2024-01-01 11:30`:
516513

517-
| id | status |
518-
| -- | ------- |
519-
| 1 | shipped |
514+
| id | status |
515+
| -- | ------- |
516+
| 1 | shipped |
520517

521-
Snapshot results:
518+
Snapshot results:
522519

523-
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at |
524-
| --- | ------- | ---------------- | ---------------- | ---------------- |
525-
| 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 |
526-
| 1 | shipped | 2024-01-01 11:30 | | 2024-01-01 11:30 |
520+
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at |
521+
| --- | ------- | ---------------- | ---------------- | ---------------- |
522+
| 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 |
523+
| 1 | shipped | 2024-01-01 11:30 | | 2024-01-01 11:30 |
527524

528-
Snapshot results with `hard_deletes='new_record'`:
525+
Snapshot results with `hard_deletes='new_record'`:
529526

530-
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at | dbt_is_deleted |
531-
|----|---------|------------------|------------------|------------------|----------------|
532-
| 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | False |
533-
| 1 | shipped | 2024-01-01 11:30 | 2024-01-01 11:40 | 2024-01-01 11:30 | False |
534-
| 1 | deleted | 2024-01-01 11:40 | | 2024-01-01 11:40 | True |
527+
| id | status | dbt_valid_from | dbt_valid_to | dbt_updated_at | dbt_is_deleted |
528+
|----|---------|------------------|------------------|------------------|----------------|
529+
| 1 | pending | 2024-01-01 11:00 | 2024-01-01 11:30 | 2024-01-01 11:00 | False |
530+
| 1 | shipped | 2024-01-01 11:30 | 2024-01-01 11:40 | 2024-01-01 11:30 | False |
531+
| 1 | deleted | 2024-01-01 11:40 | | 2024-01-01 11:40 | True |
535532

536-
</details>
533+
</Expandable>
537534

538535
## Configure snapshots in versions 1.8 and earlier
539536

website/docs/reference/resource-configs/snapshot_meta_column_names.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ snapshots:
1919
dbt_valid_to: <string>
2020
dbt_scd_id: <string>
2121
dbt_updated_at: <string>
22-
dbt_is_deleted: <boolean>
22+
dbt_is_deleted: <string>
2323

2424
```
2525

@@ -35,7 +35,7 @@ snapshots:
3535
"dbt_valid_to": "<string>",
3636
"dbt_scd_id": "<string>",
3737
"dbt_updated_at": "<string>",
38-
"dbt_is_deleted": "<boolean>",
38+
"dbt_is_deleted": "<string>",
3939
}
4040
)
4141
}}
@@ -54,7 +54,7 @@ snapshots:
5454
dbt_valid_to: <string>
5555
dbt_scd_id: <string>
5656
dbt_updated_at: <string>
57-
dbt_is_deleted: <boolean>
57+
dbt_is_deleted: <string>
5858
```
5959
6060
</File>
@@ -67,17 +67,17 @@ In order to align with an organization's naming conventions, the `snapshot_meta_
6767

6868
By default, dbt snapshots use the following column names to track change history using [Type 2 slowly changing dimension](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) records:
6969

70-
| Field | Meaning | Notes |
71-
| -------------- | ------- | ----- |
72-
| `dbt_valid_from` | The timestamp when this snapshot row was first inserted and became valid. | The value is affected by the [`strategy`](/reference/resource-configs/strategy). |
73-
| `dbt_valid_to` | The timestamp when this row is no longer valid. | |
74-
| `dbt_scd_id` | A unique key generated for each snapshot row. | This is used internally by dbt. |
75-
| `dbt_updated_at` | The `updated_at` timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt. |
76-
| `dbt_is_deleted` | A boolean value indicating if the record has been deleted. `True` if deleted, `False` otherwise. | Added when `hard_deletes='new_record'` is configured. |
70+
| Field | <div style={{width:'250px'}}>Meaning</div> | Notes | Example |
71+
| -------------- | ------- | ----- | ------- |
72+
| `dbt_valid_from` | The timestamp when this snapshot row was first inserted and became valid. | The value is affected by the [`strategy`](/reference/resource-configs/strategy). | `snapshot_meta_column_names: {dbt_valid_from: start_date}` |
73+
| `dbt_valid_to` | The timestamp when this row is no longer valid. | | `snapshot_meta_column_names: {dbt_valid_to: end_date}` |
74+
| `dbt_scd_id` | A unique key generated for each snapshot row. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_scd_id: scd_id}` |
75+
| `dbt_updated_at` | The `updated_at` timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt. | `snapshot_meta_column_names: {dbt_updated_at: modified_date}` |
76+
| `dbt_is_deleted` | A string value indicating if the record has been deleted. (`True` if deleted, `False` if not deleted). |Added when `hard_deletes='new_record'` is configured. | `snapshot_meta_column_names: {dbt_is_deleted: is_deleted}` |
7777

78-
However, these column names can be customized using the `snapshot_meta_column_names` config.
78+
All of these column names can be customized using the `snapshot_meta_column_names` config. Refer to the [Example](#example) for more details.
7979

80-
:::warning
80+
:::warning
8181

8282
To avoid any unintentional data modification, dbt will **not** automatically apply any column renames. So if a user applies `snapshot_meta_column_names` config for a snapshot without updating the pre-existing table, they will get an error. We recommend either only using these settings for net-new snapshots, or arranging an update of pre-existing tables prior to committing a column name change.
8383

0 commit comments

Comments
 (0)