Skip to content

Commit

Permalink
fix: Reinstate our own connection name function
Browse files Browse the repository at this point in the history
The connection name cleaning function from Dagster works, but doesn't map to the same
values which prevents properly associating the schedule history.
  • Loading branch information
blarghmatey committed Feb 10, 2025
1 parent c1f8ed8 commit 1fe494d
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions src/ol_orchestrate/definitions/lakehouse/elt.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,16 @@
key_prefix="ol_warehouse_raw_data",
connection_filter=lambda conn: re.search(r"S3 (Glue )?Data Lake", conn.name)
is not None,
connection_to_group_fn=(
# Airbyte uses the unicode "right arrow" (U+2192) in the connection names for
# separating the source and destination. This selects the source name specifier
# and converts it to a lowercased, underscore separated string.
lambda conn_name: re.sub(
r"[^A-Za-z0-9_]", "", re.sub(r"[-\s]+", "_", conn_name)
)
.strip("_")
.lower()
),
)


Expand Down

0 comments on commit 1fe494d

Please sign in to comment.