-
Notifications
You must be signed in to change notification settings - Fork 281
[Feature] Improve how Cosmos renders & runs seeds #1576
Copy link
Copy link
Open
Labels
area:executionRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcarea:renderingRelated to rendering, like Jinja, Airflow tasks, etcRelated to rendering, like Jinja, Airflow tasks, etccustomer requestAn Astronomer customer made requested thisAn Astronomer customer made requested thisdbt:seedPrimarily related to dbt seed command or functionalityPrimarily related to dbt seed command or functionalityenhancementNew feature or requestNew feature or requeststaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestoneItems need to be reviewed / assigned to milestone
Milestone
Metadata
Metadata
Assignees
Labels
area:executionRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcarea:renderingRelated to rendering, like Jinja, Airflow tasks, etcRelated to rendering, like Jinja, Airflow tasks, etccustomer requestAn Astronomer customer made requested thisAn Astronomer customer made requested thisdbt:seedPrimarily related to dbt seed command or functionalityPrimarily related to dbt seed command or functionalityenhancementNew feature or requestNew feature or requeststaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestoneItems need to be reviewed / assigned to milestone
Description
Currently, if the dbt project contains seeds, Cosmos will render them and attempt to run
dbt seedunless the end-user customises how Seeds should be rendered:https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#customizing-how-nodes-are-rendered-experimental
However, in most cases, seeds do not need to be continuously run.
It would be great if Cosmos could allow users to opt for different ways of rendering/running seeds, similar to source nodes:
SeedRenderingBehavior.ALWAYS: Current behaviour of Cosmos, always add the seed tasks in the DAG/TaskGroup and run seedsSeedRenderingBehavior.NONE: New behavior, don't render any seeds in DAG/TaskGroupSeedRenderingBehavior.WHEN_SEED_CHANGES: Only render and run a seed if thecsvfile has changed since the last executionThere are a few open questions:
SeedRenderingBehavior.NONEandSeedRenderingBehavior.WHEN_SEED_CHANGES, and use anEmptyOperatorin those cases?.csvwas run and if it changed? As an Airflow variable? In a remote object store? Somewherelse..?I'd love your thoughts on these ideas.
Use case/motivation
Allow users to opt out of running seeds and only run seeds when needed.
Related issues
No response
Are you willing to submit a PR?