Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] microbatch_size should not be aligned to partition_by #11278

Open
3 tasks done
Tracked by #11292
jschintz-nytimes opened this issue Feb 5, 2025 · 0 comments
Open
3 tasks done
Tracked by #11292
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy triage

Comments

@jschintz-nytimes
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

My upstream data changed & I want to be able to run a large backfill - 8 years worth.
This could be a common occurrence, so I want to set a project variable to change on job run whether the batch size is daily or monthly.

For bigquery, today I get a config error:

Exception on worker thread. Compilation Error in model 
  The 'microbatch' strategy requires a `partition_by` config with the same granularity as its configured `batch_size`.
      Got:
        `batch_size`: month
        `partition_by.granularity`: day

Some incremental strategies don't need this (copy_partitions for sure).
Should this error only occur for merge strategies?

Describe alternatives you've considered

Just run it foe each day - but its wasted time & compute

Who will this benefit?

Users of mcirobatch - specifically BQ adapter

Are you interested in contributing this feature?

Sure

Anything else?

No response

@jschintz-nytimes jschintz-nytimes added enhancement New feature or request triage labels Feb 5, 2025
@jschintz-nytimes jschintz-nytimes changed the title [Feature] Microbatch size should not be aligned to partition_by [Feature] microbatch_size should not be aligned to partition_by Feb 5, 2025
@graciegoheen graciegoheen added the microbatch Issues related to the microbatch incremental strategy label Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request microbatch Issues related to the microbatch incremental strategy triage
Projects
None yet
Development

No branches or pull requests

2 participants