[Feature] microbatch_size should not be aligned to partition_by #11278
Labels
enhancement
New feature or request
microbatch
Issues related to the microbatch incremental strategy
triage
Is this your first time submitting a feature request?
Describe the feature
My upstream data changed & I want to be able to run a large backfill - 8 years worth.
This could be a common occurrence, so I want to set a project variable to change on job run whether the batch size is daily or monthly.
For bigquery, today I get a config error:
Some incremental strategies don't need this (copy_partitions for sure).
Should this error only occur for merge strategies?
Describe alternatives you've considered
Just run it foe each day - but its wasted time & compute
Who will this benefit?
Users of mcirobatch - specifically BQ adapter
Are you interested in contributing this feature?
Sure
Anything else?
No response
The text was updated successfully, but these errors were encountered: