If you partition by forest, and you have 3 forests, setting a limit of 1000 will read 3 x 1000, and write 1000 (doc + 1000 meta) to the zip.
The end result is okay, but it is a pity to waste time on reading too many records. That becomes the more painful the more forests you have.
Read logging prints nicely how many records have been collected, so flux could send a signal to the reading threads to stop reading once limit has been reached.
If you partition by forest, and you have 3 forests, setting a limit of 1000 will read 3 x 1000, and write 1000 (doc + 1000 meta) to the zip.
The end result is okay, but it is a pity to waste time on reading too many records. That becomes the more painful the more forests you have.
Read logging prints nicely how many records have been collected, so flux could send a signal to the reading threads to stop reading once limit has been reached.