`collect` should write data as it is gathered

[jhlegarreta]

What would you like to see added in this software?

The collect command does not write the files contained in each dataset tree to the corresponding TSV files until the entire dataset has been walked. If the process fails, it needs to be started from scratch, which is undesirable, as it may take a few hours. Thus, it would be desirable to:

• Write the files contained in each dataset to be written to the corresponding TSV file as they are collected.
• Be able to resume the process for the remaining dataset if the process fails by inspecting the datasets for which a TSV exists already. If the last dataset is dubious, as the datasets are walked, the tool could keep track of the last dataset that was analyzed. This may need some strategy, as datasets are collected in parallel.

Do you have any interest in helping implement the feature?

Yes

Additional information / screenshots

N/A