What to do with additional metadata in CSVs

[clbarnes]

I'm not sure if it's normal operation or some option, but we also get CSV files in the year, month, and day directories which have more metadata. I believe the metadata here overlaps with the metadata found in the DAT file (although the names are much more informative here), but I also think there are fields in the CSV not found in the DAT and vice versa. Ideally this information would be accessible to the next stage of processing, e.g. dat2hdf5 should be able to read it as well.

Is this CSV written to after each DAT is finished? In that case it would be a useful index for completed files.

[mkitti]

I've have to look into it further. My recollection is that the CSV files correspond to information in the "recipe" trailer, which has some overlap with the header metadata. The LabView acquisition code seems to mainly rely on reloading information from the trailer rather than the header. The code that reads the header back in is limited. I'll take a more thorough look later today.

[mkitti]

This is from Recipe read write.vi:

It would be nice if the CSV files had a strict superset of the header metadata.

[clbarnes]

Not sure if it's possible but it would also be helpful to write the path to the DAT file in the CSV, that would make loading that extra data in much easier.