Csvhandlerpolar

[mathysgrapotte]

There are now two classes for handling csv, CsvParser (for parsing the data, doing noising etc.) and CsvLoader (for loading the data into models). Additionally both now inherit from the same CsvHandler class.

Then, Polars has replaced Pandas as it is much faster.

[mathysgrapotte]

Please do not delete branch as it requires testing of some functionalities.

[mathysgrapotte]

branch can be deleted

[alessiovignoli]

The metadata now is the belonging to one of the sets. But is the split function (selected by user) going to assign this value modifying the csv in input?

[mathysgrapotte]

Yes, the split:meta:int column is special, it will be added or modified by the CsvParser but can also be passed by the user in the input csv

[alessiovignoli]

ok i like it but the skip of split when user passes it should be handled correctly. Can you open a issue for it?

[mathysgrapotte]

How do you see this happening ? Currently, we throw an error if there is no split:meta:int column, I think that even if a split:meta:int column exists, if a splitter is called, this wil be overwritten, if a splitter isn't called, it will stay as is.

[alessiovignoli]

the first error is more than fine because it caver cases in which the user did not specify neither manually neither explicitly a splitter scheme. I would like to have another error in case that column already exist and the split mthod is called on such a csv. Going something like "a split meta column is already present but a splitter function has been called. remove the split meta column if you want to call a split function."