New CSV implementation

[Jolanrensen]

See the issue for the progress: #827

TODO:

• Split double parser into separate PR when FastDoubleParser publishes ConfigurableDoubleParser: Fast double parser #935
• Clean git history after merging fast double parser

Some high level overview of the changes:

• introduces new :dataframe-csv module
• new functions, requiring @OptIn(ExperimentalCsv::class):
  • read(Csv|Tsv|Delim)(Str) based on Deephaven
  • write(Csv|Tsv|Delim), and to(Csv|Tsv|Delim)Str based on Apache commons csv
• Added ParserOptions.skipTypes
• parse can now recognize Char
• Extended supported ColTypes to all the types we can parse
• Added OOM warning to old readCSV function informing the user about our new experimental functions :)
• See all checked off boxes in ☂ CSV rework #827 for all the bugs this fixes or issues it takes in mind.

Fixes #787
Fixes #508
Fixes #589
Fixes #469
Fixes #921

Example of the new KDocs and all supported parameters:

[Jolanrensen]

NOTE: maybe we can try {} catch(OutOfMemoryError) {} around the apache commons read csv and recommend the new experimental csv reader.

[Jolanrensen]

Maybe skipTypes be better be inverted to checkTypes. But that would mean exposing all Parsers so users/csv readers can filter which types to consider when parsing and which to skip. WDYT?

[zaleslaw]

Really weird feature, let's live a little bit with that solution to reflect later in code

[Jolanrensen]

It's mostly a performance thing, but the problem is that we don't have a user-facing way to refer to parsers, so we cannot say: "please parse this string column and try this type but not this type". I'm not sure how the API should look here

[Jolanrensen]

I managed to leverage convert to at least avoid skipTypes = allTypesExcept(...). Let's see if I can remove it altogether

[Jolanrensen]

Released dev version to test: 0.15.0-dev-4759 0.15.0-dev-4803 0.15.0-dev-4939

[Jolanrensen]

@zaleslaw @koperagen My two latest commits (https://github.com/Kotlin/dataframe/compare/703338a7042c30f8629bfc9e5979c0a6560a067f..53e2f64daa1b8167c56d5c4a4fd33a66cf78bfe2) contain a lot of new changes because I figured out how ParserOptions actually work in the library:

• There's a global setting DataFrame.parser which should be respected if either: ParserOptions == null or any of its parameters are null. I made sure the csv readers respect this too.
• I added my new ParserOptions arguments to the global one as well (and I made it not a data class anymore for better future binary compatibility).
• "skip types except" makes no sense, we have convertTo for that. But there was no overload for stringColumn.convertTo that took any ParserOptions, so I added that.
• Moved convertToDouble to convertToDoubleImpl and added explanation in KDocs for why (I think) it exists (and it now accepts nullStrings as well).
• Added more relevant tests

Could you rereview if you like?

[zaleslaw]

Is it a first global setting in KDF, or we had something earlier? Should user set up it manually and if it will be replaced is it global for the whole project, using our library? In that way it's closer to the CONFIG file which is visible or Spring Bean.

I see that it's inherited from external component, isn't it?

I'm ok, but looks like a temporary solution, could you create a ticket with research tag for that

[Jolanrensen]

Is it a first global setting in KDF, or we had something earlier? Should user set up it manually and if it will be replaced is it global for the whole project, using our library? In that way it's closer to the CONFIG file which is visible or Spring Bean.

I see that it's inherited from external component, isn't it?

I'm ok, but looks like a temporary solution, could you create a ticket with research tag for that

DataFrame.parser has been part of the library for years, I just didn't know about it yet (it's also part of the documentation https://kotlin.github.io/dataframe/parse.html, at the bottom). It's similar to Locale.setDefault() on the JVM, so yes, it's global, like a config file, but it can be changed programmatically because it's stored in a singleton object.

In all cases, the global default can also be overridden by providing a ParserOptions instance to the function, so we embrace both a functional and imperative style which I think is cool.