-
Notifications
You must be signed in to change notification settings - Fork 326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add parser for line by line processing #8719
Conversation
ab8ec9e
to
915b253
Compare
90f1c8c
to
f766ca7
Compare
c92bd0a
to
cad33e9
Compare
## PRIVATE | ||
Convert from a Text to a File. | ||
File.from (that:Text) = File.new that |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been adding a similar conversion as part of #8809 work, I think it may be worth discussing some future improvements for it as we actually may want to support not only local paths but also e.g. s3://
paths as well, if AWS
is imported.
My idea is to do a File_Like.from (that:Text) = ...
that would rely on a ServiceProvider
logic that allows to register new protocols for resolving files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy for us to end up with that once yours is ready just felt a good change to get rid of all the File.new
dotted about and use the conversion point to catch on the way through.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't File
a builtin? If so, then it is backed by TruffleFile and not java.io.File
. TruffleFile
allows any kind of FileSystem including S3, once somebody writes it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, I think we were not aware of this. We may consider going in this direction, but right now we are not handling S3 at builtin level, but as a separate type (S3_File
), same as our cloud files (Enso_File
).
This approach does allow us to do some more special handling for these special files and have better control of what is supported and what is not. For example, the S3_File
holds AWS credentials associated to it, in case the user is working with files from different accounts, I don't think that would be possible to implement something like that using the TruffleFile
abstraction.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, moreover - S3 support is included in a separate library (Standard.AWS
) as it should not really be part of Base
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do think we should investigate the options of a FileSystem but that is one for a little down the road.
distribution/lib/Standard/Base/0.0.0-dev/src/System/File_By_Line.enso
Outdated
Show resolved
Hide resolved
distribution/lib/Standard/Base/0.0.0-dev/src/System/File_By_Line.enso
Outdated
Show resolved
Hide resolved
distribution/lib/Standard/Base/0.0.0-dev/src/System/File_By_Line.enso
Outdated
Show resolved
Hide resolved
09e3bb6
to
6732be0
Compare
distribution/lib/Standard/Table/0.0.0-dev/src/Data/Conversions/Convertible_To_Columns.enso
Show resolved
Hide resolved
distribution/lib/Standard/Table/0.0.0-dev/src/Data/Conversions/Convertible_To_Rows.enso
Outdated
Show resolved
Hide resolved
distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Fan_Out.enso
Outdated
Show resolved
Hide resolved
distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Fan_Out.enso
Outdated
Show resolved
Hide resolved
Expand Vector (and like things) to columns.
a4f3335
to
2062247
Compare
Pull Request Description
File.from that:Text
and useFile
conversions instead of taking bothFile
andText
and callingFile.new
.Date_Time
using it.long
based array builder.File_By_Line
to read a file line by line.to_vector
to be a proxy Vector.at
andget
toDatabase.Column
.get
toTable.Column
.Vector
,Array
Range
,Date_Range
to columns.expand_to_column
default column name will be the same as the input column (i.e. noValue
suffix).Map
,JS_Object
andJackson_Object
to rows with two columns coming out (and extra key column).Important Notes
Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
./run ide build
.