S3 File Writing

[radeusgd]

We have support for reading files from S3, now we want to add support for writing them as well.

• First version should not support append, just overwrite/new
  • If in append mode we should raise an error, with a message explaining it is unsupported and that read+overwrite needs to be used.
  • We should support creating a bak file in the normal way
    • Test case for: permission to modify the file, but no permission to create a .bak file.
  • File ⇔ S3 File ⇔ Enso File should be interchangeable
    • review any missing functionality on S3
  • Create a CI write bucket with the appropriate user permissions
    • Write tests should run inside of a ‘subdirectory’ to avoid interference with other runs running in parallel on CI

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2024-01-22):

Progress: Adding missing tests for S3_File read, adding tests for some metadata (similar to Enso_File). Adding tests for write. Figuring out license review issues with #8719 It should be finished by 2024-01-26.

Next Day: Next day I will be working on the same task. Finish resolving the licence review problem. Work on the added tests, implementing write.

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2024-01-23):

Progress: Fixed legal review tool issues. Fixing broken functionality in S3_File, adding some missing methods, making more tests pass. It should be finished by 2024-01-26.

Next Day: Next day I will be working on the same task. implementing write.

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2024-01-24):

Progress: Implemented writing to output stream to S3, and file deletion and duplication. TODO: integrate with our write infra (Text.write, write_bytes, Table.write) It should be finished by 2024-01-26.

Next Day: Next day I will be working on the #8805 task. Work on self type check

[enso-bot]

Radosław Waśko reports a new 🔴 DELAY for the provided date (2024-01-26):

Summary: There is 5 days delay in implementation of the S3 File Writing (#8809) task.
It will cause 2 days delay for the delivery of this weekly plan.

Delay Cause: Initial estimates focused on getting the output stream writing (done), but I did not think of the complexity of integrating the writing logic with our other facilities (Text.write, Table.write) etc.

[enso-bot]

Radosław Waśko reports a new STANDUP for the provided date (2024-01-26):

Progress: Work in progress on common logic for file writing. It should be finished by 2024-01-31.

Next Day: Next day I will be working on the same task. Continuing that.

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2024-01-29):

Progress: Realized current Writable_File approach is lacking because it relies on a single strategy for backups but S3 does not support renaming-move (it handles it as copy+delete) so the current strategy would apply pporly for S3. Designed a better approach that allows us more cusomization where needed; started implementing that. It should be finished by 2024-01-31.

Next Day: Next day I will be working on the same task. Continue implementing the customization of file write behaviour.

[enso-bot]

Radosław Waśko reports a new STANDUP for today (2024-01-30):

Progress: Implemented the new approach for local files, mostly works but a few warts to be fixed. It should be finished by 2024-01-31.

Next Day: Next day I will be working on the same task. Make tests pass, implement it for S3. Add a test for writing an Excel file to S3.

[enso-bot]

Radosław Waśko reports a new STANDUP for yesterday (2024-01-31):

Progress: Implemented the file write strategy with backup for S3 backend. Fixing bugs in path traversal (/, parent) logic on S3. Most tests are now passing, just 2 failures to investigate. Created draft PR. It should be finished by 2024-01-31.

Next Day: Next day I will be working on the same task. Fix the remaining tests, final touches to get the PR ready. Then Types work: Get the #8867 merged (fix remaining test failures).

[enso-bot]

Radosław Waśko reports a new 🔴 DELAY for today (2024-02-01):

Summary: There is 2 days delay in implementation of the S3 File Writing (#8809) task.
It will cause 1 days delay for the delivery of this weekly plan.

Delay Cause: I needed more time to correctly implement backup logic, was making some mistakes at first. I also ran into an unexpected very nasty bug that took quite some time to track down (#8937).

[enso-bot]

Radosław Waśko reports a new STANDUP for today (2024-02-01):

Progress: Fixed remaining tests, backup logic now is tested. Discovered an issue when trying to decode unknown format - instead of a proper error, 'uninitialized value' is returned. After some digging, this led me to discover #8937. It should be finished by 2024-02-02.

Next Day: Next day I will be working on the same task. Find some workaround for #8937 to get #8809 finished. Get the pending #8867 ready to merge (fixing Array/Vector helpers). Do some work on types (as did not do it on thursday after all).

[enso-bot]

Radosław Waśko reports a new STANDUP for the provided date (2024-02-02):

Progress: Implemented a workaround. Fixing remaining tests. Initial work on refactor for #8805. Found a local repro for #8910. It should be finished by 2024-02-02.

Next Day: Next day I will be working on the #8833 task. Finish the PR. Look into next tasks.