Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982

sdd · 2025-02-21T09:01:51Z

Extends the DeleteFileManager introduced in #950 To include loading of delete files, storage and retrieval of parsed delete files from shared state, and the outline for how parsing will connect up to this new work.

Issue: #630

jonathanc-n

This is nice, will look at the parsed records next.

crates/iceberg/src/arrow/delete_file_manager.rs

sdd · 2025-04-03T07:09:02Z

@liurenjie1024, @Xuanwo, @Fokko - this is ready for re-review, if you could take a look that would be great!

liurenjie1024

Thanks @sdd for this pr. There are some missing points in current design. Also I would suggest not putting too much in DeleteFilterManager. I suppose DeleterFilterManager acting more like a delete loader, which manages the io and caching of record batch. The actual filtering part, could delegate to DeleteFilter, WDYT? I think a good reference implementation is java's DeleteFilter, see https://github.com/apache/iceberg/blob/af8e3f5a40f4f36bbe1d868146749e2341471586/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L50

crates/iceberg/src/delete_vector.rs

crates/iceberg/src/arrow/delete_file_manager.rs

sdd · 2025-04-14T06:38:39Z

Thanks for the review @liurenjie1024 - much appreciated. Will come back with a revised design.

liurenjie1024 · 2025-04-27T03:03:48Z

Actually @liurenjie1024 I'll go ahead with the structural changes to split this into separate loader and filter structs and update this PR.

Sorry for late reply. I'm fine with deferring performance improvement, but have concerns with correctness problem.

liurenjie1024 · 2025-04-29T07:39:54Z

Hi, @sdd I saw you opened a series of pr for handling reading of deletions. I have a suggestion about it, instead of opening a series prs, you can have a large draft pr containing all your changes, while pick one component to open a small pr for review. This way reviewer could understand your whole design by walking through the large pr, and review carefully small pr. Also, when reviewer have comments, you only need to change one large pr instead of several small ones, WDYT?

sdd · 2025-04-30T19:10:11Z

Thanks @sdd for this pr. There are some missing points in current design. Also I would suggest not putting too much in DeleteFilterManager. I suppose DeleterFilterManager acting more like a delete loader, which manages the io and caching of record batch. The actual filtering part, could delegate to DeleteFilter, WDYT? I think a good reference implementation is java's DeleteFilter, see https://github.com/apache/iceberg/blob/af8e3f5a40f4f36bbe1d868146749e2341471586/data/src/main/java/org/apache/iceberg/data/DeleteFilter.java#L50

I've refactored as you suggested and you're right, it is a better design.

sdd · 2025-04-30T19:18:15Z

Hi, @sdd I saw you opened a series of pr for handling reading of deletions. I have a suggestion about it, instead of opening a series prs, you can have a large draft pr containing all your changes, while pick one component to open a small pr for review. This way reviewer could understand your whole design by walking through the large pr, and review carefully small pr. Also, when reviewer have comments, you only need to change one large pr instead of several small ones, WDYT?

I Could do, if you think it's worth it - the other two remaining PRs after this one are much smaller and it feels like just as much work to merge those into a single PR and then break PRs out of that again as it does to simply keep rebasing those two PRs on top of this one.

I'll be very glad once this delete file read support is done - it's been a long, hard slog to be honest and I'm struggling to stay motivated with it, but we're not far off now, hopefully.

sdd · 2025-05-10T12:32:54Z

I had a bug in here that was causing the tests to deadlock in the follow-up PRs. I was missing a waker for my custom futures.That's been rectified now and this PR plus the two follow-ups are now ready for review once more.

sdd · 2025-05-14T06:34:15Z

@liurenjie1024, @Xuanwo - could do with a review again when you get chance! Thanks :-)

liurenjie1024

Thanks @sdd , it looks better now, left some questions to refine.

crates/iceberg/src/arrow/delete_file_loader.rs

liurenjie1024 · 2025-05-14T10:09:39Z

crates/iceberg/src/arrow/delete_file_loader.rs

+            CachingDeleteFileLoader::parquet_to_batch_stream(&task.file_path, self.file_io.clone())
+                .await?;
+
+        Self::evolve_schema(raw_batch_stream, schema).await


We need to have a reader api so that we could we don't need to repead this part everywhere, but this could be left to later.

crates/iceberg/src/arrow/reader.rs

liurenjie1024 · 2025-05-14T10:16:11Z

crates/iceberg/src/arrow/reader.rs

    /// Build the ArrowReader.
    pub fn build(self) -> ArrowReader {
        ArrowReader {
            batch_size: self.batch_size,
            file_io: self.file_io.clone(),
-            delete_file_manager: CachingDeleteFileManager::new(
+            delete_file_loader: CachingDeleteFileLoader::new(


We should not store a loader, instead we should store a DeleteFilter. Wha't should be called in ArrowReader should be things like following:

deleteFilter.filter(recordBatchStream)

We're not making best use of the features that ArrowRecordBatchReader provides us if we do that. We can't implement the delete filter as a simple filter on a RecordBatchStream unless we ditch using parquet's ParquetRecordBatchStream.

We're using parquet's ParquetRecordBatchStream to do the predicate filtering before we even get access to the RecordBatchStream. So by the time we have a stream of RecordBatches, we can't apply positional deletes or delete vectors because we no longer know what row number in the original file a record batches row corresponds to, as some rows can have been filtered out.

I may be able to update ArrowReader to store a DeleteFileFilter rather than a DeleteFileLoader. But not to change the semantics of how the filter itself is used to match deleteFilter.filter(recordBatchStream). Not without rewriting the ArrowReader entirely not use ParquetRecordBatchStream and as a consequence needing to reimplement all the predicate filtering logic, row selection, projection, page skipping, and row group skipping that it gives us.

I think I got your point, and you are right maybe java's interface is not best for rust implementation. I think it's fine that we move forward first and refactor later, as long as we don't expose public apis.

liurenjie1024 · 2025-05-14T10:18:27Z

crates/iceberg/src/arrow/reader.rs

@@ -207,7 +233,8 @@ impl ArrowReader {
            record_batch_stream_builder = record_batch_stream_builder.with_batch_size(batch_size);
        }

-        let delete_predicate = delete_file_manager.build_delete_predicate(task.schema.clone())?;
+        let delete_filter = delete_filter_rx.await.unwrap()?;
+        let delete_predicate = delete_filter.build_equality_delete_predicate(&task).await?;


We should hide these building of filter under DeleterFilter, rather than calling them directly in ArrowReader.

Please can you explain a bit more? I don't understand what you're asking for here. As per the previous comment, we're using ParquetRecordBatchStream to do most of the heavy lifting for us on a number of features. This means that the design of the DeleteFilter can't be as simple as just exposing a method that filters a recordbatch stream.

liurenjie1024 · 2025-05-14T10:21:09Z

crates/iceberg/src/arrow/delete_filter.rs

+
+impl DeleteFilter {
+    /// Retrieve a delete vector for the data file associated with a given file scan task
+    pub fn get_delete_vector(


It's odd to expose a public api to construct filters. For DeleteFilter, we should only expose one public api: pub fn filter(input: ArrowRecordBatchStream)

I've made all of DeleteFilter to be pub(crate) now then in that case. As per previous replies, pub fn filter(input: ArrowRecordBatchStream) does not make sense for usage with ArrowReader. If we also want to expose a public API for filtering that does implement that interface, we can add that in a follow-up PR.

crates/iceberg/src/arrow/delete_file_loader.rs

liurenjie1024 · 2025-05-14T10:27:34Z

I Could do, if you think it's worth it - the other two remaining PRs after this one are much smaller and it feels like just as much work to merge those into a single PR and then break PRs out of that again as it does to simply keep rebasing those two PRs on top of this one.

It's just a minor suggestion, just do it with your favourite approach.

sdd · 2025-05-15T06:57:21Z

@liurenjie1024 back to you. I've addressed your suggestions that I think make sense to immediately change. I look forward to hearing back from you on the other points! :-)

liurenjie1024

Thanks @sdd for this pr. Rust's reader is some how different from java, so we should not blindly follow java's api design. I think it's hard to define best api before we have a complete implementation, so I prefer to move forward first. cc @Xuanwo WDYT?

crates/iceberg/src/arrow/reader.rs

liurenjie1024 · 2025-05-15T09:45:21Z

crates/iceberg/src/arrow/reader.rs

    /// Build the ArrowReader.
    pub fn build(self) -> ArrowReader {
        ArrowReader {
            batch_size: self.batch_size,
            file_io: self.file_io.clone(),
-            delete_file_manager: CachingDeleteFileManager::new(
+            delete_file_loader: CachingDeleteFileLoader::new(


I think I got your point, and you are right maybe java's interface is not best for rust implementation. I think it's fine that we move forward first and refactor later, as long as we don't expose public apis.

…DeleteVector::intersect_assign pub(crate)

…our of it being always on

dentiny · 2025-05-16T00:52:45Z

crates/iceberg/src/arrow/caching_delete_file_loader.rs

+    ///      as per the other delete file types - only this time it is accompanied by a one-shot
+    ///      channel sender that we will eventually use to resolve the shared future that we stored
+    ///      in the state.
+    ///  * When this gets updated to add support for delete vectors, the load phase will return


Looking forward to the puffin / deletion vector support!

Me too! 😁

liurenjie1024

Thanks @sdd for this pr!

liurenjie1024 · 2025-05-16T07:34:33Z

Let's wait for a moment to merge it after 0.5.0 release

sdd force-pushed the feat/delete-fila-manager-loading branch 5 times, most recently from edb1d27 to 8e90bdd Compare February 23, 2025 14:55

This was referenced Feb 23, 2025

Delete Files in Table Scans #630

Open

Scan Delete Support Part 5: Positional Delete Parsing #1011

Open

sdd marked this pull request as ready for review February 26, 2025 09:20

sdd mentioned this pull request Mar 1, 2025

Scan Delete Support Part 6: Equality Delete Parsing #1017

Open

sdd force-pushed the feat/delete-fila-manager-loading branch 4 times, most recently from ec8e7c1 to 06f0df5 Compare March 5, 2025 19:53

jonathanc-n reviewed Mar 14, 2025

View reviewed changes

crates/iceberg/src/arrow/delete_file_manager.rs Outdated Show resolved Hide resolved

crates/iceberg/src/arrow/delete_file_manager.rs Outdated Show resolved Hide resolved

crates/iceberg/src/arrow/delete_file_manager.rs Outdated Show resolved Hide resolved

sdd force-pushed the feat/delete-fila-manager-loading branch 6 times, most recently from 5530bc3 to e997fc6 Compare March 31, 2025 17:27

sdd force-pushed the feat/delete-fila-manager-loading branch from e997fc6 to 056e73f Compare April 3, 2025 07:28

liurenjie1024 reviewed Apr 9, 2025

View reviewed changes

sdd force-pushed the feat/delete-fila-manager-loading branch 2 times, most recently from bd33aa5 to 39a26ab Compare April 17, 2025 06:39

sdd mentioned this pull request Apr 23, 2025

Add equality_ids to FileScanTaskDeleteFile #1235

Merged

sdd force-pushed the feat/delete-fila-manager-loading branch 3 times, most recently from 5739a46 to 52cf8b9 Compare April 23, 2025 21:07

sdd force-pushed the feat/delete-fila-manager-loading branch from a227fa7 to e1edc95 Compare April 30, 2025 19:08

sdd force-pushed the feat/delete-fila-manager-loading branch from e1edc95 to 324a872 Compare April 30, 2025 19:15

sdd force-pushed the feat/delete-fila-manager-loading branch 3 times, most recently from 69b4da9 to 3804bda Compare May 1, 2025 06:20

sdd force-pushed the feat/delete-fila-manager-loading branch from af14046 to 5f0b073 Compare May 10, 2025 11:24

liurenjie1024 reviewed May 14, 2025

View reviewed changes

liurenjie1024 reviewed May 15, 2025

View reviewed changes

sdd added 9 commits May 15, 2025 19:49

feat: delete file manager loading

4297f7d

feat: changes suggested in review

44e1d71

feat: return Err for unimplemented delete vec parse methods and make …

e174841

…DeleteVector::intersect_assign pub(crate)

feat: schema evolution of equality delete file record batches

112772b

refactor: split DeleteFileManager into DeleteFileLoader and DeleteFilter

301c90b

fix: add waker for DeleteFileIndex

bdee21f

refactor: remove DeleteFileFilter from CachingDeleteFileLoader

49b456d

refactor: extract BasicDeleteFileLoader from CachingDeleteFileLoader

25b83d3

feat: remove flag to selectively enable delete file processing in vav…

94d9307

…our of it being always on

sdd force-pushed the feat/delete-fila-manager-loading branch from b0eaa66 to 94d9307 Compare May 15, 2025 18:49

dentiny reviewed May 16, 2025

View reviewed changes

jonathanc-n mentioned this pull request May 16, 2025

[EPIC] Transaction Support Issues and Pull Requests #1339

Open

14 tasks

liurenjie1024 approved these changes May 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982

Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982

sdd commented Feb 21, 2025 •

edited

Loading

jonathanc-n left a comment

sdd commented Apr 3, 2025

liurenjie1024 left a comment

sdd commented Apr 14, 2025

liurenjie1024 commented Apr 27, 2025

liurenjie1024 commented Apr 29, 2025

sdd commented Apr 30, 2025

sdd commented Apr 30, 2025

sdd commented May 10, 2025

sdd commented May 14, 2025

liurenjie1024 left a comment

liurenjie1024 May 14, 2025

liurenjie1024 May 14, 2025

sdd May 14, 2025

sdd May 14, 2025

liurenjie1024 May 15, 2025

liurenjie1024 May 14, 2025

sdd May 14, 2025

liurenjie1024 May 14, 2025

sdd May 15, 2025

liurenjie1024 commented May 14, 2025

sdd commented May 15, 2025

liurenjie1024 left a comment

liurenjie1024 May 15, 2025

dentiny May 16, 2025

sdd May 16, 2025

liurenjie1024 left a comment

liurenjie1024 commented May 16, 2025

Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982

Are you sure you want to change the base?

Scan Delete Support Part 4: Delete File Loading; Skeleton for Processing #982

Conversation

sdd commented Feb 21, 2025 • edited Loading

jonathanc-n left a comment

Choose a reason for hiding this comment

sdd commented Apr 3, 2025

liurenjie1024 left a comment

Choose a reason for hiding this comment

sdd commented Apr 14, 2025

liurenjie1024 commented Apr 27, 2025

liurenjie1024 commented Apr 29, 2025

sdd commented Apr 30, 2025

sdd commented Apr 30, 2025

sdd commented May 10, 2025

sdd commented May 14, 2025

liurenjie1024 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liurenjie1024 commented May 14, 2025

sdd commented May 15, 2025

liurenjie1024 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liurenjie1024 left a comment

Choose a reason for hiding this comment

liurenjie1024 commented May 16, 2025

sdd commented Feb 21, 2025 •

edited

Loading