feat!(io): FileIO::list_prefix — recursively list files under a prefix

[raghav-reglobe]

Summary

Adds a recursive file-listing capability to FileIO, the missing primitive for table-maintenance operations (orphan-file cleanup foremost) that need to enumerate everything under a table location and compare it against paths recorded in table metadata.

• ListEntry { path, size, last_modified_ms } — paths come back absolute, in the same form as the listed prefix, so they round-trip directly to other FileIO operations and compare against metadata-recorded paths without normalization.
• last_modified_ms is Option: storages that don't track modification times report None, and age-based callers must treat such files as not eligible (protects possibly-in-flight writes).
• Storage::list_prefix ships with a FeatureUnsupported default and is implemented for the OpenDAL storages (recursive lister, entries mapped back to absolute form), the resolving router, local fs (recursive walk with real mtimes), and memory (which now stamps write times so age-based logic is unit-testable).

Motivation

First consumer is a RemoveOrphanFilesAction (Java/Go parity — cf. the maintenance-operations proposal in #1453), which I'll submit as a follow-up PR stacked on this one. Kept separate so this stays a small, standalone review.

Validation

• Unit tests for memory + local fs list_prefix (prefix-form round-trip, sizes, mtimes, empty-prefix behavior).
• Full iceberg suite: 1377 passed. cargo fmt clean; public-api.txt regenerated.

[u70b3]

Could we clarify the exact semantics of FileIO::list_prefix in the API docs, and ideally add a regression test for the edge case where the input points to an existing file?

I did a quick local check on this PR head by adding temporary tests for MemoryStorage and LocalFsStorage:

• write memory://warehouse/t/data/a.parquet
• call list_prefix("memory://warehouse/t/data/a.parquet")
• result is empty

The same is true for local fs: if the input path is an existing file, LocalFsStorage::list_prefix returns an empty list because it first checks !root.is_dir(). For memory, the implementation normalizes a non-trailing-slash path into path/, so an exact file key does not match either. OpenDAL appears to follow the same directory-style behavior by listing relative_path/.

That behavior may be exactly what we want for the intended first consumer, orphan-file cleanup, because the operation is normally scoped to a table location or a data/metadata directory. This also lines up with the Iceberg docs/spec side: the table metadata location is the table's base location used for data files, manifests, and metadata files; maintenance docs describe orphan cleanup as removing files "under a table location", and Spark's remove_orphan_files location argument is documented as a directory to look for files in.

However, the name list_prefix can also be read as raw object-key prefix matching. I cross-checked the Java API and it is intentionally explicit about this ambiguity: SupportsPrefixOperations.listPrefix says hierarchical file systems may require the prefix to fully match a directory, while key/value object stores may allow arbitrary prefixes. Java's orphan-file listing path also normalizes the directory by appending / before calling listPrefix, which suggests directory/table-location traversal is the expected maintenance use case. On the other hand, Java HadoopFileIO can return a single file for an exact file path via FileSystem.listFiles(path, true), so Rust's current local/memory behavior is not exactly the same for that edge case.

So I think either behavior can be defensible, but the contract should be explicit. Could we document whether list_prefix is intended to:

1. list files under a directory/table-location prefix only, excluding an exact file-path match, or
2. behave like raw prefix matching / Java-style listPrefix, where an exact file path may be returned by some implementations?

If the current directory-style semantics are intentional, a small test asserting that listing an exact file path returns empty would make the contract much clearer for future orphan-file cleanup callers.

[raghav-reglobe]

Thanks for digging into this — running temp tests against the branch head is exactly the check this needed, and you're right that the contract was implicit.

The directory-style behavior is intentional, so I've gone with your option 1 and made it explicit. Two reasons: the intended callers are maintenance operations (listing a table location or its data//metadata/ directories for orphan-file cleanup), and it's also the semantics OpenDAL gives us naturally — supporting raw object-key prefix matching would mean fighting the underlying library for a case no caller needs. Rather than inherit the "may be either, depending on the implementation" ambiguity from SupportsPrefixOperations, the contract is now pinned to directory-style uniformly:

• docs on FileIO::list_prefix (and the Storage trait) spell out the normalization: a prefix without a trailing / gets one appended, so list_prefix("…/t") ≡ list_prefix("…/t/"), a path naming an existing file yields an empty list (exists() is the tool for that), and a missing prefix is empty rather than an error;
• added the regression tests you suggested for MemoryStorage and LocalFsStorage, asserting the exact-file-path case returns empty plus the trailing-slash equivalence.

Pushed as 4bc0260. If you think there's a real caller for the Hadoop-style "exact file path returns itself" edge, happy to reconsider — but one fixed semantic seems better than implementation-defined here.

[u70b3]

That makes sense to me. Pinning this to one directory-style contract seems easier for callers than inheriting implementation-defined behavior.

I did one more quick local check against the OpenDAL memory backend: raw op.list_with(exact_file).recursive(true) can return the exact file, while OpenDalStorage::list_prefix(exact_file) returns empty because this adapter appends / before listing. So the current implementation does enforce the uniform directory-style behavior.

I opened a small PR with a regression test for that adapter behavior here, in case you think it would be useful to include in this PR: raghav-reglobe#1

[raghav-reglobe]

That raw-list_with vs adapter contrast is exactly the right level to pin it at — it proves the directory-style behavior comes from this adapter's /-append, not from OpenDAL itself, which is precisely what would regress silently if someone "simplified" that line away. Merged your PR into the branch — thanks for going the extra step of writing it rather than just flagging it. Ran the crate suite on top: all green.