Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Manifest list files are not deleted #967

Open
sfc-gh-rxing opened this issue Feb 8, 2025 · 0 comments
Open

[BUG] Manifest list files are not deleted #967

sfc-gh-rxing opened this issue Feb 8, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@sfc-gh-rxing
Copy link
Contributor

sfc-gh-rxing commented Feb 8, 2025

Describe the bug

I noticed that the current implementation of dropTableWithPurge doesn’t delete snapshots. I took a look into the TaskExecution, TableCleanupTaskHandler, ManifestFileCleanupTaskHandler, and it appears that only the metadata JSON files, manifest files, and data files are removed.

To Reproduce

  1. Launch Polaris locally
  2. Create a Polaris catalog: polaris_catalog
  3. Configure Spark to connect to the local Polaris instance
  4. Create a namespace: create namespace ns;
  5. Create an Iceberg table: create table ns.t1 (id INT, name STRING);
  6. Insert some data into it: insert into ns.t1 values(1, 'XJKDC')
  7. Do drop table with purge: drop table ns.t1 purge;
  8. Check the s3 bucket to see if there are some manifest lists file remaining.

Actual Behavior

The snapshots files (manifest list files) are not deleted.

Expected Behavior

The table location should not have any files in it after we drop the table with purge.

Additional context

Introduced in this PR: #312

System information

MacOS
Lastest Polaris

@sfc-gh-rxing sfc-gh-rxing added the bug Something isn't working label Feb 8, 2025
@sfc-gh-rxing sfc-gh-rxing changed the title Manifest list files are not deleted [BUG] Manifest list files are not deleted Feb 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant