Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task handling is incomplete #774

Open
snazy opened this issue Jan 15, 2025 · 1 comment
Open

Task handling is incomplete #774

snazy opened this issue Jan 15, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@snazy
Copy link
Member

snazy commented Jan 15, 2025

Describe the bug

Polaris uses some asynchronously executed tasks to run operations for table and manifest file cleanup. Those tasks are potentially executed in a separate thread in the same JVM. There is however no guarantee that those tasks will eventually run for multiple reasons:

  • Tasks (e.g. via org.apache.polaris.service.catalog.BasePolarisCatalog#dropTable) are triggered after the fact.
  • Although tasks are persisted, there is no mechanism to pick up tasks that did not start or did not finish ("long lasting" failures, JVM terminates).

Overall this means that for example a "drop table with purge" returns a successful result to the user, the actual purge may never ever happen.

To Reproduce

No response

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

@snazy snazy added the bug Something isn't working label Jan 15, 2025
@eric-maynard
Copy link
Contributor

This is true; it's the reason for #270 but ideally we should make the operation reliable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants