This is true; it's the reason for #270 but ideally we should make the operation reliable.

This issue is also mentioned in #1179. I can help patch this by:

1. When a cleanup task fails, persist and requeue it instead of retrying inline (to prevent losing track if the service dies).
2. Recover failed tasks on Polaris startup.
3. Drain the queue on a regular interval (e.g., every 15 minutes) to ensure no tasks remain pending.

But in the long term, we might want to move this purge execution logic to the Table Maintenance Service #538 for better scalability.

I think that it's also very important that no two instances run the same task.
Otherwise it's not safe to run Polaris in an HA/LB setup.

I think that it's also very important that no two instances run the same task. Otherwise it's not safe to run Polaris in an HA/LB setup.

Yup, we prevent this from happening by storing a LAST_ATTEMPT_START_TIME for each task entity.

When loadTasks is called, each task is selected based on whether it’s timed out (LAST_ATTEMPT_START_TIME < now - TASK_TIMEOUT_MILLIS), and then updated transactionally to set a new LAST_ATTEMPT_START_TIME and assign to the executor.

So in an HA/LB setup, once one executor picks and updates a task, no others will be able to pick the same one until it times out again.