Delete tasks history

[gmourier]

Today it is impossible to delete terminated tasks (failed/succeeded).

The size taken on disk by the task resources will keep growing as time goes on, and some users seem to have encountered difficulties with this aspect.

Besides the task data size being taken on disk, it is not necessarily beneficial for everyone to keep a history of tasks since day 0.

It would be interesting to automatically purge tasks that have been there for a while and/or to use the API to delete them ad-hoc.

The automatic deletion could be de-activated by default and specified at launch.

e.g. --flush-tasks-after=7d Tasks completed since 7 days are deleted from the system.

Related to #113

[gmourier]

Iteration 1

After a first design jam within the core team, we have a design proposal:

Signal: @Kerollmops, @irevoire

Note: The naming can be debated; this is an initial suggestion.

• Implement a new API endpoint to flush finished tasks. DELETE /tasks
  • Reminder: Finished tasks are tasks having a status equal to succeeded, failed, or canceled.
  • To delete a task being enqueued, it must be canceled first.
  • This endpoint will respond with a 202 Accepted if the API Key used is allowed to do so.
    • It means that the API Key used to authorize the request should have the action tasks.flush defined.
      • The Default Admin API Key will have this capacity by default due to the * being defined for the actions field.
    • A new task type is added; taskFlushing.
      • This task type will contain a dedicated details object indicating the number of matchedTasks and the number of flushedTasks.
  • Filtering parameters can be used in the request URL to specify tasks to be flushed from the disk in a fine-grained manner.
    • indexUid
    • type
    • status
    • uid
  • If filters are not specified. e.g DELETE /tasks; it will flush every older tasks being finished when the request is processed.

This addition permits the deletion of tasks resources being finished that take up space on the disk. Nothing is currently automated, as initially proposed in the discussion description. This is something that can be added for a future iteration. Our priority is to provide a way to delete old tasks.

We will conduct a second design iteration to propose a filter on dates, thus making it possible to automate the removal of tasks in the form of a rolling window using a CRON if needed.

Naming question

Is flush really the best term to choose? We could simply use deletion/ delete. WDYT? Do you have a better idea?

[dewey]

That sounds good. I feel like the naming could be a bit more explicit. For me the expection of running DELETE /tasks would be that if I call GET /tasks afterwards I'll be presented with an empty result. That under the hood it's only deleting a subset (succeeded, failed and cancelled) of tasks could lead to confusion.

Maybe it would be more obvious what's happening if this is split into:

• DELETE /tasks which will cancel all enqueued tasks and deletes them along side all succeeded, failed and cancelled tasks.
• DELETE /tasks_processed (Not sure about the naming. Other ideas: /tasks/processed, /tasks/history) which would be used in a periodic "clean up" way like through a cron job.

[irevoire]

DELETE /tasks, which will cancel all enqueued tasks and deletes them [...]

This is something we don't want to implement in the first implementation of the feature.
For the first iteration, you'll need to manually cancel your tasks and then flush them 😒.

DELETE /tasks_processed [...]

Just wondering, why do you think you need a new route and a new name? That seems harder to understand, document and discover while using a filter on the DELETE /tasks would be enough, right?
Is it because you want an alias like processed = [succeeded OR failed OR cancelled]?
Because that could be implemented in the filters and thus be used in every other /tasks route.

[dewey]

A filter would also work (to explicitly only delete a subset, while keeping the enqueued tasks), my main point is that for me as an outside consumer of the API running a DELETE /tasks which actually doesn't delete all tasks is misleading and will lead to unexpected results.

The reason for the separate route was mostly to prevent buggy behaviour where someone forgets to set the filter correctly and it wipes all enqueued tasks (instead of just the succeeded, failed, cancelled. Might not be worth extending the API interface for though so I don't feel very strongly about that.

[gmourier]

Thanks, @dewey!

That sounds good. I feel like the naming could be a bit more explicit. For me, the expectation of running DELETE /tasks would be that if I call GET /tasks afterward, I'll be presented with an empty result. That under the hood, it's only deleting a subset (succeeded, failed, and cancelled) of tasks could lead to confusion.

The reasoning is that the DELETE verb should be used to hard delete a task while a task cancelation is a state update for an existing resource.

We asked ourselves if it was beneficial to cancel tasks being asked to be deleted if they are not finished; We think it could lead to confusion to do so e.g canceling all tasks while it's not necessarily wanted.

In addition to that, having a dedicated endpoint for canceling tasks seems more versatile (people may want to cancel tasks without deleting them for historical purpose).

Having paths separated by jobs to be done seems easier to understand and deliver at first, even though we could introduce that behavior in the future to speed-up things.

DELETE /tasks_processed (Not sure about the naming. Other ideas: /tasks/processed, /tasks/history) which would be used in a periodic "clean up" way like through a cron job.

Yes, that is exactly what we thought as a use case when we decided to introduce filtering capacity on task date fields. Automating a task history cleaning process with a cron job will be possible.

In the future, this could be facilitated by implementing a native probe that automatically deletes tasks based on a specific delay in after it has been switched to a finished state. (succeeded, failed, canceled)

A filter would also work (to explicitly only delete a subset, while keeping the enqueued tasks), my main point is that for me as an outside consumer of the API running a DELETE /tasks which actually doesn't delete all tasks is misleading and will lead to unexpected results.

We could still delete all finished tasks when this URL is called. WDYT?

Does it seem to solve your immediate needs?

Thanks again @dewey

[gmourier]

Iteration 2

We will conduct a second design iteration to propose a filter on dates, thus making it possible to automate the removal of tasks in the form of a rolling window using a CRON if needed.

After a second design jam focused on introducing date filters to flush tasks in an extended way, we have a design proposal:

Signal: @Kerollmops, @irevoire

• Add new filter capacity for date fields enqueuedAt, startedAt, finishedAt as URL parameters
  • It follows the given format after|before{Date Field Name}={rfc3339 valid format}
  • Filters on dates take the outer bounds; they do not equals on the given value. e.g Writing beforeFinishedAt="2022-01-01" translates to "2021-12-31".
  • Following the RFC 3339 format it means that those expression are valid (✅) / invalid (❌)
    • Y ❌
    • Y-M ❌
    • Y-M-D ✅
    • Y-M-DTH ❌
    • Y-M-DTH:M ❌
    • Y-M-DTH:M:S ❌
    • Y-M-DTH:M:SZ ✅
    • Y-M-DTH:M:S+01:00 ✅
  • In the future, we could support easier formats to express, but this is a consequent technical work that we can't do now if we want to bring value to the product now.
• ✨ It's also available to select tasks to cancel POST /tasks/cancel or fetch GET /tasks !

[matepaiva]

Hello everyone! While this is not implemented, what is the workaround? Deleting the data.ms folder? Thank you! :)

[dewey]

Yes, I believe that's the current workaround that was shared in Slack (and which I used to recover from this).

[matepaiva]

Yes, I believe that's the current workaround that was shared in Slack (and which I used to recover from this).

Thanks!

[brunoocasali]

Hi @matepaiva, in the next release, v0.30, we will add a way to remove old processed tasks that could solve your problem without workarounds!

This is the official issue about the subject: meilisearch/meilisearch#2751

The release candidate is scheduled for October 31st.
The official release is scheduled for November 28th.

[matepaiva]

amazing! thank you!

[gmourier]

Hey everyone 👋

v0.30 introduced the possibility to delete finished tasks (succeeded, failed, canceled) 🪄

📚 More information here.

[gmourier]

Hello everyone 👋

I'm locking that discussion.

Please, feel free to open a new discussion if it's needed.

Thank you 🙇‍♂️