Rename indexes

[MarinPostma]

As per meilisearch/meilisearch#1080 and the public roadmap, renaming indexes is a feature we want to support. This can be done thanks to the index update route without any changes to the API.

There are a certain number of constraints that must be resolved in order to perform the rename.

The renames must be serializable with the updates to the same indexes. Say I have an index test:

1. push an update -> POST indexes/test/documents
2. rename the index -> POST indexes/test/updates rename to test2
3. push another update to the index with new name: POST indexes/test2/documents

requirements:
i) If the first update is still enqueued at the moment the index is renamed, it should not cause any problem.
ii) Any update that occurs after the rename should be linked to the correct update queue.
iii) Renames need to be asynchronous to support distributed system

i) and ii) can be addressed by resolving the index names to their Uuid before pushing them to the queue and have the rename be performed synchronously. As long as 2) occurs before 3) the name will always resolve to the correct uuid, so 1) and 3), despite being referred to with different names, will actually refer to the same index uuid.

iii) is more complicated, because 3) may be pushed to an update queue before 2) is processed. This means that test2 does not yet point to an uuid, but it still needs to be pushed to an update queue somehow. There is no way to know that this update does not refer to a newly created index and not an already existing one, so a new index will be created (cf. Lazy index creation). This is not the behavior we intend.

There is no straightforward solution for iii):

Intermediate update queue.

A way to address 3 would be to defer the resolution of the index name to it's uuid thanks to an intermediate, global, update queue.
All updates, no matter what index they concern are pushed onto a global update queue. When they are popped, their uuid is resolved and then processed in one of two ways:

• async updates: async updates concern document additions and settings updates, these updates a pushed onto the per index uuid queues, and are processed in order.
• sync updates: sync updates concern index operations, such as creation, deletions and updates. When such an update is popped from the queue, its uuid is resolved, and it is performed right away.

EDIT: the sync updates are in fact not so sync, since they are to be replicated. They must be processed before any other update on the replica nodes AND immediately taken into account on the master node so new updates are pushed in the correct queue, and performed atomically:

1. perform the rename on the master node
2. push the update on the front of the index update queue

Since synchronous updates are always processed before the next update is popped from the queue, an asynchronous update will always be resolved to the correct update uuid.

There are still issues:

• What happens if the rename fails?

This is an issue because the failure of the rename will cause the next update to point to a unexisting index, and trigger index creation. I don't see any way to prevent this, but disallowing lazy index creation.

• How do we prevent 3) from happening before 2)
  Once again, no easy solution, if any... if the order is inverted, this will cause a new index test2 to be created, and then test will be renamed to test2, effectively leaking the index and overwrite it with all the data from test. Not good. Once again, lazy index creation gets in the way.

• How do we prevent one index to be aliased with a name that already exists?

Related to the previous issue, we have to be very careful about this:

• either we disallow this altogether,
• or it effectively delete the overwritten index, with all that it implies.

-What of per index update ids?
This seems to rule out this possibility altogether, since updates are enqueued before the index they refer to is actually resolved.

We see there that renaming indexes is not at all a straightforward task, and we may need to make tradeoffs to make it possible.

Everything has not yet been thought of, and this architecture is somewhat similar to what @Kerollmops had suggested in a meeting some time ago.

EDIT 2: This issue only talks about rename, but the same goes for index deletion and swap.

[MarinPostma]

Further notes:

Algolia support for renames

Algolia support atomic index moves https://www.algolia.com/doc/api-reference/api-methods/move-index/.

However, this has a lot of restrictions:

• Moving to an index having replicas is possible, it will replace the destination index data with the source index data, and the >data will be propagated to the replicas
• Moving an index having replicas is not possible, as it would break the relationship with its replicas

also they state that the move is an expensive operation, since it is a clone and rename operation.

Atomicity seems rather complicated to ensure in our case. This would require a hard sync of all replicas in the cluster, blocking any further progression of the cluster, and performing a 2-phase commit to make sure that every replica perform the rename at the same time and succeed.

We opt for an eventually consistent rename.

[K2ouMais]

Is there any progression on this?

Thanks in advance

[fraank]

Hey @irevoire!

Sorry that's a lot of questions
--> Yep. I try to keep my answers short.

Would you see the aliases creation as an asynchronous task.
--> IMO alias creating or changing can be asynchron, thus it is not that timing relevant. For data inserting / synchronising I usually use the "real" index names. For all reading actions, the alias is used.

Ssynchronous / Should it be prioritized and executed asap?
--> Nope.

Can an alias point to multiple indexes?
--> That is the way how it works in Elasticsearch. IMO it is not really needed for HA, but a really handy feature to working with huge datasets, as I can split / combine data on behalf.

Should the tasks store the name of the alias and resolve it once they get executed, or should they resolve the name while enqueuing the task?
--> Where is the difference when doing asynchron?

Cheers!

[irevoire]

Hey, @fraank, thanks A LOT for your answers it really helps to clear a bunch of things from my head!

--> Where is the difference when doing asynchron?
Yeah, actually, if the task is asynchronous, this is not an issue; we'll resolve the name when we execute it.

The discussion is not over, internally we're thinking of first providing a simple swap API (because that's way more beginner friendly) and later, we will probably add aliases. But currently, there is no ETA

[gmourier]

Hey @irevoire and @fraank 👋

I see that the discussion switched to aliasing edge cases. That's interesting!

I have done some additional research, and I think @frankk points out an important aspect of aliasing.

If several indexes point to an alias, it allows for a rolling window of searchable indexes.

An index becomes a temporal partition, and the alias is updated to include the new partitions and discard the old ones. This is what elastic search seems to solve with the aliasing. At least, most conversations about aliasing on the net point to that aspect.

To be completely transparent, the discussion is misnamed; it is old. I'm sorry for this.

Our goal for v0.30 is to resolve as many of the blockages in the Meilisearch workflow that exists today.

It means we are working on:

• Cancelling tasks
• Deleting old tasks to free up disk space.
• Introducing a zero-downtime on the client-side when a new version of the index should be available for search and replace the old one.

It leads us to this present thought according to our time box.

If offering a dedicated swapping API solves the immediate zero-downtimes (let alone date partitioning), it's something we'd like to move towards to have a quick impact on this critical need.

We could introduce an aliasing API that addresses several needs, but it also raises a lot of edge cases and product exploration that needs to be done. (Your previous comments are considered as an exploration 😇)

Our current thinking would be to say that it might be better to release a swapping API asap (If it solves the simplest case of zero-downtime needs).

In the future, if we introduce an aliasing feature, we will probably have two ways of doing zero downtime. Either simply by swapping or by using aliasing.

The advantage would be that we would have a clear and identified path for novice users who don't need aliasing, and we would also have a more complex path for advanced users with more particular needs.

What do you think? It's a dilemma, but here I'm mainly looking to solve that need for a simple zero-downtime for users who need to replace old indexes data with newer ones without updating their client or finding a complex workaround.

Thanks a lot, guys.

[fraank]

mainly looking to solve that need for a simple zero-downtime for users who need to replace old indexes data with newer ones without updating their client or finding a complex workaround.

@gmourier Yes, that's the biggest pain point for HA. +1 for me. My initial thought was, that aliasing might be easier to implement than swapping / replacing. If that's working, aliasing can solve other (complex) needs which might be relevant later on. I would love it to be included in that discussion! 🍻

[gmourier]

Thanks for your answer and your help @fraank. I will share the temporal partitioning discussion once created. (Maybe it should be centered on aliasing and list possible needs discovered as we go along)

[gmourier]

Hey everyone 👋

v0.30 introduced a way to deploy a new index version with 0 downtimes. 🪄

📚 More information here.
📚 API Reference

[gmourier]

Hey everyone 👋

I'm locking that discussion.

Please open a new discussion if you have feedback about the index swapping or if you just need to rename an index (which is a different need) 😇

Thank you! 🙇‍♂️