Add a way to force search results to be exhaustive

[gmourier]

Description

We are opening this discussion to gather and consolidate feedback that are related to the fact that it is not possible to force MeiliSearch to return an exhaustive number of results.

The initial idea would be to have a way to force the search to return exhaustive results knowing that it will impact performance in a negative way.

Related discussions

• Make CANDIDATES_THRESHOLD configurable without recompilation #327
• Inconsistent nbHits with different offset meilisearch#1993
• get exact number of records on request meilisearch#1929

[homerjam]

I would like this feature as I'm using Meilisearch as the primary method of querying/filtering/sorting data in my stack, a small performance penalty would be an acceptable compromise and I will implement a cache in any case.

Another reason I would like this feature is in order to retrieve a definitive list of all the IDs in an index for verification that everything has synced from the primary database. Alternatively, a simple method just to get those IDs would work.

[gmourier]

Hello @homerjam

Another reason I would like this feature is in order to retrieve a definitive list of all the IDs in an index for verification that everything has synced from the primary database. Alternatively, a simple method just to get those IDs would work.

Did you tried to use the indexes/{indexUid}/documents endpoint in combination with limit/ offset parameters? You can also only retrieve the id by providing attributesToRetrieve=id as a query parameter

[curquiza]

More explanation about why the nbHits returned by Meilisearch is not exhaustive: #467

[ManyTheFish]

Hello @homerjam!
We are working on improving pagination in Meilisearch and we just released a first prototype that makes creating numbered pagination way easier! 🔥
It would be super helpful if you could let us know if this helps with your issue.
To know more about the prototype and how to use it, take a look at the dedicated discussion.

[mmachatschek]

on our end, the estimatedTotalHits returned a fairly accurate number until we started to use the distinctAttribute feature https://docs.meilisearch.com/reference/api/settings.html#distinct-attribute

now the numbers are off by ~20%

having the option to tell the search that we want to receive an exact count would be great.

possible fixes to work around this issue:

• add a specific attribute to one of the documents that is distinguished by the distinctAttribute
• fetch all documents in the index for this search (limit: 1000000) and get the exact count

[mmachatschek]

hi @curquiza after reading it three times now it starts to make sense. probably I'm just not reading careful enough.

These new search parameters can be used in the place of offset and limit when creating interfaces that require knowing the exhaustive number of search results.

this mentions it.

[curquiza]

Thanks @mmachatschek, we have a PR regarding the changelog, we will try to update this, you can follow the discussion (and participate if you want and have the time) here

[mmachatschek]

hi there @ManyTheFish @curquiza

I wasn't able to test v0.30 until now but for me, using page and hitsPerPage do not show an exhaustive number of hits.

using different values of page and hits per page show different numbers.

I can share the dataset privately on discord if needed

Check this (the index data is not changing while testing):

(page: 0, hitsPerPage 10, results: 2539)

(page: 1, hitsPerPage 10, results: 2531)

(page 1, hitsPerPage 20, results: 2510)

(page 2, hitsPerPage 10, results: 2510)

(page 2, hits per Page 20, results: 2479)

[gmourier]

Thank you @mmachatschek !

[mmachatschek]

I created an issue from my comment here: meilisearch/meilisearch#3200

[gmourier]

Hey everyone 👋

v0.30 introduced a new way to navigate search results. It's now possible to implement an exhaustive page selection component. 🪄

📚 More information here.

[gmourier]

Hey everyone 👋

I'm locking that discussion.

Please open a new discussion if you want to share feedback / ask for a feature request.

Thank you! 🙇‍♂️