Fields distribution

[MarinPostma]

Summary

We want to know what fields are present in the documents stored in the database and their distribution. This is a followup on meilisearch/meilisearch#824.

Motivations

Currently, MeiliSearch stores a schema of fields it knows. This schema isn't known to the user, and we would like the user to know what fields have already be encountered while indexing, along with how many times these field is present in the database.

MeiliSearch is already able to return the distribution of indexed fields. This caused problems with the meilisearch/meilisearch#824 that was only partially implemented due to this limitation. The plan is to go even further and return the count for any encountered field.

Explanation

Querying on /indexes/:index_uid/stats or /stats returns the distribution for all the fields present in store, not only the one that are indexed.

Implementation

During indexing, a count of all encountered fields should be kept. This would probably be a good idea to outsource this responsibility to the schema.

Related

• Validated by the team
• Test added
• Documentation //Change xxx or remove the line
• SDK/Integrations //Change xxx or remove the line

[gmourier]

Hello 👋

I'm locking that discussion since it has been resolved. The current fieldsDistribution returns it extracted from the raw documents being present in the database.

Thanks!