Hierarchical Facet Search

[bradenmacdonald]

Hi!

I'm trying to build a search experience like this using Meilisearch, where there are hierarchical tags used to categorize content. The tags come from external taxonomies and there could be thousands of tags in each taxonomy, so we want a way for users to be able to search in the "tags" filter before checking/unchecking tags to filter by.

I was hoping that the "Searching facet values" feature would support this, but it doesn't seem to work for this use case at all.

For context, we have the hierarchical tags denormalized into several fields as recommended in the guide: tags.level0, tags.level1, tags.level2, tags.level3, etc. This works great for basic filtering and since I specified tags as a filterable attribute when creating the index, I can do basic filtering using the tags. It also works with Instansearch's HierarchicalMenu widget, though we had to replace it with a custom implementation because their widget only allows selecting a single tag at a time.

However, here are the problems:

1. It seems like it's not possible to do a keyword search on the hierarchical facets - if I call client.index(indexName).searchForFacetValues({ facetName: "tags", facetQuery: "..." }); then Meilisearch gives empty results (which I guess is expected, because the API doesn't even have a way to express nested taxonomy values in the result from this API call). And if I call searchForFacetValues with facetName: "tags.level0", it gives an error that "Attribute tags.taxonomy is not facet-searchable. To make it facet-searchable add it to the filterableAttributes index settings." This seems inconsistent with how filtering works in general, because for all other API methods, I only needed to include tags in the filterableAttributes and it allows me to filter and facet on all the sub-fields of tags. I did try adding tags.level0 etc. to filterableAttributes, and it does work, but this is still problematic because I want to search the whole tree of hierarchical facets, and I would have to make at least four separate API calls to search each level and then combine the results.
2. Even when searchForFacetValues does work, it's far too limited for our use case. As you can see in the screenshot, there are many tags with multiple words in the name, and searchForFacetValues only allows searching on the first word - which would be a confusing and broken experience for users.

So, I guess this is a feature request for two things:

1. Allow searchForFacetValues to search sub-fields like tags.level0 and tags.level1 simultaneously.
2. Allow better keyword search on facet values. I would prefer it work just as if I did a regular query on the tags compound field; for example, .search("Middle Word", { attributesToSearchOn: ["tags.level0", "tags.level1", ...]}) will match text from the middle of the facet names; the only problem with this approach is that it's not possible to combine with an existing overall query keyword, like the searchForFacetValues field allows.

Relates to:

• https://roadmap.meilisearch.com/c/97-nested-facets
• https://blog.meilisearch.com/nested-hierarchical-facets-guide/
• https://github.com/orgs/meilisearch/discussions/544

[macraig]

Hi @bradenmacdonald , apologies for the delayed response. We've opened an issue to investigate the behavior you mentioned here: meilisearch/meilisearch#4638

[bradenmacdonald]

Thanks @macraig. I commented on that issue too.

For reference, here is the code I ended up writing to search hierarchichal facets in a "best effort" fashion, despite the current limitations of the Meilisearch API: https://github.com/openedx/frontend-app-course-authoring/blob/732b7ed86cb3845fbed0471a217c065c14091b3b/src/search-modal/data/api.js#L318-L388
I would definitely prefer a "proper" solution that does a facet search rather than this attempt to reconstruct the matching facets on the frontend.

[bradenmacdonald]

I'm excited to hear about meilisearch/meilisearch#4693 / meilisearch/meilisearch#4611 because I think it will solve the main limitation of the "hack" that I'm using to support hierarchical search. By using a distinct: tags.level0, tags.level1, tags.level2 condition in the search, it will allow us to perform a keyword search against hierarchical facets and get a complete set of facet results. I think when you announce the feature, you may even want to point out how it can be used to support hierarchical facet keyword search.

Though it would still be much nicer to have a "facet keyword search" that supports hierarchical fields built in to Meilisearch :)

[curquiza]

Hello @bradenmacdonald thank you for this feedback
I will let you know once the RC containing the feature is out, so that you can test it, if you are interested!
If it works as expected for you, I will add this to our changelogs and communication (I'm responsible for the changelogs)

[bradenmacdonald]

Yes, I'd definitely like to test it and see how it works for this use case! Thanks.

[curquiza]

Hello @bradenmacdonald
the distinct attribute at search is available in the new Meilisearch RC: https://github.com/meilisearch/meilisearch/releases/tag/v1.9.0-rc.3

How to use:

• Put the fields you want to use the distinct on in filterableAttributes
• Use the parameter distinct at search

[bradenmacdonald]

Thanks @curquiza ! I tried it out today, but I realized it's not going to help with hierarchical facet keyword search, unfortunately. The problem is that I need distinct results among multiple fields (tags.level0, tags.level1, tags.level2) but the new distinct keyword only allows specifying a single distinct field. I tried specifying distinct: "tags" but it had no effect on the sub-fields.

So I guess we're back to needing a better version of the facet search endpoint. I'm going to try to post a clarification of my request as a separate comment on this thread; perhaps my initial post wasn't as clear as can be.

[bradenmacdonald]

I'm going to try to clarify the feature request I'm asking for here.

Consider a search index with thousands of documents about computer systems, and hierarchical tags used to categorize them. The hierarchical tags are structured as recommended in the Meilisearch Guide to hierarchical faceted search.

Here are a few of the documents:

[{
    "id": 1, "name": "MacBook M1",
    "tags": {
        "level0": ["Architecture", "Manufacturer"],
        "level1": ["Architecture > ARM", "Manufacturer > Apple"]
    }
},{
    "id": 1, "name": "MacBook M2",
    "tags": {
        "level0": ["Architecture", "Manufacturer"],
        "level1": ["Architecture > ARM", "Manufacturer > Apple"]
    }
},{
    "id": 1, "name": "Old MacBook",
    "tags": {
        "level0": ["Architecture", "Manufacturer"],
        "level1": ["Architecture > Intel", "Manufacturer > Apple"]
    }
},{
    "id": 1, "name": "Old iMac",
    "tags": {
        "level0": ["Architecture", "Manufacturer"],
        "level1": ["Architecture > Intel", "Manufacturer > Apple"]
    }
}]

Now, our goal is to implement this UX where users can refine their search using a keyword search and/or selecting multiple tags from the list of hierarchical tags:

However, because there are thousands and thousands of hierarchical tags, we only want to display the tags that match the current overall search keywords ("MacBook"), and we want to allow users to do a keyword search among the resulting tags to reduce the set of possible tag filters even further ("ARM" keyword in this example).

In order to implement this efficiently, a new version of the Facet Search endpoint is necessary, which could be used like this:

Retrieve all the "level0" tags that match the keywords or have sub-tags that match the keywords:

  const { facetHits } = await client.index(indexName).searchForFacetValues({
    facetName: "tags.level0",
    facetPrefix: "",
    facetKeywords: "ARM",
    facetKeywordsAttributesToSearchOn: ["tags.level0", "tags.level1", "tags.level2"],
    q: "MacBook",
    filter: [...otherFilters],
  });

Which would return {"facetHits": [ { "value": "CPU", "count": 1, "indirectHits": true } ] }

Then, if the user expands the "Architecture" tag in the UI, we need to load all of the "level1" tags below "Architecture" that match the keyword(s) and filter(s):

  const { facetHits } = await client.index(indexName).searchForFacetValues({
    facetName: "tags.level1",
    facetPrefix: "CPU > ", // This is currently called "facetQuery" but "facetPrefix" would be more accurate
    facetKeywords: "ARM",
    facetKeywordsAttributesToSearchOn: ["tags.level1", "tags.level2"],
    q: "MacBook",
    filter: [...otherFilters, "tags.level0 = Architecture"],
  });

As you can see, supporting hierarchical search well requires adding support for facetKeywords and facetKeywordsAttributesToSearchOn to the facet search endpoint. I also think that facetQuery should be renamed to facetPrefix to reduce confusion. It would also be extremely valuable to have the results include a "indirectHits": true/false that indicates whether the facetKeywords match was found in the facetName facet or in one of the other facetKeywordsAttributesToSearchOn facets - this allows the UI to display an expand/collapse widget for tags that have children, but hide it for leaf tags that don't have children.

(I did manage to implement the UI shown with the current version of Meilisearch, and you can see my code linked above, but it requires making a lot more separate queries than it should, and it depends on a hack which means that the results are sometimes incomplete. I believe that the suggested improvements to searchForFacetValues would resolve it.)

[mmmcorpsvit]

i too have similar issue:

import pprint

import meilisearch

client = meilisearch.Client('http://127.0.0.1:7700', 'meili_master_key')
index_name = 'cars'
documents = [
    {'id': 1, 'slug': 'damage1', 'name': 'Damage 1', 'name_ua': 'Пошкодження 1'},
    {'id': 2, 'slug': 'damage3', 'name': 'Damage 3', 'name_ua': 'Пошкодження 3'},
    {'id': 3, 'slug': 'damage3', 'name': 'Damage 3', 'name_ua': 'Пошкодження 3'},
]
client.create_index(uid=index_name, options={'primaryKey': 'id'})
client.index(index_name).add_documents(documents)
client.index(index_name).update_settings(
    {
        'filterableAttributes': ['id', 'slug', 'damage', 'name', 'name_ua'],
    }
)

results = client.index(index_name).facet_search(
    'slug',
    opt_params={
        'attributesToRetrieve': ['name', 'name_ua'],
        'facetAttributes': ['slug'],
    }
)

pprint.pprint(results, depth=10, indent=4)

# how get count, and additionally: "name", "name_ua"?

https://blog.meilisearch.com/id-based-facets-guide/ - only one way - merge id and data, but this very ugly idea - because very slow)

are there any plans to implement this feature?

[macraig]

Hi @mmmcorpsvit , we currently don't have plans to implement this for now, but we are open to considering it in the future. In the meantime we'll leave the discussion open for other users to share their needs and potential workarounds.