Authentication Layer

[MarinPostma]

Summary

Right now, Meilisearch only support very basic authentication with 3 authentication layers (public, private and admin) and 3 associated keys. Restriction are hard-coded for each route, statically defining what each user is allowed to see based on his privileges.

This tracking issue proposes extensions to what already exists to allow for more flexibility in the API authentication Layer of Meilisearch.

Motivations

This feature has been asked multiple times, and should address business requirements that our users may have.

Explanation

Only the master key will remain by default, and has R/W rights on everything.

It will be possible to generate other API keys and link them to permission on the engine.

routes already protected with private, will remain so by default on will only be accessible with the master key.

The authentication layer will support:

• ACL: keys can be associated with R/W permissions on on every sub-routes
• Protected indexes: indexes access can be restricted to set of keys. By default, an index remain public.
• Restrict usage of some parameter: (Dis)allow some parameters to be overwritten by the user with insufficient privileges

Documentation

Protected Indexes

A new field is added to the index creation/update payload called authorizedKeys. It is an array of strings that defaults to []. An empty array means that no specific restriction are applied to the index. If the array is not empty, only keys in the array are authorized to perform actions on it.

A call to indexes or indexes/:index_uid returns the list of authorized keys for each index in the field authorizedKeys

Trying to access an index with insufficient rights results in a protected_index error code and authentication_error error type

[eskombro]

Excellent!

Little idea/suggestion, would it be possible to implement ['*'] for the authorizedKeys on a protected index?

It would mean "It requires a key, but it can be any of the existing keys" as in opposition to "no key required / public". I am not sure if this can have a logic implementation

[MarinPostma]

@eskombro what would [] mean? no one is authorized?

[eskombro]

@MarinPostma, something like:

[] => No key is required (anyone is authorized)
['*'] =. A key is required (any issued key would be authorized)

[ManyTheFish]

Meilisearch Authentication Layer with kong

After several investigations on the possibility to use kong as meilisearch authentication layer, I write here some Pro&Con about it and what should we implement to use kong in production.

Meilisearch needs

As an meilisearch admin I want to be able to create access-keys with custom rights/accesses on meilisearch (843)

The authentication layer needs to support:

• ACL: keys can be associated with R/W permissions on on every sub-routes
• Protected indices: indices access can be restricted to set of keys. By default, an index remain public.
• Restrict usage of some parameter: (Dis)allow some parameters to be overwritten by the user with insufficient privileges

The authentication layer may support:

• Rate limit: limit number of request made by an user
• Size limit: limit the size (mb) of request made by an user
• HTTP referrers: authorize HTTP referrers.

the authentication layer should be transparent for user and shouldn't make meilisearch installation difficult.

Does kong fit the needs?

• ACL: kong is able to manage ACLs, create group, allow groups to access specific routes.
• Protected indices: beacause kong ACL is based on route patterns, protected indices should be managed with ACL
• Restrict usage of some parameter: kong is able to manage query-params/body modifications on any field
• Rate limit: implemented
• Size limit: implemented
• HTTP referrers: not for a specific user but for a specific route

BUT:

It's not so easy to make it completely transparent for user in-case of self installed meilisearch:

• to be fully functional kong needs an SQL database (Postgres)
• some actions need to call several kong routes, so we need to have a "kong-manager" micro service to agregates routes

query-params modifications works well, but only add, rename and replace actions are allowed, that's make facetsFilter impossible to manage without some modifications

About assuming to have a more complex architecture for authentication

First of all, here we will assume that user will install by himself kong.

We can provide:

• a script or a kong.yaml file to initialize meilisearch service in kong
• some tips/docs to configurate kong (add users, filter on query-params...)

this permit to user to choose there authentication layer and have 1 authentication layer for all services, and in an other way it's easy to setup

for meilisearch we don't have to provide and maintain an authentication layer which would be
different between a self-hosted and a SaaS version.

Pro&Con

Pro

• all needed features are allready implemented
• no maintainance for us on this part
• lot of plugins, lot of mainstream features are allready integrated
• work well for several type of architecture

Con

• really difficult to package it transparently in one service
• we need to change some meilisearch query-params
• needs to have more than 1 service to have an authentication layer (at least: meilisearch + kong)

[tpayet]

Thank you for this review :) I have a three questions

1- What can we do if Kong have a state (postgres) and what can't we do?

2 - How the query params change might impact the developer using MeiliSearch?

3 - How much longer is the request when adding the Kong layer between the browser and the MeiliSearch server?
As a reminder, we are trying to answer in less than 50 ms of latency to give the impression of instantanneity. If we add an http Layer like kong (let's say on the same local network where MeiliSearch service is), what is the difference in terms of speed?

[ManyTheFish]

Hello @tpayet thanks for your questions 👍

1- What can we do if Kong have a state (postgres) and what can't we do?

you have 2 ways to configure kong:

1. with a postgres where kong will store every configuration, users, ACL groups, used plugins... This allow admin to add new user, new services via kong API.
2. with a config file which store every configuration, users, ACL groups, used plugins... But this is completely constant and if you want to add a new user, a new route... you will have to change the config file and reload kong. Moreover you can't have a kong cluster without sqldb.

note: the config-file approach could fit small use cases

2- How the query params change might impact the developer using MeiliSearch?

the main change is for array-query-params:
instead of doing ?arrayOfNames="[many, kero, tpayet]" user will do ?name=many&name=kero&name=tpayet (issue only concern GET request)

note: see more here

3 - How much longer is the request when adding the Kong layer between the browser and the MeiliSearch server?
As a reminder, we are trying to answer in less than 50 ms of latency to give the impression of instantanneity. If we add an http Layer like kong (let's say on the same local network where MeiliSearch service is), what is the difference in terms of speed?

I didn't made any benchmark, but on kong website it's written that compute time is under 1ms, so if both services are in the same data-center, kong shouldn't be the bottle neck.

[bidoubiwa]

It has to be postgres or could it be sqlite ?
Do you want to ship MeiliSearch with a postgreSQL db or do you want users to provide a postgreSQL database ?

[eskombro]

really difficult to package it transparently in one service
needs to have more than 1 service to have an authentication layer (at least: meilisearch + kong)

MeiliSearch + Kong + probably Postgres -> I think this really introduces a lot of complexity installing and configuring MeiliSearch. It would clearly fit all the needs we have, but it is maybe too hard on the simplicity we seek for MS install and launch :(

[ManyTheFish]

@bidoubiwa, sqlite is not compatible with kong, probably because sqlite make multithreding impossible.

@eskombro I don't really know, it depends of which use case need to have a complete authentication layer and is it correlated to big architectures or not.
I would be really interested to know in which kind of architectures meilisearch is used.

[qdequele]

@eskombro It's perhaps something we can handle in a new repo meilisearch-app, an all in one docker image/docker-compose repository with all the tools a user need to run MeiliSearch well. Like the core + the future dashboard + kong for auth + timescale for analytics + postgres (for dashboard/kong/timescale).

[deshetti]

We use Ory stack in our applications for auth. https://www.ory.sh/
They are developed in Go, pretty lightweight and performant. Worth looking into if you guys haven't already: https://www.ory.sh/docs/ecosystem/projects

[LorenzBischof]

It would also be awesome if Meilisearch could use an existing SSO setup (e.g. Keycloak). This would make it easier to integrate in enterprise environments.

[raphaeltm]

+1 on Keycloak SSO. We've been using it on a couple projects, and that would be practical.

I don't know if this is the best issue to flag this in, but since it's on the topic of authentication and keys, and since Meilisearch is inspired by Algolia... we would be very interested in a feature similar to Algolia's Secured API Keys. Right now the architecture we're considering involves proxying calls to Meilisearch through a server to filter based on a user ID, but Algolia's system which basically adds a filtering parameter to a specific key strikes me as a really elegant way to restrict access to certain data.

[cabello]

I like this issue and suggested solutions, I am working on a feature that would have private search functionality, each index would have a few thousand documents and I was hoping to be able to restrict index search to only people that should have access to search that index.

Example:

• Live streaming event A has 3000 attendees, some people should be able to search that index.
• Live streaming event B has another set of 5000 attendees, they should not be able to search for people in event A.

[ThisIsMyFavouriteJar]

Hi @cabello, that would be great. In your example you mention 2 livestreams, any idea how this would perform if there where e.g. 1000 livestreams, with 500-5000 attendees each, and many attendees being scattered over multiple livestreams simultaneously?

In other words, how would this perform if the key array gets bigger, with e.g. > 1000 keys per record?

[cabello]

I am using Algolia right now, this is how I structured the solution:

• single index production_users.
• each user record has _tags that include the organization canonical ID (single entry), the upcoming events canonical IDs (usually a few entries).
• when an organizer (they can search for everyone) joins an event, I generate a search key where tags in [organizationId, eventId]

That has been working reasonably well for me.

I know this might not answer your question, it really depends on the data model of the search provider in order to come up with a solution.

What was important for me was:

• Prevent people from searching for everyone (scanning, scraping).
• Prevent leaking people's emails or any other field (Algolia has an option to allow a field to be searchable but not retrievable)
• Guarantee separation between organizations (you shouldn't be able to find users from other organizations).
• Guarantee separation between events (you shouldn't be able to find people that have not registered for that event).

[curquiza]

Hello @cabello! Thanks for your feedback! It helps a lot!

FYI @gmourier

[gmourier]

Hi everyone 👋!

We're dusting off this conversation to share with you the proto-solutions we've come up with, we'd love to hear your feedback on it.

API Keys
Tenant Tokens

[gmourier]

Hey everyone 👋

I'm locking that discussion since an auth layer was released a while ago.

📚https://docs.meilisearch.com/learn/security/master_api_keys.html#master-key-and-api-keys
📚https://docs.meilisearch.com/learn/security/tenant_tokens.html#multitenancy-and-tenant-tokens

Please open a new discussion if needed so we can start fresh on a specific need.

Thank you 🙇‍♂️