Key generation is not deterministic

[curquiza]

Following this discussion: meilisearch/meilisearch#2261
(I still cannot transfer the discussion because of the GitHub bug...)

Is it possible @geoffatsource, can you detail what is your usecase? It would definitely help the product team 🙂

[geoffatsource]

We have GitHub Actions that provision and deploy new Meilisearch instances behind load balancers.

These instances have fresh installs if Meilisearch with the version of our choice.

We want to be able to deploy these at will, bring them online once indexing is complete.

Unfortunately this is no longer possible because the API key values are different for each new instance.

API keys used to be computed from the master key, but now they are completely random. This is problematic for the reason above and it complicates our front-end deployment

[bidoubiwa]

We could introduce a unique name field on an API Key resource and use it in combination with the master key to generate the value of the key field.

This might raise a security issue. Imagine I create a key named super_admin, it creates an API key based on the masterKey. Later I remove this key because it was leaked. Now, even later, I recreate a key named super_admin because why not/ I forgot / someone else created it. If it has the same name, the generated key will be the same as the previous one which is still leaked

we will have a name to retrieve API Key

🔥

I propose to keep the tenant token as they are and to use the full apiKey field to fill them as we do today

Just to be sure I understood correctly! Do you mean to keep the apiKeyPrefix instead of a name field in the payload of the token?

[geoffatsource]

I would think the simplest method would be to give admins the ability to specify the key value when creating API keys. That way we can keep it in our vault and apply it at provision time.

That said, the previous public/private keys seemed to do an adequate job in creating deterministic keys while maintaining security.

[gmourier]

Indeed, the point that @bidoubiwa brings up is important. Relying on a name field to generate the key is not viable.

As some comments point out a solution would be to open the key field for editing.

We could make it non-mandatory: If filled it's take the given value, if not it generate a random key. I'm afraid it's complicated to explain but it would allow not to force people to specify a key if they don't need it.

The advantage of using the master-key to generate its value is that it is possible to revoke all keys by changing the master-key, which is also a point that can be useful if the master-key has leaked. In the case of handwritten keys, this will not be possible anymore and the keys will have to be revoked by hand. We will probably want to add a revoke-all and a revoke key endpoints at some point to ease that.

@geoffatsource We could restore the fact that the default keys are generated in cascade from the master-key as before. The question is what happens if the master-key changes and the keys are used by other clients already? I think they should be decoupled from the master-key once generated. Thus said if the master key is modified after being set, the keys do not change.

[geoffatsource]

I am fine with API keys being revoked if the master key changes. That made perfect sense before and seems to be a valuable security feature.

[gmourier]

Thanks for the answer @geoffatsource

[fraank]

I stumbled upon the same missing feature a month ago, while setup a somehow deterministic k8s cluster. I had a discussion with @gmourier on slack what's missing, and he requested me to sum up it here.

IMO two simple changes should make key handling more easier in bigger setups:

1. Allow the creation of an API Key from the endpoint POST /keys with a value for the key. So there could be user scripts executed to automatically add or sync needed keys.
2. Add an ENV-Variable / startup param MEILI_DEFAULT_KEY / --default-key (analog to MEILI_MASTER_KEY or --master-key) which would overwrite the key creation process of the default key.

Both changes could cause security issues used carelessly, but they are optional. So default setup remain unchanged.

[gmourier]

Thanks @fraank

1. Allow the creation of an API Key from the endpoint POST /keys with a value for the key. So there could be user scripts executed to automatically add or sync needed keys.

Yes. It seems to be favored instead of 2. IMO. It keeps the handling of key at one place, plus we have 2 Default Keys: Default Search API Key and the Default Admin API Key. To go further than 2, a config file that allows you to generate keys from it, with descriptions, actions, etc could be more consistent in the long term and best suited. It would be like using the endpoints keys.

[brunoocasali]

If we proceed to enable the user to create its own key, can we ensure some security constraints, like having at least X characters, containing lower and upper case values (all the good practices to create good passwords)?

I know this is something people will care about, but you know, just in case...

[bidoubiwa]

Not sure about lower and upper case as the token could be a long number. But having a length limit is a very nice idea

[gmourier]

Hello everyone 👋

Some news on this subject!

We evaluated different solutions, among them our constraint was to keep something simple enough to use that would work for both people who need determinism and those who don't. We also wanted to avoid as much as possible having two completely different ways of generating keys (i.e. to have keys hashed by the master key and another one completely independent) for the clarity and simplicity of the codebase.

Generating the value of key with a hash of the master key allows having interesting features that we would like to keep.

• Being able to regenerate all the keys when the master-key changes.
• Not having to pass in clear text the key field of API Keys in dumps and snapshots.

Here is the solution that seems to check all the boxes:

Introduce a unique customizable uid field at creation

The combination of this uid field and the master-key gives the value of the key field of an API key, so the keys can now
be deterministic.

If this field is not specified at creation, the engine takes care of generating it, so users who don't need it don't have to define it.

This uid field is auto-generated for the two default keys. If you need to create deterministic keys for the two default keys, you will have to create them on your own.

e.g. Having a master-key defined to ThQ7sTbvQKX4JwLsBRwJcZSn24gk3Ju6

POST /keys

{
    "uid": "0330c498-5fa0-4c30-94c3-e46433df44a6",
    "description": "My Search Key",
    "indexes": "*",
    "actions": "search",
    "expiresAt": "2023-01-01T00:00:00Z"
}

results in

{
    "uid": "0330c498-5fa0-4c30-94c3-e46433df44a6",
    "key": "c86da00c26c3645cd9ddde6881befbdb92bd680f0a82ccf0ff2d5599be5ce327" //determined with a hash combination of `uid` and the master key
    "description": "My Search Key",
    "indexes": "*",
    "actions": "search",
    "expiresAt": "2023-01-01T00:00:00Z",
    "createAt": "2022-04-26T09:42:00Z",
    "updatedAt": "2022-04-26T09:42:00Z"
}

If uid had not been given here, the engine would have generated one automatically.

uid expected format

The uid field expects a uuidv4 value. We chose this type of value to prevent, as much as we can, users from re-entering a previously leaked or deleted key, which might have been the case if we had a field like name to determine the value of key instead of uid.

The API returns a synchronous error if the given uid field is not a valid uuidv4 or if a key already exists with this uid.

Benefits

• This keeps the fact that all keys are automatically changed if the master key is modified, moreover in a deterministic way.
• By keeping the generation of the key field through a hash with the master-key, e.g. hash(uid, master-key), the complete value of key does not pass through dumps or snapshots which is a plus for security.
• We minimize the possibility of re-entering a previously leaked key if the user chooses to define the uid field using the uuidv4 format instead of a business value field like name.

---

Do you see anything we would have missed with this solution? Does it meet your need to be able to generate deterministic keys?

Clearly, having a way to distribute keys between instances would be more convenient and is something that could probably come in the future when we tackle Meilisearch clusterization.

[bidoubiwa]

I don't think these options are necessary: GET /keys/:salt or GET /keys/:secret neither is the actual GET /keys/:key.

The only ones that should exist for me are:

• GET /keys/:name
• GET /keys/:uid (with uid is 1, 2, 3, ...)

uid should be used to access a unique resource, not as a tool for another task. In my opinion of course but this may not be accurate.

[MarinPostma]

@bidoubiwa in this case uid is unique so it works. We should keep name for potentially allowing aliasing in the future.

We are using sha2 now for key derivation but we should use https://en.wikipedia.org/wiki/PBKDF2 instead.

[MarinPostma]

also we can't rely on incremental ids, unless we have some some of synchronization between instances

[gmourier]

Yes, we could also add a name field to help the retrieval of a key.

For me, uid replaces the way to GET a specific key in a Restful manner. e.g. GET /keys/:uid to be consistent with other endpoints, while we can have this additional way to retrieve a key by its key using a query parameter. GET /keys?key=:key.

This can be useful in case a developer wants to tell an admin that his key is not working without knowing its uid, usually it is the values of the key fields that will be shared once generated.

uid should be used to access a unique resource, not as a tool for another task.

Yes, and it works with this proposal! It's just that the key field is determined from uid and master-key. It permits to not complexifying the API for users who will never need determinism. It will be totally transparent for them.

This is a completely different topic though.

[brunoocasali]

(before reading it: don't bother with the names but with the overall response, objects/arrays)

When I read GET /keys/:uid I expect a response like this (pure object in the root of the response):

// GET /keys/123
{
   "id": 123,
   "key": "xxabcyzx" 
}

Because I'm querying the keys collection directly and asking by the resource with "id" = "123", if the server has a value will be returned directly to the root of the json response. If the server does not find it, a 404 will be returned.

Now if I read this GET /keys?key=xxabcyzx I expect this response (array of objects/object w/ results oriented list):

// GET /keys?key=xxabcyzx
[
  {
   "id": 123,
   "key": "xxabcyzx" 
  },
  ...
]

Because I was applying just a filter in the keys resources, is the same if I had other filters like status=revoked|active or something else.

Some solutions that can be applied:

Suppose I want to find this resource:

key: "c86da00c26c3645cd9ddde6881befbdb92bd680f0a82ccf0ff2d5599be5ce327" 
uid: "3eeddf57-cee1-4b02-b8f3-d73c9a584"

1. A sub route with the name of the field:

GET /keys/uid/:uid GET /keys/uid/3eeddf57-cee1-4b02-b8f3-d73c9a584 => 200/object
GET /keys/key/:key GET /keys/key/c86da00c26c3645cd9ddde6881befbdb92bd680f0a82ccf0ff2d5599be5ce327 => 200/object

2. The same route but with different behaviors (I think this could work since a uid and a key are not similar):

GET /keys/:data GET /keys/3eeddf57-cee1-4b02-b8f3-d73c9a584 => 200/object
GET /keys/:data GET /keys/c86da00c26c3645cd9ddde6881befbdb92bd680f0a82ccf0ff2d5599be5ce327 => 200/object

WDYT folks?

[geoffatsource]

Just a quick follow-up to find out if this is on the near-term roadmap :)

[curquiza]

For the moment, it's planned to be released in v0.28.0, so in July! First RC in June 😄

[gmourier]

Hey @geoffatsource 👋

Did you have time to test the RC? if yes, did the provided solution solve your problem?

Thank you!

[curquiza]

Hello everyone here!
The first RC (release candidate) of Meilisearch v0.28.0 is out, meaning the deterministic key feature is now testable! 🔥 🔥 It is now possible to create a deterministic key value by specifying a uid field at creation. uid value must be a UUID v4. If omitted, Meilisearch generates it.

You can use the v0.28.0rc0 docker image or download the available binaries in the release assets.
The official v0.28.0 release is planned for 11th July!

Any feedback is more than welcome and would definitely help us improve our product 🙏 ❤️

[gmourier]

If you need to know the value of key without hitting the GET /keys endpoint, the algorithm used to generate the key field is this one. It may be interesting to mention it here if needed for a specific use case.

HMAC(secret: SHA-256(:master-key), data: :uid)

[fraank]

@gmourier Awesome. That was the question that popped up yesterday, but I didn't make it to look into the source yet. So a quick line should do the trick:

echo -n "UID" | openssl dgst -sha256 -hmac "MASTER-KEY"

[gmourier]

Hey @fraank 👋

I wanted to know if you had time to test this and if the solution in v0.28 solved your problem.

Thank you!

[moweerkat]

Hello @curquiza @gmourier , thanks a lot for the effort. I needed this for docker automated deployments

However echo -n "f42363a8-96cb-4ad6-ad5b-8db9490f0fb4" | openssl dgst -sha256 -hmac "masterkey" gives 253bed7b1f5042ca9a9e3e9c91ede17e4a9e3489671dadbcd18b5821b53f1bb6

and the create key endpoint with the payload

meilisearch_client.create_key(
            options={
                "uid": "f42363a8-96cb-4ad6-ad5b-8db9490f0fb4",
                "description": "API Search Key",
                "indexes": ["*"],
                "actions": ["search"],
                "expiresAt": None,
            }
        )

generates

{'name': None, 'description': 'API Search Key', 'key': '7Gw1JNKOQdfq-MCTlLxwGfJY9xg6q6Wqq0inrL_omo0', 'uid': 'f42363a8-96cb-4ad6-ad5b-8db9490f0fb4', 'actions': ['search'], 'indexes': ['*'], 'expiresAt': None, 'createdAt': '2022-07-04T16:27:16.656318003Z', 'updatedAt': '2022-07-04T16:27:16.656318003Z'},

What am I missing here ? I'm using v0.28.0rc1

[curquiza]

Hello @glamothe9
Sorry for the inconvenience
It should be fixed with v0.28.0rc3 we've just released 😊

[gmourier]

Hello everyone 👋

Just to say that we released version v0.28 today, feel free to give us your feedback!

This discussion will be locked in the next few days. We encourage you to create a new discussion if needed!

Thanks