RedisMemory

[slorello89]

Motivation and Context (Why the change? What's the scenario?)

Implements #199

High level description (Approach, Design)

Adds Redis Vector Search as an IMemoryDb.

Couple of high-level notes:

1. Tags in Redis need to be pre-defined if you want to filter on them. Also the way the Redis stores tags in a hash means you use a separator character to mark individual tags e.g. 'topics': ['tech', 'microsoft', 'redis'] would be stored in Redis as HSET key topics tech,microsoft,redis then when you search on the tag e.g. @topics:{redis} it will match. What this means from the user's perspective if that if they want to use tags they will need to predefined it before they create the index. I've added a bit to the dependency injection to note this. I also added as a blurb in the README (don't know if there's anywhere else to add those notes to socialize it better)
2. Redis Vector Search uses outputs a cosinal distance rather than cosinal relevance (hence the weird 1 - distance in the post-filter)
3. WRT compatibility - I wrote this against the latest Redis Search and Query library, so the items that are listed in the README (Azure Cache for Redis Enterprise tier, Redis Enterprise, Redis Cloud, Redis Stack).

I also added a new test project in line with the other console test projects I saw in here. There's really not too much to test at a unit level here so idk if there's anything else you want on that end.

[WeihanLi]

Maybe we could just use Tags[tag.Key] = tag.Value

[dluc]

why is it a dictionary rather than a list of tag names?

[slorello89]

@WeihanLi - good point not sure why I didn't do that in the first place

@dluc - due to the Semantics of a tag field in Redis. Tag fields are separated by a separator character (default is a comma)

Hence you could have blog.tag tech,ai,redis,azure and match on tech, ai, redis, or azure. The dictionary is to give the user the flexibility to use fields with those tag separators in them. Might be worth adding a constructor with a list that just defaults everything to a comma.

[dluc]

A couple of thoughts:

• I would consider having a prefix for DBs used to store an index, so that DeleteIndex can not delete Redis DBs used for other data (see the same logic in Postgres connector)
• can we normalize DB names similarly to Postgres or Azure AI Search, e.g. restrict the chars to [a-z0-9] + "dash", automatically converting some other common chars? (see Postgres and Azure AI Search connectors)

[dluc]

I just added this folder: https://github.com/microsoft/kernel-memory/tree/main/extensions/Redis
Could you move all the code there? (see also comment in the initial GH issue). Thanks! :-)

[slorello89]

Hi @dluc - FYI:

I just added this folder: https://github.com/microsoft/kernel-memory/tree/main/extensions/Redis
Could you move all the code there? (see also comment in the initial GH issue). Thanks! :-)

I've moved everything over to the new directory structure/project

I would consider having a prefix for DBs used to store an index, so that DeleteIndex can not delete Redis DBs used for other data (see the same logic in Postgres connector)

Done, I changed RedisConfig -> RedisIndexConfig and added AppPrefix to it (defaults to km). That AppPrefix is used as the prefix for whatever indexes you add to Redis.

can we normalize DB names similarly to Postgres or Azure AI Search, e.g. restrict the chars to [a-z0-9] + "dash", automatically converting some other common chars? (see Postgres and Azure AI Search connectors)

I did this, but I would at least consider moving this logic into a utility class in Abstractions and having all the extensions leverage the same normalizer. FWIW all strings in Redis are parameterized in RESP(Redis Serialization Protocol) and are binary safe, so you could call your indexes/key-names whatever.

[dluc]

Merging - will continue on a separate PR