Semantic Cache

Semantic cache is a caching mechanism in Portkey that goes beyond fixed string matching. It evaluates the contextual similarity between input prompts and cached requests using cosine similarity. When the similarity exceeds a threshold, Portkey retrieves the response from the cache, improving response retrieval efficiency. It can be enabled by setting the x-portkey-cache header to "semantic" in the requests.