[BUG] Script Score Queries Not getting Cached for K-NN painless Script for Exact Search

[navneet1v]

What is the bug?
While trying to do the exact search via Script Score queries as suggested in this documentation: https://opensearch.org/docs/latest/search-plugins/knn/knn-score-script/ what I am observing is even after sending the search request with request_cache=true url param in the request the requests are not getting cached.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Use the provided documentation to create a k-nn index and ingest the data in the index.
2. Now, run the exact search query like, multiple times:

curl --request POST \
  --url 'http://localhost:9200/my-knn-index-1/_search?request_cache=true' \
  --header 'Content-Type: application/json' \
  --data '{
 "query": {
   "script_score": {
     "query": {
       "match_all": {}
     },
     "script": {
       "source": "knn_score",
       "lang": "knn",
       "params": {
         "field": "my_vector2",
         "query_value": [2.0, 3.0, 5.0, 6.0],
         "space_type": "cosinesimil"
       }
     }
   }
 }
}'

3. Now hit the index stats api to see that request are not getting cached in the request_cache object.

curl --request GET \
  --url http://localhost:9200/my-knn-index-1/_stats

What is the expected behavior?
The expected behavior is if the request_cache parameter is passed the request should be cached. I tested this behavior with another painless script query

curl --request POST \
  --url 'http://localhost:9200/my-knn-index-1/_search?request_cache=true' \
  --header 'Content-Type: application/json' \
  --data '{
  "query": {
    "script_score": {
      "query": {
        "match": { "message": "opensearch" }
      },
      "script": {
        "source": "doc['\''price'\''].value / 10 "
      }
    }
  }
}'

and I can see the requests are getting cached.

What is your host/environment?
Happening on all envs.

Do you have any screenshots?
NA

Do you have any additional context?
On doing some deep-dive in the code I can see that the issue request is getting marked as false from this function: https://github.com/opensearch-project/OpenSearch/blob/aca2e9de7daaadee77c33e21ff5215c3f1e8600f/server/src/main/java/org/opensearch/index/query/QueryShardContext.java#L493-L499 and this

https://github.com/opensearch-project/OpenSearch/blob/aca2e9de7daaadee77c33e21ff5215c3f1e8600f/server/src/main/java/org/opensearch/index/query/QueryShardContext.java#L518-L524

[junqiu-lei]

In OS core level script compile, the isResultDeterministic from XScoreScriptFactory(Interface ScriptFactory) need to be true to perform expected query cache, or else it won't perform cache even if set request_cache=true.

However, in existing KNNScoringScriptEngine, we directly use KNNScoreScriptFactory(leafFactory) to new the factory, it implements from ScoreScript.LeafFactory, which doesn't contain method isResultDeterministic, so finally the kNNScriptFactory in OS core just use interface level isResultDeterministic method which by default return false.

I found a example of ExpertScriptPlugin, which factory implemented from ScoreScript.Factory and ScriptFactory interface and make isResultDeterministic method to returntrue to perform the right search cache. The leafFactory is newed by newFactory method.

So as reference from ExpertScriptPlugin on how it implements the script factory, I made the code change to k-NN and tested it works as expected finally. PR #1367 is raised to fix the issue.