Switch to gpt-41-mini with evaluations

Author: pamelafox

README.md

@@ -51,7 +51,7 @@ This template, the application code and configuration it contains, has been buil
 
 [📺 Watch a video overview of the app.](https://youtu.be/3acB0OWmLvM)
 
-This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4o-mini), and Azure AI Search for data indexing and retrieval.
+This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4.1-mini), and Azure AI Search for data indexing and retrieval.
 
 The repo includes sample data so it's ready to try end to end. In this sample application we use a fictitious company called Contoso Electronics, and the experience allows its employees to ask questions about the benefits, internal policies, as well as job descriptions and roles.
 

docs/agentic_retrieval.md

@@ -24,7 +24,7 @@ See the agentic retrieval documentation.
 
 2. **(Optional) Set the agentic retrieval model**
 
-   You can configure which model agentic retrieval uses. By default, gpt-4o-mini is used
+   You can configure which model agentic retrieval uses. By default, gpt-4o-mini is used.
 
    For gpt-4o:
 

docs/deploy_existing.md

@@ -26,8 +26,8 @@ You should set these values before running `azd up`. Once you've set them, retur
 1. Run `azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}`
 1. Run `azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}`
 1. Run `azd env set AZURE_OPENAI_LOCATION {Location of existing OpenAI service}`
-1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4o-mini'.
-1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4o-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4.1-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4.1-mini'.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION {Version string for existing chat deployment}`. Only needed if your chat deployment model version is not the default '2024-07-18'. You definitely need to change this if you changed the model.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU {Name of SKU for existing chat deployment}`. Only needed if your chat deployment SKU is not the default 'Standard', like if it is 'GlobalStandard' instead.
 1. Run `azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing embedding deployment}`. Only needed if your embeddings deployment is not the default 'embedding'.

docs/deploy_features.md

@@ -40,24 +40,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the GPT model to a different [available model](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate model name.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
     ```bash
-    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4.1-mini
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o-mini
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash
@@ -66,24 +72,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the Azure OpenAI model version from the [available versions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate version string.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
-    ```bash
-    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    ```
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2025-04-14
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-05-13
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-07-18
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash

docs/gpt4v.md

@@ -23,7 +23,7 @@ For more details on how this feature works, read [this blog post](https://techco
 * The ability to deploy a gpt-4o model in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o deployment from your Azure OpenAI deployments page.
 * Ensure that you can deploy the Azure OpenAI resource group in [a region and deployment SKU where all required components are available](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability):
   * Azure OpenAI models
-    * gpt-4o-mini
+    * gpt-4.1-mini
     * text-embedding-3-large
     * gpt-4o (for vision/evaluation features)
   * [Azure AI Vision](https://learn.microsoft.com/azure/ai-services/computer-vision/)

evals/results/baseline/README.md

@@ -1,6 +1,6 @@
 {
     "evaluation_gpt_model": "gpt-4o",
-    "evaluation_timestamp": 1746818372,
+    "evaluation_timestamp": 1748932460,
     "testdata_path": "/Users/pamelafox/azure-search-openai-demo/evals/ground_truth.jsonl",
     "target_url": "http://localhost:50505/chat",
     "target_parameters": {

evals/results/baseline/eval_results.jsonl

@@ -1,31 +1,31 @@
 {
     "gpt_groundedness": {
-        "pass_count": 43,
-        "pass_rate": 0.86,
-        "mean_rating": 4.5
+        "pass_count": 47,
+        "pass_rate": 0.94,
+        "mean_rating": 4.76
     },
     "gpt_relevance": {
-        "pass_count": 42,
-        "pass_rate": 0.84,
-        "mean_rating": 4.22
+        "pass_count": 47,
+        "pass_rate": 0.94,
+        "mean_rating": 4.42
     },
     "answer_length": {
-        "mean": 919.26,
-        "max": 1647,
-        "min": 193
+        "mean": 829.06,
+        "max": 1380,
+        "min": 275
     },
     "latency": {
-        "mean": 4.46,
-        "max": 15.129978,
-        "min": 2.465542
+        "mean": 2.89,
+        "max": 5.183753,
+        "min": -1.0
     },
     "citations_matched": {
-        "total": 24,
-        "rate": 0.49
+        "total": 26,
+        "rate": 0.52
     },
     "any_citation": {
-        "total": 50,
-        "rate": 1.0
+        "total": 49,
+        "rate": 0.98
     },
     "num_questions": {
         "total": 50

evals/results/baseline/evaluate_parameters.json

@@ -0,0 +1,28 @@
+{
+    "testdata_path": "ground_truth.jsonl",
+    "results_dir": "results/experiment<TIMESTAMP>",
+    "requested_metrics": ["gpt_groundedness", "gpt_relevance", "answer_length", "latency", "citations_matched", "any_citation"],
+    "target_url": "http://localhost:50505/chat",
+    "target_parameters": {
+        "overrides": {
+            "top": 3,
+            "temperature": 0.3,
+            "minimum_reranker_score": 0,
+            "minimum_search_score": 0,
+            "retrieval_mode": "hybrid",
+            "semantic_ranker": true,
+            "semantic_captions": false,
+            "suggest_followup_questions": false,
+            "use_oid_security_filter": false,
+            "use_groups_security_filter": false,
+            "vector_fields": [
+                "embedding"
+            ],
+            "use_gpt4v": false,
+            "gpt4v_input": "textAndImages",
+            "seed": 1
+        }
+    },
+    "target_response_answer_jmespath": "message.content",
+    "target_response_context_jmespath": "context.data_points.text"
+}