Skip to content

Commit a6da946

Browse files
committed
test: remove mistral from smoke tests
The smoke tests for `mistral-large-2402` are failing because its responses for certain test cases have changed since the initial golden files were generated. After a bit of investigation, it looks like this model was deprecated in November 2024 and retired in June 2025. Also, from what I can tell, contemporary Mistral models like `mistral-large-2411` and `mistral-medium-2505` don't actually make the expected tool calls required to pass the GPTScript's smoke tests, so swapping out the model is not feasible at this time. In light of these findings, remove mistral from the set of smoke tested model providers. Signed-off-by: Nick Hale <[email protected]>
1 parent 7a49337 commit a6da946

File tree

3 files changed

+0
-1307
lines changed

3 files changed

+0
-1307
lines changed

.github/workflows/smoke.yaml

Lines changed: 0 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -147,34 +147,3 @@ jobs:
147147
echo "Running smoke test for model claude-3-5-sonnet-20240620"
148148
export PATH="$(pwd)/bin:${PATH}"
149149
make smoke
150-
151-
mistral-large-2402:
152-
needs: check-label
153-
if: ${{ needs.check-label.outputs.run_smoke_tests == 'true' }}
154-
runs-on: ubuntu-22.04
155-
steps:
156-
- name: Checkout base repository
157-
uses: actions/checkout@v4
158-
with:
159-
fetch-depth: 1
160-
- name: Checkout PR code if running for a PR
161-
if: ${{ github.event_name == 'pull_request_target' }}
162-
uses: actions/checkout@v4
163-
with:
164-
fetch-depth: 1
165-
repository: ${{ github.event.pull_request.head.repo.full_name }}
166-
ref: ${{ github.event.pull_request.head.ref }}
167-
- uses: actions/setup-go@v5
168-
with:
169-
cache: false
170-
go-version: "1.21"
171-
- env:
172-
OPENAI_API_KEY: ${{ secrets.SMOKE_OPENAI_API_KEY }}
173-
GPTSCRIPT_DEFAULT_MODEL: mistral-large-2402 from github.com/gptscript-ai/mistral-laplateforme-provider
174-
MISTRAL_API_KEY: ${{ secrets.SMOKE_MISTRAL_API_KEY }}
175-
GPTSCRIPT_CREDENTIAL_OVERRIDE: "github.com/gptscript-ai/mistral-laplateforme-provider/credential:MISTRAL_API_KEY"
176-
name: Run smoke test for mistral-large-2402
177-
run: |
178-
echo "Running smoke test for model mistral-large-2402"
179-
export PATH="$(pwd)/bin:${PATH}"
180-
make smoke

0 commit comments

Comments
 (0)