Skip to content

Commit ca57ae3

Browse files
committed
Improve test points: flaky and verbose
* Make test point less brittle `tazureChat/generateWithStructuredOutput` has somewhat frequent spurious failures of this type: ``` ================================================================================ Verification failed in tazureChat/generateWithStructuredOutput. --------------------- Framework Diagnostic: --------------------- OrConstraint failed. --> + [First Condition]: | OrConstraint failed. | --> + [First Condition]: | | IsEqualTo failed. | | --> StringComparator failed. | | --> The strings are not equal. | | | | Actual Value: | | "western honey bee" | | Expected Value: | | "honeybee" | --> OR | + [Second Condition]: | | IsEqualTo failed. | | --> StringComparator failed. | | --> The strings are not equal. | | | | Actual Value: | | "western honey bee" | | Expected Value: | | "honey bee" | -+--------------------- --> OR + [Second Condition]: | IsEqualTo failed. | --> StringComparator failed. | --> The strings are not equal. | | Actual Value: | "western honey bee" | Expected Value: | "bee" -+--------------------- ``` Rather than trying to add more and more valid alternatives, it is probably better to relax the requirements a bit. * Make test less noisy `texampleTests/testAnalyzeTextDataUsingParallelFunctionCallwithOllama` was lacking the `evalc` against noise cluttering the output.
1 parent 08897d1 commit ca57ae3

File tree

2 files changed

+3
-4
lines changed

2 files changed

+3
-4
lines changed

tests/hstructuredOutput.m

+2-3
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,13 @@
1010
methods(Test)
1111
% Test methods
1212
function generateWithStructuredOutput(testCase)
13-
import matlab.unittest.constraints.IsEqualTo
13+
import matlab.unittest.constraints.ContainsSubstring
1414
import matlab.unittest.constraints.StartsWithSubstring
1515
res = generate(testCase.structuredModel,"Which animal produces honey?",...
1616
ResponseFormat = struct(commonName = "dog", scientificName = "Canis familiaris"));
1717
testCase.assertClass(res,"struct");
1818
testCase.verifySize(fieldnames(res),[2,1]);
19-
testCase.verifyThat(lower(res.commonName), IsEqualTo("honeybee") | IsEqualTo("honey bee") | ...
20-
IsEqualTo("bee"));
19+
testCase.verifyThat(lower(res.commonName), ContainsSubstring("bee"));
2120
testCase.verifyThat(res.scientificName, StartsWithSubstring("Apis"));
2221
end
2322

tests/texampleTests.m

+1-1
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ function testAnalyzeTextDataUsingParallelFunctionCallwithChatGPT(testCase)
7777

7878
function testAnalyzeTextDataUsingParallelFunctionCallwithOllama(testCase)
7979
testCase.startCapture("AnalyzeTextDataUsingParallelFunctionCallwithOllama");
80-
AnalyzeTextDataUsingParallelFunctionCallwithOllama;
80+
evalc("AnalyzeTextDataUsingParallelFunctionCallwithOllama");
8181
end
8282

8383
function testCreateSimpleChatBot(testCase,ChatBotExample)

0 commit comments

Comments
 (0)