@@ -10,13 +10,146 @@ See https://swift.org/LICENSE.txt for license information
1010See https://swift.org/CONTRIBUTORS.txt for Swift project authors
1111-->
1212
13- Highlight known issues when running tests.
13+ Mark issues as known when running tests.
1414
1515## Overview
1616
17- The testing library provides a function, ` withKnownIssue() ` , that you
18- use to mark issues as known. Use this function to inform the testing library
19- at runtime not to mark the test as failing when issues occur.
17+ The testing library provides several functions named ` withKnownIssue() ` that
18+ you can use to mark issues as known. Use them to inform the testing library that
19+ a test should not be marked as failing if only known issues are recorded.
20+
21+ ### Mark an expectation failure as known
22+
23+ Consider a test function with a single expectation:
24+
25+ ``` swift
26+ @Test func grillHeating () throws {
27+ var foodTruck = FoodTruck ()
28+ try foodTruck.startGrill ()
29+ #expect (foodTruck.grill .isHeating ) // ❌ Expectation failed
30+ }
31+ ```
32+
33+ If the value of the ` isHeating ` property is ` false ` , ` #expect ` will record an
34+ issue. If you cannot fix the underlying problem, you can surround the failing
35+ code in a closure passed to ` withKnownIssue() ` :
36+
37+ ``` swift
38+ @Test func grillHeating () throws {
39+ var foodTruck = FoodTruck ()
40+ try foodTruck.startGrill ()
41+ withKnownIssue (" Propane tank is empty"
42+ #expect (foodTruck.grill .isHeating ) // Known issue
43+ }
44+ }
45+ ```
46+
47+ The issue recorded by ` #expect ` will then be considered "known" and the test
48+ will not be marked as a failure. You may include an optional comment to explain
49+ the problem or provide context.
50+
51+ ### Mark a thrown error as known
52+
53+ If an ` Error ` is caught by the closure passed to ` withKnownIssue() ` , the issue
54+ representing that caught error will be marked as known. Continuing the previous
55+ example, suppose the problem is that the ` startGrill() ` function is throwing an
56+ error. You can apply ` withKnownIssue() ` to this situation as well:
57+
58+ ``` swift
59+ @Test func grillHeating () {
60+ var foodTruck = FoodTruck ()
61+ withKnownIssue {
62+ try foodTruck.startGrill () // Known issue
63+ #expect (foodTruck.grill .isHeating )
64+ }
65+ }
66+ ```
67+
68+ Because all errors thrown from the closure are caught and interpreted as known
69+ issues, the ` withKnownIssue() ` function is not throwing. Consequently, any
70+ subsequent code which depends on the throwing call having succeeded (such as the
71+ ` #expect ` after ` startGrill() ` ) must be included in the closure to avoid
72+ additional issues.
73+
74+ - Note: ` withKnownIssue() ` is recommended instead of ` #expect(throws:) ` for any
75+ error which is considered a known issue so that the test status and results
76+ will reflect the situation more accurately.
77+
78+ ### Match a specific issue
79+
80+ By default, ` withKnownIssue() ` considers all issues recorded while invoking the
81+ body closure known. If multiple issues may be recorded, you can pass a trailing
82+ closure labeled ` matching: ` which will be called once for each recorded issue
83+ to determine whether it should be treated as known:
84+
85+ ``` swift
86+ @Test func batteryLevel () throws {
87+ var foodTruck = FoodTruck ()
88+ try withKnownIssue {
89+ let batteryLevel = try #require (foodTruck.batteryLevel ) // Known
90+ #expect (batteryLevel >= 0.8 ) // Not considered known
91+ } matching : { issue in
92+ guard case .expectationFailed (let expectation) = issue.kind else {
93+ return false
94+ }
95+ return expectation.isRequired
96+ }
97+ }
98+ ```
99+
100+ ### Resolve a known issue
101+
102+ If there are no issues recorded while calling ` function ` , ` withKnownIssue() `
103+ will record a distinct issue about the lack of any issues having been recorded.
104+ This notifies you that the underlying problem may have been resolved so that you
105+ can investigate and consider removing ` withKnownIssue() ` if it's no longer
106+ necessary.
107+
108+ ### Handle a nondeterministic failure
109+
110+ If ` withKnownIssue() ` sometimes succeeds but other times records an issue
111+ indicating there were no known issues, this may indicate a nondeterministic
112+ failure or a "flaky" test.
113+
114+ The first step in resolving a nondeterministic test failure is to analyze the
115+ code being tested and determine the source of the unpredictable behavior. If
116+ you discover a bug such as a race condition, the ideal resolution is to fix
117+ the underlying problem so that the code always behaves consistently even if
118+ it continues to exhibit the known issue.
119+
120+ If the underlying problem only occurs in certain circumstances, consider
121+ including a precondition. For example, if the grill only fails to heat when
122+ there's no propane, you can pass a trailing closure labeled ` when: ` which
123+ determines whether issues recorded in the body closure should be considered
124+ known:
125+
126+ ``` swift
127+ @Test func grillHeating () throws {
128+ var foodTruck = FoodTruck ()
129+ try foodTruck.startGrill ()
130+ withKnownIssue {
131+ // Only considered known when hasPropane == false
132+ #expect (foodTruck.grill .isHeating )
133+ } when : {
134+ ! hasPropane
135+ }
136+ }
137+ ```
138+
139+ If the underlying problem is unpredictable and fails at random, you can pass
140+ ` isIntermittent: true ` to let the testing library know that it will not always
141+ occur. Then, the testing library will not record an issue when zero known issues
142+ are recorded:
143+
144+ ``` swift
145+ @Test func grillHeating () throws {
146+ var foodTruck = FoodTruck ()
147+ try foodTruck.startGrill ()
148+ withKnownIssue (isIntermittent : true ) {
149+ #expect (foodTruck.grill .isHeating )
150+ }
151+ }
152+ ```
20153
21154## Topics
22155
0 commit comments