Represent non-encodable test argument values in Test.Case.ID (#1000)

This expands `Test.Case.ID` to represent combinations of arguments which
aren't fully encodable, and uniquely distinguishes test cases associated
with the same parameterized test function which have otherwise identical
IDs.

### Motivation:

It is possible to declare a parameterized test function with a
collection of arguments that includes the same element more than once.
As a trivial example:

```swift
@test(arguments: [1, 1])
func repeatedArg(value: Int) { ... }
```

This can happen in more realistic scenarios if you dynamically construct
a collection of arguments that accidentally or intentionally includes
the same value more than once.

The testing library attempts to form a unique identifier for each
argument passed to a parameterized test. It does so by checking whether
the value conforms to `Encodable`, or one of the other protocols
mentioned in the documentation (see [Run selected test
cases](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/parameterizedtesting#Run-selected-test-cases)).

One problem with the current implementation is that if the value
_doesn't_ conform to one of those known protocols, the testing library
gives up and doesn't produce a unique identifier for that test case at
all. Specifically, in that situation the `argumentIDs` property of
`Test.Case.ID` will have a value of `nil`.

Another problem is that if an argument is passed more than once, the
derived identifiers for each argument's test case will be the same and
the result reporting will be ambiguous at best; or worse, the lack of a
unique identifier could cause an integrated tool to misbehave or crash.
(Current versions of Xcode 16 experience this issue
non-deterministically for projects which have test parallelization
enabled.) To solve this, there needs to be a way to deterministically
distinguish test cases which, from the testing library's perspective,
appear identical—either because they actually are the same value, or
because they encode to the same representation.

### Modifications:

- Add a new `isStable` boolean property to `Test.Case.Argument.ID`
representing whether or not the testing library was able to encode a
stable representation of the argument value it identifies.
- Add a new SPI property named `discriminator` to `Test.Case` which
distinguishes test cases associated with the same test function whose
arguments are identical.
- Add a corresponding property to `Test.Case.ID` with the same name,
`discriminator`.
- Change the `Test.Case.ID.argumentIDs` property to non-Optional, so
that it always has a value even if one or more argument IDs is
non-stable.
- Add a derived boolean property `isStable` to `Test.Case.ID` whose
value is `true` iff all of its argument IDs are stable.
- Add `Hashable` conformance to `Test.Case`, since the reason for
avoiding such conformance has been resolved.
- Add new tests.

### Checklist:

- [x] Code and documentation should follow the style of the [Style
Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md).
- [x] If public symbols are renamed or modified, DocC references should
be updated.

Resolves #995
Resolves rdar://119522099

Author: stmontgomery

Sources/Testing/ABI/Encoded/ABI.EncodedTest.swift

@@ -124,9 +124,13 @@ extension ABI {
     var displayName: String
 
     init(encoding testCase: borrowing Test.Case) {
+      guard let arguments = testCase.arguments else {
+        preconditionFailure("Attempted to initialize an EncodedTestCase encoding a test case which is not parameterized: \(testCase). Please file a bug report at https://github.com/swiftlang/swift-testing/issues/new")
+      }
+
       // TODO: define an encodable form of Test.Case.ID
       id = String(describing: testCase.id)
-      displayName = testCase.arguments.lazy
+      displayName = arguments.lazy
         .map(\.value)
         .map(String.init(describingForTest:))
         .joined(separator: ", ")

Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift

@@ -171,8 +171,14 @@ extension Test.Case {
   /// - Parameters:
   ///   - includeTypeNames: Whether the qualified type name of each argument's
   ///     runtime type should be included. Defaults to `false`.
+  ///
+  /// - Returns: A string containing the arguments of this test case formatted
+  ///   for presentation, or an empty string if this test cases is
+  ///   non-parameterized.
   fileprivate func labeledArguments(includingQualifiedTypeNames includeTypeNames: Bool = false) -> String {
-    arguments.lazy
+    guard let arguments else { return "" }
+
+    return arguments.lazy
       .map { argument in
         let valueDescription = String(describingForTest: argument.value)
 
@@ -494,14 +500,14 @@ extension Event.HumanReadableOutputRecorder {
       return result
 
     case .testCaseStarted:
-      guard let testCase = eventContext.testCase, testCase.isParameterized else {
+      guard let testCase = eventContext.testCase, testCase.isParameterized, let arguments = testCase.arguments else {
         break
       }
 
       return [
         Message(
           symbol: .default,
-          stringValue: "Passing \(testCase.arguments.count.counting("argument")) \(testCase.labeledArguments(includingQualifiedTypeNames: verbosity > 0)) to \(testName)"
+          stringValue: "Passing \(arguments.count.counting("argument")) \(testCase.labeledArguments(includingQualifiedTypeNames: verbosity > 0)) to \(testName)"
         )
       ]
 

Sources/Testing/Parameterization/CustomTestArgumentEncodable.swift

@@ -44,18 +44,25 @@ public protocol CustomTestArgumentEncodable: Sendable {
 }
 
 extension Test.Case.Argument.ID {
-  /// Initialize this instance with an ID for the specified test argument.
+  /// Initialize an ID instance with the specified test argument value.
   ///
   /// - Parameters:
   ///   - value: The value of a test argument for which to get an ID.
   ///   - parameter: The parameter of the test function to which this argument
   ///     value was passed.
   ///
-  /// - Returns: `nil` if an ID cannot be formed from the specified test
+  /// - Returns: `nil` if a stable ID cannot be formed from the specified test
   ///   argument value.
   ///
   /// - Throws: Any error encountered while attempting to encode `value`.
   ///
+  /// If a stable representation of `value` can be encoded successfully, the
+  /// value of this instance's `bytes` property will be the the bytes of that
+  /// encoded JSON representation and this instance may be considered stable. If
+  /// no stable representation of `value` can be obtained, `nil` is returned. If
+  /// a stable representation was obtained but failed to encode, the error
+  /// resulting from the encoding attempt is thrown.
+  ///
   /// This function is not part of the public interface of the testing library.
   ///
   /// ## See Also
@@ -83,7 +90,7 @@ extension Test.Case.Argument.ID {
       return nil
     }
 
-    self = .init(bytes: try Self._encode(encodableValue, parameter: parameter))
+    self.init(bytes: try Self._encode(encodableValue, parameter: parameter))
 #else
     nil
 #endif

Sources/Testing/Parameterization/Test.Case.Generator.swift

@@ -62,7 +62,7 @@ extension Test.Case {
       // A beautiful hack to give us the right number of cases: iterate over a
       // collection containing a single Void value.
       self.init(sequence: CollectionOfOne(())) { _ in
-        Test.Case(arguments: [], body: testFunction)
+        Test.Case(body: testFunction)
       }
     }
 
@@ -257,7 +257,32 @@ extension Test.Case {
 
 extension Test.Case.Generator: Sequence {
   func makeIterator() -> some IteratorProtocol<Test.Case> {
-    _sequence.lazy.map(_mapElement).makeIterator()
+    let state = (
+      iterator: _sequence.makeIterator(),
+      testCaseIDs: [Test.Case.ID: Int](minimumCapacity: underestimatedCount)
+    )
+
+    return sequence(state: state) { state in
+      guard let element = state.iterator.next() else {
+        return nil
+      }
+
+      var testCase = _mapElement(element)
+
+      if testCase.isParameterized {
+        // Store the original, unmodified test case ID. We're about to modify a
+        // property which affects it, and we want to update state based on the
+        // original one.
+        let testCaseID = testCase.id
+
+        // Ensure test cases with identical IDs each have a unique discriminator.
+        let discriminator = state.testCaseIDs[testCaseID, default: 0]
+        testCase.discriminator = discriminator
+        state.testCaseIDs[testCaseID] = discriminator + 1
+      }
+
+      return testCase
+    }
   }
 
   var underestimatedCount: Int {

Sources/Testing/Parameterization/Test.Case.ID.swift

@@ -15,50 +15,125 @@ extension Test.Case {
   /// parameterized test function. They are not necessarily unique across two
   /// different ``Test`` instances.
   @_spi(ForToolsIntegrationOnly)
-  public struct ID: Sendable, Equatable, Hashable {
+  public struct ID: Sendable {
     /// The IDs of the arguments of this instance's associated ``Test/Case``, in
     /// the order they appear in ``Test/Case/arguments``.
     ///
-    /// The value of this property is `nil` if _any_ of the associated test
-    /// case's arguments has a `nil` ID.
+    /// The value of this property is `nil` for the ID of the single test case
+    /// associated with a non-parameterized test function.
     public var argumentIDs: [Argument.ID]?
 
-    public init(argumentIDs: [Argument.ID]?) {
+    /// A number used to distinguish this test case from others associated with
+    /// the same parameterized test function whose arguments have the same ID.
+    ///
+    /// The value of this property is `nil` for the ID of the single test case
+    /// associated with a non-parameterized test function.
+    ///
+    /// ## See Also
+    ///
+    /// - ``Test/Case/discriminator``
+    public var discriminator: Int?
+
+    /// Whether or not this test case ID is considered stable across successive
+    /// runs.
+    public var isStable: Bool
+
+    init(argumentIDs: [Argument.ID]?, discriminator: Int?, isStable: Bool) {
+      precondition((argumentIDs == nil) == (discriminator == nil))
+
       self.argumentIDs = argumentIDs
+      self.discriminator = discriminator
+      self.isStable = isStable
     }
   }
 
   @_spi(ForToolsIntegrationOnly)
   public var id: ID {
-    let argumentIDs = arguments.compactMap(\.id)
-    guard argumentIDs.count == arguments.count else {
-      return ID(argumentIDs: nil)
-    }
-
-    return ID(argumentIDs: argumentIDs)
+    ID(argumentIDs: arguments.map { $0.map(\.id) }, discriminator: discriminator, isStable: isStable)
   }
 }
 
 // MARK: - CustomStringConvertible
 
 extension Test.Case.ID: CustomStringConvertible {
   public var description: String {
-    "argumentIDs: \(String(describing: argumentIDs))"
+    if let argumentIDs, let discriminator {
+      "Parameterized test case ID: argumentIDs: \(argumentIDs), discriminator: \(discriminator), isStable: \(isStable)"
+    } else {
+      "Non-parameterized test case ID"
+    }
   }
 }
 
 // MARK: - Codable
 
-extension Test.Case.ID: Codable {}
+extension Test.Case.ID: Codable {
+  private enum CodingKeys: String, CodingKey {
+    /// A coding key for ``Test/Case/ID/argumentIDs``.
+    ///
+    /// This case's string value is non-standard because ``legacyArgumentIDs``
+    /// already used "argumentIDs" and this needs to be different.
+    case argumentIDs = "argIDs"
 
-// MARK: - Equatable
+    /// A coding key for ``Test/Case/ID/discriminator``.
+    case discriminator
 
-// We cannot safely implement Equatable for Test.Case because its values are
-// type-erased. It does conform to `Identifiable`, but its ID type is composed
-// of the IDs of its arguments, and those IDs are not always available (for
-// example, if the type of an argument is not Codable). Thus, we cannot check
-// for equality of test cases based on this, because if two test cases had
-// different arguments, but the type of those arguments is not Codable, they
-// both will have a `nil` ID and would incorrectly be considered equal.
-//
-// `Test.Case.ID` is Equatable, however.
+    /// A coding key for ``Test/Case/ID/isStable``.
+    case isStable
+
+    /// A coding key for the legacy representation of ``Test/Case/ID/argumentIDs``.
+    ///
+    /// This case's string value is non-standard in order to maintain
+    /// legacy compatibility with its original value.
+    case legacyArgumentIDs = "argumentIDs"
+  }
+
+  public init(from decoder: any Decoder) throws {
+    let container = try decoder.container(keyedBy: CodingKeys.self)
+
+    if container.contains(.isStable) {
+      // `isStable` is present, so we're decoding an instance encoded using the
+      // newest style: every property can be decoded straightforwardly.
+      try self.init(
+        argumentIDs: container.decodeIfPresent([Test.Case.Argument.ID].self, forKey: .argumentIDs),
+        discriminator: container.decodeIfPresent(Int.self, forKey: .discriminator),
+        isStable: container.decode(Bool.self, forKey: .isStable)
+      )
+    } else if container.contains(.legacyArgumentIDs) {
+      // `isStable` is absent, so we're decoding using the old style. Since the
+      // legacy `argumentIDs` is present, the representation should be
+      // considered stable.
+      let decodedArgumentIDs = try container.decode([Test.Case.Argument.ID].self, forKey: .legacyArgumentIDs)
+      let argumentIDs = decodedArgumentIDs.isEmpty ? nil : decodedArgumentIDs
+
+      // Discriminator should be `nil` for the ID of a non-parameterized test
+      // case, but can default to 0 for the ID of a parameterized test case.
+      let discriminator = argumentIDs == nil ? nil : 0
+
+      self.init(argumentIDs: argumentIDs, discriminator: discriminator, isStable: true)
+    } else {
+      // This is the old style, and since `argumentIDs` is absent, we know this
+      // ID represents a parameterized test case which is non-stable.
+      self.init(argumentIDs: [.init(bytes: [])], discriminator: 0, isStable: false)
+    }
+  }
+
+  public func encode(to encoder: any Encoder) throws {
+    var container = encoder.container(keyedBy: CodingKeys.self)
+
+    try container.encode(isStable, forKey: .isStable)
+    try container.encodeIfPresent(discriminator, forKey: .discriminator)
+    try container.encodeIfPresent(argumentIDs, forKey: .argumentIDs)
+
+    // Encode the legacy representation of `argumentIDs`.
+    if argumentIDs == nil {
+      try container.encode([Test.Case.Argument.ID](), forKey: .legacyArgumentIDs)
+    } else if isStable, let argumentIDs = argumentIDs {
+      try container.encode(argumentIDs, forKey: .legacyArgumentIDs)
+    }
+  }
+}
+
+// MARK: - Equatable, Hashable
+
+extension Test.Case.ID: Equatable, Hashable {}