diagnostics.check doesn't work correctly if used in a Swift testing-based test

[mattmassicotte]

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

• Confirmed reproduction steps with SwiftPM CLI. The description text must include reproduction steps with either of command-line SwiftPM commands, swift build, swift test, swift package etc.

Description

I was working on writing some tests, and I was copying existing implementations. To my surprise, my test passed without me finishing the implementation correctly. I think I know what's up.

DiagnosticsTestResult.check(diagnostic:severity:file:line:) makes use internally of XCTest primitives. Thankfully, this still catches missing diagnostics. However, the validation for both the message and the severity do not work in this configuration.

Expected behavior

I would expect these checks to work in both situations.

Actual behavior

These are only capable of validating the their inputs with XCTest.

Steps to reproduce

Taken from "PackageBuilderTests.swift"

    @Test
    func testEmptyProductNameError() throws {
        let fs = InMemoryFileSystem(emptyFiles: "/Sources/best/best.swift")

        let manifest = Manifest.createRootManifest(
            displayName: "pkg",
            products: [
                try ProductDescription(name: "", type: .library(.automatic), targets: ["best"]),
            ],
            targets: [
                try TargetDescription(name: "best"),
            ]
        )

        try PackageBuilderTester(manifest, in: fs) { package, diagnostics in
            diagnostics.check(diagnostic: "nope this check will still pass", severity: .error)
        }
    }

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version ; uname -a)

No response