Improve the example test(s) in README.md

[dabrahams]

Description

The example here that is supposed to demonstrate its “Clear Expressive API” is confusing as heck.

@Test("Continents mentioned in videos", arguments: [
    "A Beach",
    "By the Lake",
    "Camping in the Woods"
])
func mentionedContinents(videoName: String) async throws {
    let videoLibrary = try await VideoLibrary()
    let video = try #require(await videoLibrary.video(named: videoName))
    #expect(video.mentionedContinents.count <= 3)
}

What role does arguments play? After a long while, I guess they are getting passed as video names. If so, active names like “Fonzy Jumps the Shark” would help clarify that. Lastly, are we really testing that there are fewer than 4 continents mentioned in each video? That's an incredibly implausible test, but if you want to use it, the name of the test should be “3 or fewer continents mentioned in videos". Otherwise it sounds like its testing that continents are mentioned in videos, which it does not do! A count of zero would pass the test.

Reproduction

Expected behavior

Example that demonstrates the clarity of the API

Environment

Swift Package Manager - Swift 6.3.3

Additional information

No response

[leticiamazzoportela]

I proposed a fix to improve this in PR #1779 🙂

[grynspan]

A better approach would be to rework the readme file's contents. I don't think Dave's concern here is just about the name of the test. :)

[dabrahams]

No, not just the name! It's about how the whole of the example makes it hard to guess what the code means. The implausibility of the actual test makes it hard to believe you've understood correctly. The intuitability of the meaning of arguments: is low. I'd never seen a test with arguments before, and then I expected the three ”arguments” to match three parameters, but the test has only one parameter. You could help that by using a less common type than String for these things, to help distinguish from the test name and (whose role is also non-obvious since the test function already has a name).

Honestly I'm beginning to wonder if the API really is clear. XCTesting code is bog-simple and obvious; the only slightly surprising thing is that test names have to begin with test. You want to test 3 things? You write them out or you write a loop. The name of the test is… the name of the test. Every single time I think about using swift-testing again I come upon an initial example like this one and it takes me a minute of staring to guess what it probably means. Having had this experience 3 or 4 times is what caused me to file this issue; it's not just based on one impression. I didn't even mention the #require/#expect distinction, which I can't intuit. Having known the answer once upon a time doesn't help, which I take as more evidence that it isn't clear. (The need to effectively repeat the name of the test in a string also makes me question the API's expressivity.)

So my points here:

1. The initial example should be a pattern that a new user can pick up and run with to get started without wondering what it actually means. Showing off features should be a non-goal.
2. The API is probably not all that clear and could use some work. Choices like #require/#expect seem to value brevity over clarity.
3. If the doc is going to claim the API is clear, the example should support that claim, and this one doesn't.

[grynspan]

The need to effectively repeat the name of the test in a string also makes me question the API's expressivity.

For what it's worth, this isn't actually necessary. You can write either of these:

@Test func eatABagOfChips() {}
@Test func `Eat a bag of chips`() {}

[dabrahams]

Then the fact that examples seem to imply I need to give a name is a weakness. First show people how simple and clear the library can be to use.

In fact, given what you've just shown, IMO the existence of the separate name string is a misfeature.

[grynspan]

The example predates the existence of raw identifiers in Swift.

You'll note I haven't closed this issue. We agree we can improve the text and examples here. 🙂

[stmontgomery]

Hello, thanks for the feedback. At a high level I agree the README can be improved, especially since the introduction of Raw Identifiers in the language. In the issue description above, the link is to the marketing page for Swift Testing on Apple's developer website:

https://developer.apple.com/xcode/swift-testing/

The content of that page is beyond the scope of this open source project. Some of that content was borrowed/adapted when this open source project was first introduced and placed in the README. However, in the README this example code was ordered much lower in the document and given a different subtitle and description. Here's what it currently contains (emphasis mine):

Scalable coverage and execution

Parameterized tests help you run the same test over a sequence of values so you can write less code. And all tests integrate seamlessly with Swift Concurrency and run in parallel by default.

@Test("Continents mentioned in videos", arguments: [
    "A Beach",
    "By the Lake",
    "Camping in the Woods"
])
func mentionedContinents(videoName: String) async throws {
    let videoLibrary = try await VideoLibrary()
    let video = try #require(await videoLibrary.video(named: videoName))
    #expect(video.mentionedContinents.count <= 3)
}

I wrote this example code originally a couple years ago, and reflecting on it now, I agree it's somewhat contrived looking. I think we should pick a workflow that is more realistic and that readers might connect with better. However I do think that compared to the Apple marketing page, the way this material is presented in the README does a better job explaining the purpose of parameterized testing (which is a more advanced feature, hence its lower placement). Its section title is "Scalable coverage and execution" here; the section title "Clear, expressive API" is used to showcase another, much simpler "hello world" style example atop the document. So, I don't think this point at least is a concern, since it isn't the initial example in the README:

1. The initial example should be a pattern that a new user can pick up and run with to get started without wondering what it actually means. Showing off features should be a non-goal.

[dabrahams]

I expected there to be a pathway back from this project to the Apple developer doc people via a radar. Are there no Apple devs on this project?

[dabrahams]

Quoth @grynspan:

You'll note I haven't closed this issue. We agree we can improve the text and examples here. 🙂

Much appreciated, along with the lack of a defensive posture!