diff --git a/Documentation/ExpectationCapture.md b/Documentation/ExpectationCapture.md new file mode 100644 index 000000000..7bbce174f --- /dev/null +++ b/Documentation/ExpectationCapture.md @@ -0,0 +1,405 @@ +# Rethinking expectation capture in Swift Testing + + + + + +Since we announced Swift Testing and shipped it to developers with the Swift 6 +toolchain, we've been looking at how we could improve the library. Some changes +we hope to introduce, like exit tests and attachments, are major new features. +Others are smaller quality-of-life changes (bug fixes, performance improvements, +and so on.) + +### ℹ️ Be advised +This post assumes some knowledge and understanding of Swift macros and how they +work. In particular, you should know that when the Swift compiler encounters an +_expression_ of the form `#m(a, b, c)`, it passes that expression's +_abstract syntax tree_ (AST) to a compiler plugin that then replaces it with an +equivalent AST known as its _expansion_. + +## Our expectations for expectations + +One area we knew we'd want to revisit was the `#expect()` and `#require()` +macros and how they work. `#expect()` and `#require()` are more than just +functions: they're _expression macros_. We wrote them as expression macros so +that we could give them some special powers. + +When you use one macro or the other, it examines the AST of its condition +argument and looks for one of several known kinds of expression (e.g. binary +operators, member function calls, or `is`/`as?` casts). If found, the macro +rewrites the expression in a form that Swift Testing can examine at runtime. +Then, if the expectation fails, Swift Testing can produce better diagnostics +than it could if it just treated the condition expression as a boolean value. + +For example, if you write this expectation: + +```swift +#expect(f() < g()) +``` + +… Swift Testing is able to tell you the results of `f()` and `g()` in addition +to the overall expression `f() < g()`. + +### Effectively ineffective macros + +This works fairly well for simple expressions like that one, but it doesn't have +the flexibility needed to tell a test author what goes wrong when a more complex +expression is used. For example, this expression: + +```swift +#expect(x && y && !z) +``` + +… is subject to a transformation called "operator folding" that takes place +prior to macro expansion, and the AST represents it as a binary operator whose +left-hand operand is _another_ binary operator. Swift Testing doesn't know how +to recursively expand the nested binary operator, so it only produces values for +`x && y` and `!z`. + +We also run into trouble when effects (`try` and `await`) are in play. Swift +Testing's implementation can't readily handle arbitrary combinations of +subexpressions that may or may not need either keyword. As a result, it doesn't +try to expand expressions that contain effects. If `f()` or `g()` is a throwing +function: + +```swift +#expect(try f() < g()) +``` + +… Swift Testing will not attempt to do any further processing, and only the +outermost expression (`try f() < g()`) will be captured. + +## Looking forward + +Now that Swift 6.1 has branched for its upcoming release, we can start to look +at the _next_ Swift version and how we can improve Swift Testing to help make it +the _Awesomest Swift Release Ever_. And we'd like to start by revisiting how +we've implemented these macros. + +We've been working on [a branch](https://github.com/swiftlang/swift-testing/tree/jgrynspan/162-redesign-value-capture) +of Swift Testing (with a corresponding [draft PR](https://github.com/swiftlang/swift-testing/pull/840)) +that completely redesigns the implementation of the `#expect()` and `#require()` +macros. Instead of trying to sniff out an "interesting" expression to expand, +the code on this branch walks the AST of the condition expression and rewrites +_all_ interesting subexpressions. + +This expectation: + +```swift +#expect(x && y && !z) +``` + +Can now be fully expanded and will provide a full breakdown of the condition at +runtime if it fails: + +``` +◇ Test example() started. +✘ Test example() recorded an issue at Example.swift:1:2: Expectation failed: x && y && !z → false +↳ x && y && !z → false +↳ x && y → true +↳ x → true +↳ y → true +↳ !z → false +↳ z → true +✘ Test example() failed after 0.002 seconds with 1 issue. +``` + +## How you can help + +Before we merge this PR and enable these changes in prerelease Swift toolchains, +we’d love it if you could try it out! These changes are a major change for Swift +Testing and the more feedback we can get, the better. To try out the changes: + +1. _Temporarily_ add an explicit package dependency on Swift Testing and point + Swift Package Manager or Xcode to the branch. In your Package.swift file, add + this dependency: + + ```swift + dependencies: [ + /* ... */ + .package( + url: "https://github.com/swiftlang/swift-testing.git", + branch: "jgrynspan/162-redesign-value-capture" + ), + ], + ``` + + And update your test target: + + ```swift + .testTarget( + name: "MyTests", + dependencies: [ + /* ... */ + .productItem(name: "Testing", package: "swift-testing"), + ] + ) + ``` + + If you’re using an Xcode project, you can add a package dependency via the + **Package Dependencies** tab in your project’s configuration. Add the + `Testing` target as a dependency of your test target and click + **Trust & Enable** to use the locally-built `TestingMacros` target. + +1. Once you’ve added the package dependency, clean your package + (`swift package clean`) or project (**Product** → **Clean Build Folder…**), + then build and run your tests. + + Swift Testing will be built from source along with swift-syntax, which may + significantly increase your build times, so we don’t recommend doing this in + production—this is an at-desk experiment. Swift Testing will be built with + optimizations off by default, so runtime performance may be impacted, but + that’s okay: we’re mostly concerned about correctness rather than raw + performance measurements for the moment. + +1. Let us know how your experience goes, especially if you run into problems. + You can reach me by sending me [a forum DM](https://forums.swift.org/u/grynspan/summary) + or by commenting on [this PR](https://github.com/swiftlang/swift-testing/pull/840). + +## Here be caveats + +There are some expressions that Swift Testing could previously successfully +expand that will cause problems with this new implementation: + +- Expectations with effects where the effect keyword is to the left of the macro + name: + + ```swift + try #expect(h()) + ``` + + Macros cannot currently "see" effect keywords in this position. The old + implementation would often compile because the expansion didn't introduce a + nested closure scope (which necessarily must repeat these keywords in other + positions.) The new implementation does not know it needs to insert the `try` + keyword anywhere in this case, resulting in some confusing diagnostics: + + > ⚠️ No calls to throwing functions occur within 'try' expression + > + > 🛑 Call can throw, but it is not marked with 'try' and the error is not + > handled + + [Stuart Montgomery](https://github.com/stmontgomery) and I chatted with + [Doug Gregor](https://github.com/DougGregor) and [Holly Borla](https://github.com/hborla) + recently about this issue; they're looking at the problem and seeing what sort + of compiler-side support might be possible to help solve it. + + [Stuart Montgomery](https://github.com/stmontgomery) has opened [a PR](https://github.com/swiftlang/swift-syntax/pull/2724) + against swift-syntax that we hope will help resolve this issue. + + **To avoid this issue**, always place `try` and `await` _within_ the argument + list of `#expect()`: + + ```swift + #expect(try h()) + ``` + + For `#require()`, the implementation knows that `try` must be present to the + left of the macro. + +- Expectations with particularly complex conditions can, after expansion, + overwhelm the type checker and fail to compile: + + > 🛑 The compiler is unable to type-check this expression in reasonable time; + > try breaking up the expression into distinct sub-expressions + + Because macros have little-to-no type information, there aren't a lot of + opportunities for us to provide any in the macro expansion. I've reached out + to [Holly Borla](https://github.com/hborla) and her colleagues to see if + there's room for us to improve our implementation in ways that help the type + checker. + + **If your expectation fails with this error,** break up the expression as + recommended and only include part of the expression in the macro's argument + list: + + ```swift + let x = ... + let y = ... + #expect(x == y) + ``` + +- Type names are (syntactically speaking) indistinguishable from variable names. +That means there may be some expressions that we _could_ expand further, but +because we can't tell if a syntax node refers to a variable, a type, or a +module, we don't try: + + ```swift + #expect(a.b == c) // a may not be expressible in isolation + ``` + + Where we think we can expand such a syntax node, the macro expansion appends + `.self` to the node in case it refers to a type. There may be cases where the + macro expansion logic does not work as intended: please send us bug reports if + you find them! + +- The `==`, `!=`, `===`, and `!==` operators are special-cased so that we can + use [`difference(from:)`](https://developer.apple.com/documentation/swift/bidirectionalcollection/difference(from:)) + to compare operands where possible. The macro implementation assumes that + these operators eagerly evaluate their arguments (unlike operators like `&&` + that short-circuit their right-hand arguments using `@autoclosure`.) This + assumption is true of all implementations of these operators in the Swift + Standard Library, but we can't make any real guarantees about third-party code. + + We believe that this should not be a common issue in real-world code, but + please reach out if Swift Testing is expanding these operators incorrectly in + your code. + +### Disabling expression expansion + +If you have a condition expression that you don't want expanded at all (for +instance, because the macro is incorrectly expanding it, or because its +implementation might be affected by side effects from Swift Testing), you can +cast the expression with `as Bool` or `as T?`: + +```swift +let x = ... +let y = ... +#expect((x == y) as Bool) + +let z: String? +let w = try #require(z as String?) +``` + +## Example expansions + +Here (hidden behind disclosure triangles so as not to frighten children and +pets) are some before-and-after examples of how Swift Testing expands the +`#expect()` macro. I've cleaned up the whitespace to make it easier to read. + +
+#expect(f() < g()) + +#### Before +```swift +Testing.__checkBinaryOperation( + f(), + { $0 < $1() }, + g(), + expression: .__fromBinaryOperation( + .__fromSyntaxNode("f()"), + "<", + .__fromSyntaxNode("g()") + ), + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` + +#### After +```swift +Testing.__checkCondition( + { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in + __ec(__ec(f(), 0x2) < __ec(g(), 0x400), 0x0) + }, + sourceCode: [ + 0x0: "f() < g()", + 0x2: "f()", + 0x400: "g()" + ], + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` +
+ +
+#expect(x && y && !z) + +#### Before +```swift +Testing.__checkBinaryOperation( + x && y, + { $0 && $1() }, + !z, + expression: .__fromBinaryOperation( + .__fromSyntaxNode("x && y"), + "&&", + .__fromSyntaxNode("!z") + ), + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` + +#### After +```swift +Testing.__checkCondition( + { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in + __ec(__ec(__ec(x, 0x6) && __ec(y, 0x42), 0x2) && __ec(!__ec(z, 0x1400), 0x400), 0x0) + }, + sourceCode: [ + 0x0: "x && y && !z", + 0x2: "x && y", + 0x6: "x", + 0x42: "y", + 0x400: "!z", + 0x1400: "z" + ], + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` +
+ +
+#expect(try f() < g()) + +#### Before +```swift +Testing.__checkValue( + try f() < g(), + expression: .__fromSyntaxNode("try f() < g()"), + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` + +#### After +```swift +try Testing.__checkCondition( + { (__ec: inout Testing.__ExpectationContext) -> Swift.Bool in + try __ec(__ec(f(), 0xc) < __ec(g(), 0x1004), 0x4) + }, + sourceCode: [ + 0x4: "f() < g()", + 0xc: "f()", + 0x1004: "g()" + ], + comments: [], + isRequired: false, + sourceLocation: Testing.SourceLocation.__here() +).__expected() +``` +
+ +> [!NOTE] +> **What's with the hexadecimal?** +> +> You'll note that all calls to `__ec()` include an integer literal argument, +> and the `sourceCode` argument is a dictionary whose keys are integer literals +> too. These values represent the unique identifiers of each captured syntax +> node from the original AST. They uniquely encode the syntax nodes' positions +> in the tree so that we can reconstruct the (sparse) tree at runtime when a +> test fails. +> +> [I](http://github.com/grynspan) find this subtopic interesting enough to want +> to devote a whole forum thread to it, personally, but it's a bit arcane—for +> more information, see the implementation [here](https://github.com/swiftlang/swift-testing/blob/jgrynspan/162-redesign-value-capture/Sources/Testing/SourceAttribution/ExpressionID.swift) +> or feel free to reach out to me via forum DM. diff --git a/Documentation/README.md b/Documentation/README.md index ea8ff2f69..e41bc9568 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -23,12 +23,15 @@ as supplemental content located in the [`Sources/Testing/Testing.docc/`](https://github.com/swiftlang/swift-testing/tree/main/Sources/Testing/Testing.docc) directory. -## Vision document +## Vision document and API proposals The [Vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md) for Swift Testing offers a comprehensive discussion of the project's design principles and goals. +The [`Proposals`](Proposals/) directory contains API proposals that have been +accepted and merged into Swift Testing. + ## Development and contribution - The top-level [`README`](https://github.com/swiftlang/swift-testing/blob/main/README.md) @@ -49,8 +52,16 @@ principles and goals. - Instructions are provided for running tests against a [WASI/WebAssembly target](https://github.com/swiftlang/swift-testing/blob/main/Documentation/WASI.md). -## Project maintenance +## Testing library ABI + +The [`ABI`](ABI/) directory contains documents related to Swift Testing's ABI: +that is, parts of its interface that are intended to be stable over time and can +be used without needing to write any code in Swift: -- The [Releases](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Releases.md) - document describes the process of creating and publishing a new release of - Swift Testing — a task which may be performed by project administrators. +- [`ABI/JSON.md`](ABI/JSON.md) contains Swift Testing's JSON specification that + can be used by tools to interact with Swift Testing either directly or via the + `swift test` command-line tool. +- [`ABI/TestContent.md`](ABI/TestContent.md) documents the section emitted by + the Swift compiler into test products that contains test definitions and other + metadata used by Swift Testing (and extensible by third-party testing + libraries.) diff --git a/Documentation/Releases.md b/Documentation/Releases.md deleted file mode 100644 index 5f1a20cf4..000000000 --- a/Documentation/Releases.md +++ /dev/null @@ -1,131 +0,0 @@ -# How to create a release of Swift Testing - - - -This document describes how to create a new release of Swift Testing using Git -tags. - -> [!IMPORTANT] -> You must have administrator privileges to create a new release in this -> repository. - -## Version numbering - -Swift Testing uses [semantic versioning](https://semver.org) numbers for its -open source releases. We use Git _tags_ to publish new releases; we don't use -the GitHub [releases](https://docs.github.com/en/repositories/releasing-projects-on-github/about-releases) -feature. - -At this time, all Swift Testing releases are experimental, so the major version -is always `0`. We aren't using the patch component, so it's usually (if not -always) `0`. The minor component should be incremented by one for each release. - -For example, if the current release is version `0.1.0` and you are publishing -the next release, it should be `0.2.0`. - -> [!NOTE] -> Where you see `x.y.z` in this document, substitute the semantic version you -> are deploying. - -## Creating a branch for the release - -Before a release can be published, a branch must be created so that the -repository can be configured correctly. Ensure any local changes have been saved -and cleared from the repository (e.g. with `git stash` or `git reset --hard`), -then run the following commands from within the repository's root directory: - -```sh -git checkout main # or other branch as appropriate -git pull -git checkout -b release/x.y.z -``` - -## Preparing the repository's contents - -The package manifest files (Package.swift _and_ Package@swift-6.0.swift) must -be updated so that the release can be used as a package dependency: - -1. Delete any unsafe flags from `var packageSettings` as well as elsewhere in - the package manifest files. - -The repository's local state is now updated. To commit it to your branch, run -the typical commit command: - -```sh -git commit -a -m "Deploy x.y.z" -``` - -## Smoke-testing the branch - -Before deploying the tag publicly, test it by creating a simple package locally. -For example, you can initialize a new package in an empty directory with: - -```sh -swift package init --enable-experimental-swift-testing -``` - -Then modify the package's `Package.swift` file to point at your local clone of -the Swift Testing repository. Ensure that the package's test target builds and -runs successfully with: - -```sh -swift test -``` - -> [!NOTE] -> Be sure to test changes on both macOS and Linux using the most recent -> main-branch Swift toolchain. - -If changes to Swift Testing are necessary for the build to succeed, open -appropriate pull requests on GitHub, then rebase your tag branch after they're -merged. - -## Committing changes and pushing the release - -Run the following commands to push the release and make it publicly visible: - -```sh -git tag x.y.z -git push -u origin x.y.z -``` - -The release is now live and publicly visible [here](https://github.com/swiftlang/swift-testing/tags). -Developers using Swift Package Manager and listing Swift Testing as a dependency -will automatically update to it. - -## Oh no, I made a mistake… - -Don't panic. We all make mistakes. - -### … but I haven't pushed the release yet. - -If you've already created the release's tag locally, but haven't pushed it yet, -delete it with `git tag -d x.y.z`, resolve the issue, and recreate the tag by -following the steps above. - -### … but I can fix it. - -If the release is usable, but contains a bug that _cannot_ wait until the next -planned release to be fixed, a patch release can be deployed. First, fix the -issue locally. Then, follow the steps above to create a new release. Where you -would normally increment the _minor_ version component, increment the _patch_ -version component instead. For example, if the most recent release was `0.1.2`, -the fix should be released as `0.1.3`. - -### … and the release is completely unusable! - -If the release is broken and will not be usable by developers, delete the -release's tag from GitHub using `git push --delete origin x.y.z` so that -developers do not inadvertently download it. - -> [!IMPORTANT] -> Deleting a release or tag is often considered bad form, so only do so if the -> release is truly unusable.