Add support for Swift Testing in SwiftSyntaxMacrosTestsSupport by 0xTim · Pull Request #3192 · swiftlang/swift-syntax

0xTim · 2025-11-21T17:02:21Z

Explanation:
As described in Adopt Swift-Testing in test utils such as SwiftSyntaxMacrosTestSupport #2720 SwiftSyntaxMacrosTestsSupport does not work with Swift Testing, which is an issue given Swift Testing is the way to test projects and XCTest is deprecated. If produces false positives (where tests pass even if they shouldn't) which is a major issue, especially as there are no warnings.

This PR adds support for Swift Testing so that tests fail correctly. This does not introduce an issue with a circular dependency on Swift Testing. There is a circular dependency at the package level, but this is allowed due to swiftlang/swift-package-manager#7530. There is no circular dependency between targets.

Scope: No changes to existing code unless tests are passing when they shouldn't be. Only affects SwiftSyntaxMacrosTestsSupport
Issues: Adopt Swift-Testing in test utils such as SwiftSyntaxMacrosTestSupport #2720
Risk: Low risk
Testing: Tests written to ensure it fails as expected
Reviewers:

grynspan · 2025-11-21T17:26:09Z

Sources/SwiftSyntaxMacrosTestSupport/Assertions.swift

    buildConfiguration: buildConfiguration,
    failureHandler: {
+      #if canImport(Testing)
+      if Test.current != nil {


This is not the correct test to determine which library is in use because code can run in a detached task. See swiftlang/swift-testing#475

What is the correct way? Should we split it out into a expect function instead?

There is no correct way at this time, which is why that issue is still open. Jerry's work should allow us to just call #expect() here and have it work under all configurations.

The pitch looks like it will solve the issue, but still require work in the library to migrate over to Swift Testing APIs. What I propose is that we land this now, as it solves a problem that exists for users today (and potentially provide a release in the next monthly Linux release/Swift patch release) and then fix forward when the proposal lands. Given it's still in the pitch phase it likely won't be landed until 6.4 and waiting 10 months for a solution seems like a bad idea.

Regarding the Test.current issue - from my understanding this works in all instances apart from those running in a detached task. For this specific case, I can't see a scenario when a user would be using the assertMacroExpansion from a detached task so we can fix this for the majority of the users and those attempting to use it from a detached task will see no change in behaviour.

@stmontgomery Your take? You okay with this presumably being nonfunctional with the package build?

For this specific case, I can't see a scenario when a user would be using the assertMacroExpansion from a detached task so we can fix this for the majority of the users and those attempting to use it from a detached task will see no change in behaviour.

Let's at least document it as unsupported in the symbol's Markup?

If we can agree here that SwiftSyntax can provide Testing support by depending on the toolchain's Testing module can we unblock the PR on this matter?

I'd be morally okay with saying "the features in the swift-syntax repository are dependent on the built-in copy of Swift Testing even if you include a package dependency" however this will break builds on non-Apple platforms with flat linker namespaces due to duplicate symbols at link time.

I think there is still the larger question on how this is currently implemented. The if Test.current != nil check in this method is certainly less than ideal. Instead I would propose that we add an entirely new method called expectMacroExpansion that is based on Testing unconditionally. So existing users of assertMacroExpansion can continue to use it with XCTest and Testing users can start adopting the new one. How does that sound to everyone?

Once @jerryjrchen's work on the interop feature lands, it will be possible to implement this in a way that depends on neither XCTest nor Swift Testing. It may be a better idea to just wait until that work is done and revisit the problem at that point.

I'd be morally okay with saying "the features in the swift-syntax repository are dependent on the built-in copy of Swift Testing even if you include a package dependency" however this will break builds on non-Apple platforms with flat linker namespaces due to duplicate symbols at link time.

Only if there is both Testing from the toolchain and from the package right? Which I thought we agreed is only really valid in development environments.

Only if there is both Testing from the toolchain and from the package right? Which I thought we agreed is only really valid in development environments.

All testing environments are development environments.

If swift-syntax explicitly links the copy of Swift Testing in the toolchain, that will break developers who want to test their macros and have a package dependency on Swift Testing.

In addition, if they are using the package copy of Swift Testing and you are using the toolchain's copy, your calls to e.g. #expect() won't be routed to the infrastructure the developer's test target links against, so a failure will be invisible.

Jerry's work should give us an escape hatch for this problem, so we should wait until it lands and then make the necessary changes here.

My concern with waiting is that it could be a year until that's shipped, meanwhile users have no indications their tests are passing incorrectly, whereas we could fix it today for the majority of use cases

I appreciate the concern, but there are technical blockers here. We must not cause build failures for teams using Swift Testing as a package.

Sources/SwiftSyntaxMacrosTestSupport/Assertions.swift

…o use a built-in copy (#1452) This adds a new documentation article within the top-level `Documentation/` directory describing the various ways Swift Testing is distributed and offers the specific recommendation that most users prefer using a built-in copy instead of a package dependency. View the [rendered document](https://github.com/stmontgomery/swift-testing/blob/deployment-locations-doc/Documentation/Distributions.md) for more details. > Note: In this PR the new article is only being placed in the `Documentation/` directory. I chose to place it there, initially, and not within the project's official DocC documentation catalog, but that could potentially happen in the future. ### Motivation: We have been casually giving users the recommendation to prefer built-in copies of the `Testing` module for a while in settings like the Swift forums, bug reports, and informal chats. But thus far, we have not collected and formally documented the reasons why using a package copy can encounter issues. This discussion recently came up during a [discussion](swiftlang/swift-syntax#3192) about whether to adopt Swift Testing in a test-helper library in SwiftSyntax. My hope is that adding this document will help clarify expectations around the level of support we offer for the two distribution styles for Swift Testing. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated. --------- Co-authored-by: Joseph Heck <j_heck@apple.com>

0xTim · 2026-02-02T19:38:21Z

Ok all, I've introduced a separate function to be used from Swift testing and added some documentation to the assert functions. Let me know what you think!

cc @FranzBusch

grynspan · 2026-02-02T21:20:22Z

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

FranzBusch · 2026-02-02T21:24:30Z

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

I thought we identified in the other thread that this is not the case. While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting. I think the new documentation in the swift-testing makes it pretty clear that the primary use-case of swift-testing as a package is for local development.

0xTim · 2026-02-02T21:33:55Z

To add some more context as well, I was talking to a couple of maintainers at FOSDEM who have both hit this issue with false positives

I think this is still ultimately a moot point with @jerryjrchen's upcoming work?

This work is still ongoing correct? Because it won't land until 6.4 at the earliest I'd assume

grynspan · 2026-02-02T21:58:13Z

Right but that doesn't mean we should ship new API that immediately becomes obsolete with the next Swift release.

grynspan · 2026-02-02T22:01:15Z

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

The end result of Jerry's work is, as I understand it, that the existing API in this library can adopt #expect() in place of XCTAssertTrue() (etc. etc.) and it'll work in both libraries without problems.

I think the new documentation in the swift-testing makes it pretty clear that the primary use-case of swift-testing as a package is for local development.

I'm not sure I understand how that relates to my comment, but maybe you're referring to something earlier in the conversation I missed?

jerryjrchen · 2026-02-03T00:09:02Z

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

We can resolve the strict mode restriction against using XCTest API in a Swift Testing test by adopting #expect in place of XCTest API (I can see XCTAssert* and XCTFail from a quick search). Interop also ensures the conversion to #expect still works as expected if the helpers are being called from an XCTest test case.

stmontgomery · 2026-02-03T04:43:44Z

Hi folks, I want to suggest a path forward here which I hope will unblock progress on this PR and allow it to proceed in the short term (if accepted by the repo's code owners, of course!) while laying out a plan for how it will align with our anticipated and fast-approaching interoperability work.

Keep just one "assert" function, with the existing name assertMacroExpansion()
- …instead of introducing a separate function expectMacroExpansion() and having one for XCTest, one for Swift Testing.
- This is to avoid bifurcation between the XCTest and Swift Testing worlds.
Modify assertMacroExpansion() so that it calls both Issue.record() (Swift Testing) and XCTFail() (XCTest), one after the other.
- For right now, this is sufficient to ensure failures are surfaced regardless of which testing library is active.
- Note: to achieve full reporting fidelity, the function will need to gain fileID: and column: parameters with default values and plumb those through to the "generic" funnel point so Swift Testing can reference them.
(Important) Communicate the potential effects of this change to SwiftSyntax users
- This could happen via a detailed release note in the tag this change lands in. See the 602.0.0 notes for example of the format of the last release.
- This note should do several things:
  - First, celebrate the useful new functionality—Swift Testing support for macro testing!
  - Then, explain the known caveat(s). The main one being that if you have a package dependency on Swift Testing, you upgrade to this (major) version of SwiftSyntax, and you attempt to validate macros from a Swift Testing test, then it's possible failures may not be surfaced correctly.
  - Link to my new document in the Swift Testing repo about this topic, and emphasize that it's due to using a non-built in copy.
  - Offer two solutions for anyone affected by this:
    - Simplest: Remove the package dependency on Swift Testing and switch to using a built-in copy, as the document recommends.
    - Otherwise, switch from importing SwiftSyntaxMacrosTestSupport to SwiftSyntaxMacrosGenericTestSupport and specify your own failureHandler: closure in which you call Issue.record() yourself. That should work even when using a package copy of Swift Testing.
Later, in a subsequent PR, adjust this logic for Swift Testing interoperability
- If/when the pitched Interoperability feature arrives, assertMacroExpansion() will need to be adjusted to avoid redundant test issues being recorded due to it calling both Issue.record() and XCTFail().
- To avoid that, at that point in time assertMacroExpansion() can be modified to stop calling XCTFail() starting in the Swift version where interoperability is first enabled.
- (The call to XCTFail() shouldn't be removed altogether though, since SwiftSyntax could presumably still be built and used from older Swift versions.)

I've assembled a branch (atop this PR branch) with my suggested code changes for this PR in a commit you can see here: stmontgomery@c772dc6. Thoughts?

stmontgomery · 2026-02-03T04:49:04Z

There are two other points above I wanted to respond to:

While the interoperability will allow the existing XCTest based method to work. It won't work with the strict mode which seems limiting.

Building on what @jerryjrchen said, the semantics of strict mode will be that in any situation where Swift Testing offers an API, that should be preferred over the analogous XCTest variant. It does not mandate that you must only call the APIs corresponding to the active test framework, in case that's something folks are concerned about here. I do think/hope strict mode can eventually be used in conjunction with this work, if end users choose to enable it.

We must not cause build failures for teams using Swift Testing as a package.

I recognize that this plan does have the potential to break users who have a package dependency on Swift Testing. I suspect the practical impact of that is likely small due to few users relying on it, and I think the tradeoff here is very worthwhile since we'll be delivering a frequently requested feature. The document I wrote about distribution locations of Swift Testing was intended to help clarify our recommendations regarding use of the package copy, and frankly, to discourage its use more strongly than we have in the past.

This change will roll out in a new major tag of SwiftSyntax, meaning (per SemVer) it will be an explicit action users must take so the timing of the upgrade will be in their control. And I believe we can mitigate and help users make the transition gracefully with a clearly-worded release note (discussed above).

0xTim · 2026-02-03T12:43:50Z

@stmontgomery looks good to me - happy to pull those changes into my branch as I have done (just some extra changes to add if you agree around the can import stuff and expanding the test info) or have you submit the PR

rintaro · 2026-02-06T18:36:31Z

Sorry I haven't been following this PR, so I may be missing some context.

FWIW, the original intent behind the current structure was to take advantage of cross-import overlays once it's accepted and SwiftPM supports them.
The idea was:

Rename the current XCTest-based SwiftSyntaxMacrosTestSupport to SwiftSyntaxMacrosTestSupport_XCTest
Rename the current SwiftSyntaxMacrosGenericTestSupport to SwiftSyntaxMacrosTestSupport
Introduce SwiftSyntaxMacrosTestSupport_SwiftTesting, which provides the swift-testing implementation of assertMacroExpansion()
With that in place, users could simply write:

import SwiftSyntaxMacrosTestSupport
import SwiftTesting

and have SwiftSyntaxMacrosTestSupport_SwiftTesting picked up automatically via cross-import overlays.

That's said I'm a bit skeptical that cross-import overlays will be available in the near future. Also, as I understand it, we likely can't utilize Package traits because we treat XCTest and Testing as system libraries. Given that, plus the direction of the interoperability work, this PR (and the plan from @stmontgomery) seems to be the right direction.

A couple of questions:

Modify assertMacroExpansion() so that it calls both Issue.record() (Swift Testing) and XCTFail() (XCTest), one after the other.

Just to confirm, this is safe because XCTAssert failure in a Swift Testing test or an #expect failure in an XCTest test is silently ignored (from the interoperability proposal)?

What would happen after the interoperability is implemented? Would a failure be reported twice? Or the interoperability feature would consolidate the duplicated failure into one?

if you have a package dependency on Swift Testing, you upgrade to this (major) version of SwiftSyntax, and you attempt to validate macros from a Swift Testing test, then it's possible failures may not be surfaced correctly.

Could you elaborate on this? Any example that would break with this change? I don't believe a simple setup like this would work even with the current situation.

import SwiftSyntaxMacroTestSupport
import Testing

@Test func testMyMacro() {
  assertMacroExpansion(...)
}

rintaro · 2026-02-06T18:43:59Z

Sources/SwiftSyntaxMacrosTestSupport/Assertions.swift

+/// - Warning: If you call this function inside a Swift Testing test case,
+/// the test will report no assertion failures even if the test fails.
+/// Use ``expectMacroExpansion(_:expandedSource:diagnostics:macroSpecs:applyFixIts:fixedSource:testModuleName:testFileName:indentationWidth:buildConfiguration:fileID:file:line:column:)`` instead.
+///


This (and the one on the another function) isn't true now.

Since which Swift version? Because it's definitely the case in 6.2.3

I meant after your latest change. You're not even adding expectMacroExpansion anymore. :)

rintaro · 2026-02-06T18:50:51Z

Could you elaborate on this? Any example that would break with this change? I don't believe a simple setup like this would work even with the current situation.

Ah, I see it now. SwiftSytaxMacrosTestingSupport might be built against the toolchain Testing, but the clients will be built against the just-built Testing because they have the dependencies. And that will cause the issue.

0xTim added 3 commits November 21, 2025 16:35

Record issues if we're in a Swift testing environment

2ddc19b

Add support in other method

ba15ee0

Add tests

dff2179

0xTim requested review from ahoppen, bnbarham, hamishknight and rintaro as code owners November 21, 2025 17:02

0xTim mentioned this pull request Nov 21, 2025

Adopt Swift-Testing in test utils such as SwiftSyntaxMacrosTestSupport #2720

Open

grynspan requested changes Nov 21, 2025

View reviewed changes

Sources/SwiftSyntaxMacrosTestSupport/Assertions.swift Outdated Show resolved Hide resolved

0xTim added 3 commits November 21, 2025 17:39

Private imports

d7fd8ff

Fix some of the CI issues

8f55686

Fix formatting

be537f1

stmontgomery mentioned this pull request Dec 16, 2025

Document the official distribution locations and the recommendation to use a built-in copy swiftlang/swift-testing#1452

Merged

2 tasks

Introduce expect based function to be used from Swift Testing

f765a21

0xTim requested a review from hborla as a code owner February 2, 2026 19:35

canImport wrap

6287124

Unify the assertions

725d2b5

rintaro reviewed Feb 6, 2026

View reviewed changes

Conversation

0xTim commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grynspan Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grynspan Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

0xTim commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grynspan commented Feb 2, 2026

Uh oh!

FranzBusch commented Feb 2, 2026

Uh oh!

0xTim commented Feb 2, 2026

Uh oh!

grynspan commented Feb 2, 2026

Uh oh!

grynspan commented Feb 2, 2026

Uh oh!

jerryjrchen commented Feb 3, 2026

Uh oh!

stmontgomery commented Feb 3, 2026

Uh oh!

stmontgomery commented Feb 3, 2026

Uh oh!

0xTim commented Feb 3, 2026

Uh oh!

rintaro commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rintaro commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

grynspan Nov 21, 2025 •

edited

Loading

grynspan Dec 10, 2025 •

edited

Loading

0xTim commented Feb 2, 2026 •

edited

Loading

rintaro commented Feb 6, 2026 •

edited

Loading

rintaro Feb 6, 2026 •

edited

Loading

rintaro commented Feb 6, 2026 •

edited

Loading