Improve integration tests (stability + performance) (#15125)

This PR improves the integration tests in two ways:
1. Make the integration tests more reliable and thus less flakey
2. Make the integration tests faster (by introducing concurrency)

Tried a lot of different things to make sure that these tests are fast
and stable.

---

The biggest issue we noticed is that some tests are flakey, these are
tests with long running dev-mode processes where watchers are being used
and/or dev servers are created.
To solve this, all the tests that spawn a process look at stdout/stderr
and wait for a message from the process to know whether we can start
making changes.

For example, in case of an Astro project, you get a `watching for file
changes` message. In case of Nuxt project you can wait for an `server
warmed up in` and in case of Next.js there is a `Ready in` message.

These depend on the tools being used, so this is hardcoded per test
instead of a magically automatic solution.

These messages allow us to wait until all the initial necessary work,
internal watchers and/or dev servers are setup before we start making
changes to the files and/or request CSS stylesheets before the server(s)
are ready.

---

Another improvement is how we setup the dev servers. Before, we used to
try and get a free port on the system and use a `--port` flag or a
`PORT` environment variable. Instead of doing this (which is slow), we
rely on the process itself to show a URL with a port. Basically all
tools will try to find a free port if the default port is in use. We can
then use the stdout/stderr messages to get the URL and the port to use.

To reduce the amount of potential conflicts in ports, we used to run
every test and every file sequentially to basically guarantee that ports
are free. With this new approach where we rely on the process, I noticed
that we don't really run into this issue again (I reran the tests
multiple times and they were always stable)

<img width="316" alt="image"
src="https://github.com/user-attachments/assets/b75ddab4-f919-4995-85d0-f212b603e5c2"
/>
Note: these tests run Linux, Windows and macOS in this branch just for
testing purposes. Once this is done, we will only run Linux tests on PRs
and run all 3 of them on the `next` branch.

We do make the tests concurrent by default now, which in theory means
that there could be conflicts (which in practice means that the process
has to do a few more tries to find a free port). To reduce these
conflicts, we split up the integration tests such that Vite, PostCSS,
CLI, … tests all run in a separate job in the GitHub actions workflow.

<img width="312" alt="image"
src="https://github.com/user-attachments/assets/fe9a58a1-98eb-4d9b-8845-a7c8a7af5766"
/>

Comparing this branch against the `next` branch, this is what CI looks
like right now:

| `next` | `feat/improve-integration-tests` |
| --- | --- |
| <img width="594" alt="image"
src="https://github.com/user-attachments/assets/540d21eb-ab03-42e8-9f6f-b3a071fc7635"
/> | <img width="672" alt="image"
src="https://github.com/user-attachments/assets/8ef2e891-08a1-464b-9954-4153174ebce7"
/> |

There also was a point in time where I introduced sequential tests such
that all spawned processes still run after each other, but so far I
didn't run into issues if we keep them concurrent so I dropped that
code.

Some small changes I made to make things more reliable:
1. When relying on stdout/stderr messages, we split lines on `\n` and we
strip all the ANSI escapes which allows us to not worry about special
ANSI characters when finding the URL or a specific message to wait for.
2. Once a test is done, we `child.kill()` the spawned process. If that
doesn't work, for whatever reason, we run a `child.kill('SIGKILL')` to
force kill the process. This could technically lead to some memory or
files not being cleaned up properly, but once CI is done, everything is
thrown away anyway.
3. As you can see in the screenshots, I used some nicer names for the
workflows.

| `next` | `feat/improve-integration-tests` |
| --- | --- |
| <img width="276" alt="image"
src="https://github.com/user-attachments/assets/e574bb53-e21b-4619-9cdb-515431b255b9"
/> | <img width="179" alt="image"
src="https://github.com/user-attachments/assets/8bc75119-fb91-4500-a1d0-bd09f74c93ad"
/> |

They also look a bit nicer in the PR overview as well:
<img width="929" alt="image"
src="https://github.com/user-attachments/assets/04fc71fc-74b0-4e7c-9047-2aada664efef"
/>

The very last commit just filters out Windows and macOS tests again for
PRs (but they are executed on the `next` branch.

---

### Nest steps

I think for now we are in a pretty good state, but there are some things
we can do to further improve everything (mainly make things faster) but
aren't necessary. I also ran into issue while trying it so there is more
work to do.

1. More splits — instead of having a Vite folder and PostCSS folder, we
can go a step further and have folders for Next.js, Astro, Nuxt, Remix,
…
2. Caching — right now we have to run the build step for every OS on
every "job". We can re-use the work here by introducing a setup job that
the other jobs rely on. @thecrypticace and I tried it already, but were
running into some Bun specific Standalone CLI issues when doing that.
3. Remote caching — we could re-enable remote caching such that the
`build` step can be full turbo (e.g.: after a PR is merged in `next` and
we run everything again)

Author: RobinMalfait

.github/workflows/ci.yml

@@ -77,11 +77,6 @@ jobs:
       - name: Test
         run: pnpm run test
 
-      - name: Integration Tests
-        run: pnpm run test:integrations
-        env:
-          GITHUB_WORKSPACE: ${{ github.workspace }}
-
       - name: Install Playwright Browsers
         run: npx playwright install --with-deps
 

.github/workflows/integration-tests.yml

@@ -0,0 +1,102 @@
+name: Integration Tests
+
+on:
+  push:
+    branches: [next]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        node-version: [20]
+
+        runner:
+          - name: Windows
+            os: windows-latest
+
+          - name: Linux
+            os: namespace-profile-default
+
+          - name: macOS
+            os: macos-14
+
+        integration:
+          - upgrade
+          - vite
+          - cli
+          - postcss
+
+        # Exclude windows and macos from being built on feature branches
+        on-next-branch:
+          - ${{ github.ref == 'refs/heads/next' }}
+        exclude:
+          - on-next-branch: false
+            runner:
+              name: Windows
+          - on-next-branch: false
+            runner:
+              name: macOS
+
+    runs-on: ${{ matrix.runner.os }}
+    timeout-minutes: 30
+
+    name: ${{ matrix.runner.name }} /  ${{ matrix.integration }}
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: pnpm/action-setup@v4
+
+      - name: Use Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ matrix.node-version }}
+          cache: 'pnpm'
+
+      # Cargo already skips downloading dependencies if they already exist
+      - name: Cache cargo
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+            target/
+          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+
+      # Cache the `oxide` Rust build
+      - name: Cache oxide build
+        uses: actions/cache@v4
+        with:
+          path: |
+            ./target/
+            ./crates/node/*.node
+            ./crates/node/index.js
+            ./crates/node/index.d.ts
+          key: ${{ runner.os }}-oxide-${{ hashFiles('./crates/**/*') }}
+
+      - name: Install dependencies
+        run: pnpm install
+
+      - name: Build
+        run: pnpm run build
+        env:
+          CARGO_PROFILE_RELEASE_LTO: 'off'
+          CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER: 'lld-link'
+
+      - name: Test ${{ matrix.integration }}
+        run: pnpm run test:integrations ./integrations/${{ matrix.integration }}
+        env:
+          GITHUB_WORKSPACE: ${{ github.workspace }}
+
+      - name: Notify Discord
+        if: failure() && github.ref == 'refs/heads/next'
+        uses: discord-actions/message@v2
+        with:
+          webhookUrl: ${{ secrets.DISCORD_WEBHOOK_URL }}
+          message: 'The [most recent build](<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}>) on the `next` branch has failed.'

integrations/cli/config.test.ts

@@ -161,7 +161,10 @@ test(
     },
   },
   async ({ fs, spawn }) => {
-    await spawn('pnpm tailwindcss --input src/index.css --output dist/out.css --watch')
+    let process = await spawn(
+      'pnpm tailwindcss --input src/index.css --output dist/out.css --watch',
+    )
+    await process.onStderr((m) => m.includes('Done in'))
 
     await fs.expectFileToContain('dist/out.css', [
       //
@@ -214,7 +217,10 @@ test(
     },
   },
   async ({ fs, spawn }) => {
-    await spawn('pnpm tailwindcss --input src/index.css --output dist/out.css --watch')
+    let process = await spawn(
+      'pnpm tailwindcss --input src/index.css --output dist/out.css --watch',
+    )
+    await process.onStderr((m) => m.includes('Done in'))
 
     await fs.expectFileToContain('dist/out.css', [
       //
@@ -267,7 +273,10 @@ test(
     },
   },
   async ({ fs, spawn }) => {
-    await spawn('pnpm tailwindcss --input src/index.css --output dist/out.css --watch')
+    let process = await spawn(
+      'pnpm tailwindcss --input src/index.css --output dist/out.css --watch',
+    )
+    await process.onStderr((m) => m.includes('Done in'))
 
     await fs.expectFileToContain('dist/out.css', [
       //

integrations/cli/index.test.ts

@@ -1,7 +1,7 @@
 import dedent from 'dedent'
 import os from 'node:os'
 import path from 'node:path'
-import { describe, expect } from 'vitest'
+import { describe } from 'vitest'
 import { candidate, css, html, js, json, test, ts, yaml } from '../utils'
 
 const STANDALONE_BINARY = (() => {
@@ -156,9 +156,10 @@ describe.each([
       },
     },
     async ({ root, fs, spawn }) => {
-      await spawn(`${command} --input src/index.css --output dist/out.css --watch`, {
+      let process = await spawn(`${command} --input src/index.css --output dist/out.css --watch`, {
         cwd: path.join(root, 'project-a'),
       })
+      await process.onStderr((m) => m.includes('Done in'))
 
       await fs.expectFileToContain('project-a/dist/out.css', [
         candidate`underline`,
@@ -491,7 +492,7 @@ test(
       'pages/nested/foo.jsx': 'content-["pages/nested/foo.jsx"] content-["BAD"]',
     },
   },
-  async ({ fs, exec }) => {
+  async ({ fs, exec, expect }) => {
     await exec('pnpm tailwindcss --input index.css --output dist/out.css')
 
     expect(await fs.dumpFiles('./dist/*.css')).toMatchInlineSnapshot(`
@@ -722,7 +723,7 @@ test(
         ></div>`,
     },
   },
-  async ({ fs, exec, spawn, root }) => {
+  async ({ fs, exec, spawn, root, expect }) => {
     await exec('pnpm tailwindcss --input src/index.css --output dist/out.css', {
       cwd: path.join(root, 'project-a'),
     })
@@ -790,9 +791,13 @@ test(
     `)
 
     // Watch mode tests
-    await spawn('pnpm tailwindcss --input src/index.css --output dist/out.css --watch', {
-      cwd: path.join(root, 'project-a'),
-    })
+    let process = await spawn(
+      'pnpm tailwindcss --input src/index.css --output dist/out.css --watch',
+      {
+        cwd: path.join(root, 'project-a'),
+      },
+    )
+    await process.onStderr((m) => m.includes('Done in'))
 
     // Changes to project-a should not be included in the output, we changed the
     // base folder to project-b.
@@ -962,7 +967,7 @@ test(
       'pages/nested/foo.jsx': 'content-["pages/nested/foo.jsx"] content-["BAD"]',
     },
   },
-  async ({ fs, exec }) => {
+  async ({ fs, exec, expect }) => {
     await exec('pnpm tailwindcss --input index.css --output dist/out.css')
 
     expect(await fs.dumpFiles('./dist/*.css')).toMatchInlineSnapshot(`

integrations/package.json

@@ -4,7 +4,6 @@
   "private": true,
   "devDependencies": {
     "dedent": "1.5.3",
-    "fast-glob": "^3.3.2",
-    "kill-port": "^2.0.1"
+    "fast-glob": "^3.3.2"
   }
 }

integrations/postcss/config.test.ts

@@ -148,7 +148,8 @@ test(
     },
   },
   async ({ fs, spawn }) => {
-    await spawn('pnpm postcss src/index.css --output dist/out.css --watch --verbose')
+    let process = await spawn('pnpm postcss src/index.css --output dist/out.css --watch --verbose')
+    await process.onStderr((m) => m.includes('Waiting for file changes'))
 
     await fs.expectFileToContain('dist/out.css', [
       //
@@ -218,7 +219,8 @@ test(
     },
   },
   async ({ fs, spawn }) => {
-    await spawn('pnpm postcss src/index.css --output dist/out.css --watch --verbose')
+    let process = await spawn('pnpm postcss src/index.css --output dist/out.css --watch --verbose')
+    await process.onStderr((m) => m.includes('Waiting for file changes'))
 
     await fs.expectFileToContain('dist/out.css', [
       //

integrations/postcss/core-as-postcss-plugin.test.ts

@@ -1,4 +1,4 @@
-import { expect } from 'vitest'
+import { describe } from 'vitest'
 import { css, js, json, test } from '../utils'
 
 const variantConfig = {
@@ -29,9 +29,9 @@ const variantConfig = {
   },
 }
 
-for (let variant of Object.keys(variantConfig)) {
+describe.each(Object.keys(variantConfig))('%s', (variant) => {
   test(
-    `can not use \`tailwindcss\` as a postcss module (${variant})`,
+    `can not use \`tailwindcss\` as a postcss module`,
     {
       fs: {
         ...variantConfig[variant],
@@ -47,12 +47,12 @@ for (let variant of Object.keys(variantConfig)) {
         'src/index.css': css`@import 'tailwindcss';`,
       },
     },
-    async ({ exec }) => {
+    async ({ exec, expect }) => {
       expect(
         exec('pnpm postcss src/index.css --output dist/out.css', undefined, { ignoreStdErr: true }),
       ).rejects.toThrowError(
         `It looks like you're trying to use \`tailwindcss\` directly as a PostCSS plugin. The PostCSS plugin has moved to a separate package, so to continue using Tailwind CSS with PostCSS you'll need to install \`@tailwindcss/postcss\` and update your PostCSS configuration.`,
       )
     },
   )
-}
+})

integrations/postcss/index.test.ts

@@ -1,6 +1,5 @@
 import dedent from 'dedent'
 import path from 'node:path'
-import { expect } from 'vitest'
 import { candidate, css, html, js, json, test, ts, yaml } from '../utils'
 
 test(
@@ -724,7 +723,7 @@ test(
       'pages/nested/foo.jsx': 'content-["pages/nested/foo.jsx"] content-["BAD"]',
     },
   },
-  async ({ fs, exec }) => {
+  async ({ fs, exec, expect }) => {
     await exec('pnpm postcss index.css --output dist/out.css')
 
     expect(await fs.dumpFiles('./dist/*.css')).toMatchInlineSnapshot(`
@@ -954,7 +953,7 @@ test(
       `,
     },
   },
-  async ({ fs, exec, spawn, root }) => {
+  async ({ fs, exec, spawn, root, expect }) => {
     await exec('pnpm postcss src/index.css --output dist/out.css --verbose', {
       cwd: path.join(root, 'project-a'),
     })
@@ -1020,7 +1019,6 @@ test(
         cwd: path.join(root, 'project-a'),
       },
     )
-
     await process.onStderr((message) => message.includes('Waiting for file changes...'))
 
     // Changes to project-a should not be included in the output, we changed the
@@ -1215,7 +1213,7 @@ test(
       'pages/nested/foo.jsx': 'content-["pages/nested/foo.jsx"] content-["BAD"]',
     },
   },
-  async ({ fs, exec }) => {
+  async ({ fs, exec, expect }) => {
     await exec('pnpm postcss index.css --output dist/out.css')
 
     expect(await fs.dumpFiles('./dist/*.css')).toMatchInlineSnapshot(`

integrations/postcss/next.test.ts

@@ -1,4 +1,4 @@
-import { expect } from 'vitest'
+import { describe } from 'vitest'
 import { candidate, css, fetchStyles, js, json, retryAssertion, test } from '../utils'
 
 test(
@@ -56,7 +56,7 @@ test(
       `,
     },
   },
-  async ({ fs, exec }) => {
+  async ({ fs, exec, expect }) => {
     await exec('pnpm next build')
 
     let files = await fs.glob('.next/static/css/**/*.css')
@@ -70,9 +70,10 @@ test(
     ])
   },
 )
-;['turbo', 'webpack'].forEach((bundler) => {
+
+describe.each(['turbo', 'webpack'])('%s', (bundler) => {
   test(
-    `dev mode (${bundler})`,
+    'dev mode',
     {
       fs: {
         'package.json': json`
@@ -126,26 +127,35 @@ test(
         `,
       },
     },
-    async ({ fs, spawn, getFreePort }) => {
-      let port = await getFreePort()
-      await spawn(`pnpm next dev ${bundler === 'turbo' ? '--turbo' : ''} --port ${port}`)
+    async ({ fs, spawn, expect }) => {
+      let process = await spawn(`pnpm next dev ${bundler === 'turbo' ? '--turbo' : ''}`)
+
+      let url = ''
+      await process.onStdout((m) => {
+        let match = /Local:\s*(http.*)/.exec(m)
+        if (match) url = match[1]
+        return Boolean(url)
+      })
+
+      await process.onStdout((m) => m.includes('Ready in'))
 
       await retryAssertion(async () => {
-        let css = await fetchStyles(port)
+        let css = await fetchStyles(url)
         expect(css).toContain(candidate`underline`)
       })
 
-      await retryAssertion(async () => {
-        await fs.write(
-          'app/page.js',
-          js`
-            export default function Page() {
-              return <h1 className="underline text-red-500">Hello, Next.js!</h1>
-            }
-          `,
-        )
+      await fs.write(
+        'app/page.js',
+        js`
+          export default function Page() {
+            return <h1 className="underline text-red-500">Hello, Next.js!</h1>
+          }
+        `,
+      )
+      await process.onStdout((m) => m.includes('Compiled in'))
 
-        let css = await fetchStyles(port)
+      await retryAssertion(async () => {
+        let css = await fetchStyles(url)
         expect(css).toContain(candidate`underline`)
         expect(css).toContain(candidate`text-red-500`)
       })