[Bug]: toHaveScreenshot sometimes not showing actual/diff on error for webkit

[dimkin-eu]

Version

1.50.0

Steps to reproduce

I understand this is a very vague description, but I can't find any consistent pattern.
It happens for 1-2 tests out of 2K+, but this fail the whole job :(
The only idea that comes to my mind - the page wasn't stable to assert a screenshot, but animations are disabled by our code - anyway some actual screenshot must be there

    await expect(page).toHaveScreenshot(`${filename}.png`, {
        scale: "css",
        caret: "hide",
        animations: "allow",
        maxDiffPixels,
    })

Expected behavior

on error 3 images produced

Actual behavior

only expected image in stacktrace/report

    Error: Timed out 15000ms waiting for expect(page).toHaveScreenshot(expected)
      Timeout 15000ms exceeded.
    Expected: /builds/frontend/foobar.png
    Call log:
      - expect.toHaveScreenshot(foobar.png) with timeout 15000ms
        - verifying given screenshot expectation
      - taking page screenshot
      - waiting for fonts to load...
      - fonts loaded
      - Timeout 15000ms exceeded.
     

Additional context

No response

Environment

FROM mcr.microsoft.com/playwright:v1.50.0-jammy
pnpm 9.15.2

[agg23]

Please provide a complete and minimal repro of this issue. Otherwise we cannot diagnose it.

[dgozman]

@dimkin-eu If there is no actual screenshot, that means screenshot action did timeout without being able to take a single screenshot. If your environment is slow, perhaps increasing a timeout would help. Otherwise, that could be a bug where screenshot just hangs, but we would need a repro to be able to fix it.

[dimkin-eu]

@dgozman env is ok, trace is recording ( and saved ), parallelly tests were running, test before and after is passing for same device.
I understand, that repro is needed, but I have no idea about causes :( Will try to dive more.

Same happens in 1.50.1 :(

[dimkin-eu]

JFYI added some debug logs

pw:api => expect.toHaveScreenshot(AR-163_Story.png) started +17ms
pw:api expect.toHaveScreenshot(AR-163_Story.png) with timeout 15000ms +15ms
pw:api   verifying given screenshot expectation +1ms
pw:api taking page screenshot +0ms
pw:api waiting for fonts to load... +13ms
pw:api fonts loaded +70ms
pw:api failed to take screenshot -  +17s
pw:api <= expect.toHaveScreenshot(AR-163_Story.png) succeeded +2ms

what I can see from the trace - page in the timeline is alive, but in step's snapshot is blanc
Also ( from recent-ish changes ) page contain canvases

[dgozman]

Re: canvas - the recent changes were about canvas in tracing. If you disable tracing, does the problem still persist? If so, perhaps running with DEBUG=pw:protocol could tell me something, but hopes are low.

[dimkin-eu]

Yes, turning off trace doesn't fail tests anymore. Previously it was retain-on-failure

DEBUG=pw:protocol adds so many logs, so CI sends the rest to /dev/null

Job's log exceeded limit of 10240000 bytes.
Job execution will continue but no more output will be collected.

[dimkin-eu]

Maybe this could give a hint

Not exactly my bug, but some common things - canvas shown in timeline, but not shown in snapshot

    projects: [
        {
            name: "iPhone 11",
            use: {
                ...devices["iPhone 11"],
            },
        }
    ],

import {expect, test} from '@playwright/test'

test.describe("describe", () => {
    test("short storybook spec", async ({page}) => {
        await page.goto("https://zimjs.com/parallax.html")
        await expect(page).toHaveScreenshot(`foobar.png`, {
            scale: "css",
            caret: "hide",
            animations: "allow",
        })
    })
})

trace.zip