CI: Add Roborazzi Screenshot comparison GitHub Action

[david-allison]

Note

Assisted-by: Claude Opus 4.7

Purpose / Description

Brayan wanted to get our Roborazzi screenshot tests running in CI, so a user was informed explicitly if their code changes changed the app visuals.

This was primarily for his theming/Material You improvements, but this affects all users positively, allowing us to automate regression testing, or allowing a user to add a test to produce up-to-date screenshots exposing a new feature across all themes.

Fixes

• Fixes CI: Add Roborazzi Screenshot comparison GitHub Action #20942

Approach

Follow the Roborazzi guide, then make changes

This introduces 3 files:

• screenshot_store.yml: store a baseline in an artifact for commits to main
• screenshot_compare.yml: on a PR branch, build a snapshot and compare it with main
• screenshot_comment.yml: when screenshot_compare.yml completes, add a comment to the PR listing changes, OR updates the comment if changes are resolved

Major Changes

• For security, instead of creating a branch in ankidroid/Anki-Android, reference the artifact
  • We lose inline screenshots
• Action naming: map to this repo
• screenshot-diff artifact is now more useable
  • Remove a deeply nested structure
  • Name the extracted folder: screenshot-diff-pr-N
    • Contains 1 folder per test class
      • 3 images per failing test: x.png x_actual.png x_compare.png
• Limit the comment to ensure we don't go over the 65k comment size limit

How Has This Been Tested?

• Roborazzi: test happy path - no comment david-allison/Anki-Android#52
• Roborazzi: Test screenshot change - comment david-allison/Anki-Android#53
• Roborazzi: Rest screenshot change - comment then resolve david-allison/Anki-Android#54
• Roborazzi: Test screenshot change - add new test david-allison/Anki-Android#55
  • We don't yet have any specific 'new test' code, it's shown as a break will a 'null' baseline - seems acceptable

Checklist

• You have a descriptive commit message with a short title (first line, max 50 chars).
• You have commented your code, particularly in hard-to-understand areas
• You have performed a self-review of your own code
• UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
• UI Changes: You have tested your change using the Google Accessibility Scanner

[BrayanDSO]

GitHub is failing and I'm currently not able to do an actual review.

In screenshot_store.yml:

we will need to force screenshot tests to follow the ScreenshotTest naming style.

Also in paths:, doesn't **/ScreenshotTest.kt only match files named exactly ScreenshotTest.kt?

[david-allison]

Also in paths:, doesn't **/ScreenshotTest.kt only match files named exactly ScreenshotTest.kt

I fear I wasn't clear. Could I have suggestions for a better comment.

store only runs on main.

If it runs on a PR branch (🛠️ Screenshots: Store / test), it only runs to ensure that we haven't broken the store mechanism, therefore we test on the gradle files, dependencies, and the base class of all Screenshot tests (this defines the category, and is the most likely to cause 'store' issues).

For the individual tests, these are run in compare

[BrayanDSO]

made the comment too quickly. LGTM

[ericli3690]

Looks fairly good to me overall. I've never really worked with yaml GitHub Actions code before, though, so maybe one more reviewer should take a skim. I see no issues. Should be fairly safe as it touches no production code and mostly serves to unblock two of David's PRs + Brayan's work + some testing stuff for me.

Will move this to pending merge for now. David, feel free to slap it in if you think I'm being overly cautious.