Adding E2E Tests

[swissspidy]

Now that we have the E2E testing infrastructure set up (see #2696), we should look into how we can best leverage that to further improve code quality and catch bugs in the AMP Stories editor.

While we of course can add E2E tests for all aspects of the AMP plugin, we should focus on AMP Stories first. Still, I am mentioning some of the other areas in the plugin here.

#2779 is related, but I'd want to focus on E2E tests first instead of the unit tests, as I think here we have the bigger opportunity to solidify the AMP Stories editor.

Ideally, going forward we add E2E tests whenever we work on an area that is a good fit for such a test or when we introduce a new feature. Then, later we can go through pre-existing areas to add them.

Testable Areas

The following list is not necessarily complete, but should give a good idea about what we can and should test.

• Block Editor
  • Reader Mode
    • Verify that customizer controls and toggles are visible and working
  • Transitional Mode
    • Verify that AMP blocks are not available in the editor
  • Standard Mode
    • Verify that AMP blocks are available in the editor
  • Block validation
    • Verify that block validation happens when AMP is enabled for the current post.
    • Verify that block validation is not happening when AMP is disabled for the current post.
• Settings
  • Verify visibility of post type selection area depending on whether Website Mode is active.
  • Verify that Story Export URL is only visible when Stories experience is toggled
• Validation Screen
  • Verify status changes
  • Verify alter when there are unsaved changes
• Integrations
  • PWA (there is nothing user-facing yet)
  • Site Kit (maybe in the future if not already done in Site Kit)

Infrastructure Changes

• Create additional helper plugins that we might need during E2E tests, and set this up accordingly.
  • One example could be a plugin that makes sure pointers can not be dismissed permanently, makes testing of these easier
• Possibly use smarter APIs like expect(instance).toMatchElement(selector[, options]) instead of XPath -> E2E Tests: Avoid using XPath #2888.
• Possibly also run E2E tests against an older WordPress version

[miina]

FI: I'm adding some CTA-related e2e tests within #2824.

[westonruter]

For testing mobile redirection, see #5268.