Investigate and Fix Streaming Flaky Tests in ReactOnRailsPro Gem Integration Suite

[AbanoubGhadban]

Tasks

• Create a list of flaky tests
• Reproduce the test failures locally or connect to CI over SSH to get the failure logs:
  • It seems that the error happens when multiple tests are running in parallel
    The problem occurs on a page containing a router with multiple streamed components. An error occurs when the user navigates from the first page to the second.
  • The testing approach is already flaky, but no error should be thrown at the image, so I'm trying to find the cause of that JS error that happens.
    By testing, it seems that the JS error messages occur when a page is streamed using our custom navigation navigate_with_streaming helper function. When the page is opened using the visit helper function, no JS error is shown.
  • We already planned ot replace our navigate_with_streaming helper function with a more robust tool that tests streaming behavior robustly.
  • Now, we can be sure that any memory leakage from ReactOnRails does not cause the problem. So, it's OK to ignore the issue at the streaming helper as it will be replaced during this PR.
• Investigate standard ways in Rails to create end-to-end tests for streamed pages
  • Found that Playwright is a suitable framework to write end-to-end tests for our use case. It gives more control over asserting the page at any time before the page is fully loaded, has many debugging tools, and can test its compatibility with different browsers.
  • Playwright gem in Ruby is just a wrapper around the Playwright npm package. Also, writing tests at JS makes testing of streaming behavior easier (as it's easier to write async code at JS).
  • Let's write E2E tests using Playwright in JavaScript.
• If Rails doesn't have good streaming testing tools, try to use other JavaScript tools
  • Even after writing Playwright tests in JS, the tests are still flaky.
  • It's caused by the timing approach used for testing. For example, a chunk can be delayed under high load and merged into the next chunk.
  • We need to implement an async component that can be controlled from the test.
  • So, we are going to write a component that waits for data sent from the test over Redis.
  • Let's use Playwright fixture to write a fixture for every page we need to fix

Flaky Tests List

• https://app.circleci.com/insights/github/shakacode/react_on_rails/workflows/build-and-test/tests?branch=master

Environment

1. Ruby version:
2. Rails version:
3. Shakapacker/Webpacker version:
4. React on Rails version:

Expected behavior

Actual behavior

Small, reproducible repo