This document is intended to explain how various technologies are used together to conduct automated UI testing for Android at Mozila. This document is accessible for all developers including those that are new to the Mozilla community.
Recall, that the primary problem we try to solve with automated testing is preventing unintended changes, which offten lead to bugs or poor user experiences. This problem is most common -- thus we benefit the most from throrough testing.
When a Pull Request or merge request is created, we want to ensure that the changes are tested before they are merged. This is done by running a series of UI tests on the code. These tests are ran on Firebase Test Lab on virtual devices.
Recall that on Pull Requests, the status badge on Github will let you know wether it is green, running, or broken (red).
But what is the flow of this process from creating a Pull Request to seeing UI test results?
For simplicity, the flow of UI test execution starts with Taskcluster:
Taskcluster is a generic task execution service. Mozilla uses Taskcluster as the continuous (CI) pipeline to build and release Firefox. At Mozilla, we use Taskcluster to build, test, and run code tools for all commits and Pull Requests.
For the sake of simplicity, when a Pull Request (or merge request) is created, a task is created on Taskcluster (FirefoxCI) that handles the building, signing, packaging and delivery of the Android application.
Subsequently, a handoff to Firebase Test Lab occurs to provide an environment of UI test execution.
For a more detailed document on Taskcluster see here.
We use Firebase Test Lab to conduct automated on-device testing. Firebase Test Lab is a cloud-based testing platform that allows us to test our apps and websites in a way that is similar to how we would test them on a physical device.
For more information on physical device testing see here.
Executable Espresso and UI Automator UI tests run on virtual devices by default and the test suite will run automatically on all Pull Requests and merges on the main branch. Test results and test artifacts are immediately available after test execution.
Espresso and UI Automator are Google's core UI testing libraries on Android: they provide APIs to select views, perform actions on them, and assert state without reaching into implementation details.
For a more detailed document on testing styles and technologies used by the Android teams at Mozilla see here
Treeherder is a reporting dashboard for Mozilla check-ins. It allows users to see result of automatic builds and their respective tests. A common use case is looking at a particular check-in and pulling up test logs and artifacts from the UI test job.
For all Android applications, UI tests are classfied under the ui-test-x86 job symbol.
Below is a very simplified flow diagram that demonstrates the hand-off from a Github event (e.g, Pull Request) to Taskcluster and Firebase Test Lab, and lastly back to Taskcluster and Treeherder.
Pull Request
or Merge
│
│
┌───────▼───────┐ ┌────────────┐
│ │ │ │
┌─┬──┤ Taskcluster ├───────────────► Firebase │
│ │ │ │ │ │
│ │ └───────▲───────┘ └──────┬─────┘
│ │ │ │
│ │ └──────────────────────────────┘
│ │
│ │ ┌──────────────┐
│ │ │ │
│ └──────────────────┤ Treeherder │
│ │ │
│ └──────────────┘
│
│ ┌──────────┐
│ │ │
└──────────────────────┤ Github │
│ │
└──────────┘