Skip to content

Commit 9088411

Browse files
si-netbahrimootazthiagosaucelabs
authored
Document Flutter ios testing support (#3066)
* Document the flutter integration testing for ios and android - Beta * Change xctest documentation, to be less of a step by step guide and instead provide more context * document when to stop following flutters integration guide * Improve readability. * More readability improvements for xctest * Better contents and typo fixes * More typos and correct usage of :::info fields * resolve github action checks * Apply suggestions from code review Co-authored-by: bahrimootaz <[email protected]> * Document native /test API * remove unwanted semi colon * Remove section on tear down method * Apply suggestions from code review * Point to the correct jobs api * MR comments * bold for readability * Combine sections on generating app and xctestrun files * fix link * Spelling Co-authored-by: Thiago Veronese <[email protected]> * Adding a note about the required saucectl version * Shorten the description for saucectl version requirements --------- Co-authored-by: Mootaz Bahri <[email protected]> Co-authored-by: bahrimootaz <[email protected]> Co-authored-by: Thiago Veronese <[email protected]>
1 parent d2a3bfe commit 9088411

File tree

5 files changed

+267
-12
lines changed

5 files changed

+267
-12
lines changed

‎docs/dev/api/rdc.md‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1014,6 +1014,156 @@ curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \
10141014

10151015
---
10161016

1017+
### Start a XCTest, XCUITest or Espresso Job
1018+
1019+
<details>
1020+
<summary><span className="api post">POST</span> <code>/v1/rdc/native-composer/tests</code></summary>
1021+
<p/>
1022+
1023+
Start a XCTest.
1024+
1025+
#### Parameters
1026+
1027+
<table id="table-api">
1028+
<tbody>
1029+
<tr>
1030+
<td><code>test_framework</code></td>
1031+
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The type of 'native' test you want to run, either 'XCUITEST', 'XCTEST', or 'ANDROID_INSTRUMENTATION' (Espresso).</p></td>
1032+
</tr>
1033+
</tbody>
1034+
<tbody>
1035+
<tr>
1036+
<td><code>app_id</code></td>
1037+
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The id of the app you want to test. The app must be uploaded to Sauce Labs AppStorage. Format: `storage://filename=AnApp.ipa`, or `storage://{file_id}`, or just `{file_id}`.</p></td>
1038+
</tr>
1039+
</tbody>
1040+
<tbody>
1041+
<tr>
1042+
<td><code>xc_test_run_file</code></td>
1043+
<td><p><small>| BODY | REQUIRED | STRING |</small></p><p>The id of the `xctestrun` file (the xctest config). The file must be uploaded to Sauce Labs AppStorage. Format: `storage://filename=Runner.xctestrun`, or `storage://{file_id}`, or just `{file_id}`.</p></td>
1044+
</tr>
1045+
</tbody>
1046+
<tbody>
1047+
<tr>
1048+
<td><code>device_query</code></td>
1049+
<td><p><small>| BODY | REQUIRED | OBJECT |</small></p><p>Defines on which device you want to run you test. The available attributes are:<ul><li><code>type</code> - <small>String of 'DynamicDeviceQuery'</small></li><li><code>device_name</code> - <small>String - regular expression to select a device.</small></li><li><code>os_version</code> - <small>String - regular expression to select a device version.</small></li></ul></p><p>The <code>type</code> parameter is required..</p></td>
1050+
</tr>
1051+
</tbody>
1052+
<tbody>
1053+
<tr>
1054+
<td><code>tunnel_name</code></td>
1055+
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign a sauce connect tunnel to the job.</p></td>
1056+
</tr>
1057+
</tbody>
1058+
<tbody>
1059+
<tr>
1060+
<td><code>tunnel_owner</code></td>
1061+
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Define the name of the tunnel owner of the sauce connect tunnel you want to use.</p></td>
1062+
</tr>
1063+
</tbody>
1064+
<tbody>
1065+
<tr>
1066+
<td><code>settings_overwrite</code></td>
1067+
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Define instrumentations settings you want to apply to your app.</p></td>
1068+
</tr>
1069+
</tbody>
1070+
<tbody>
1071+
<tr>
1072+
<td><code>test_name</code></td>
1073+
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign a name to the job, to be displayed in the UI.</p></td>
1074+
</tr>
1075+
</tbody>
1076+
<tbody>
1077+
<tr>
1078+
<td><code>build</code></td>
1079+
<td><p><small>| BODY | OPTIONAL | STRING |</small></p><p>Assign the job to a build. You can specify an existing build name or create a new one.</p></td>
1080+
</tr>
1081+
</tbody>
1082+
<tbody>
1083+
<tr>
1084+
<td><code>tags</code></td>
1085+
<td><p><small>| BODY | OPTIONAL | ARRAY |</small></p><p>The set of tags to apply to the job.</p></td>
1086+
</tr>
1087+
</tbody>
1088+
</table>
1089+
1090+
<Tabs
1091+
groupId="dc-url"
1092+
defaultValue="us"
1093+
values={[
1094+
{label: 'United States', value: 'us'},
1095+
{label: 'Europe', value: 'eu'},
1096+
]}>
1097+
1098+
<TabItem value="us">
1099+
1100+
```jsx title="Sample Request"
1101+
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \
1102+
--request POST 'https://api.us-west-1.saucelabs.com/v1/rdc/native-composer/tests' \
1103+
--header 'Content-Type: application/json'
1104+
--data-raw '{
1105+
"test_framework": "XCTEST",
1106+
"test_name": "Your XCTest POC",
1107+
"app_id": "9349d683-5d26-46eb-a943-267cbe8b4deb",
1108+
"xc_test_run_file": "37eef454-a7f8-4a6d-a340-d39865ad6db3",
1109+
"device_query": {
1110+
"type": "DynamicDeviceQuery"
1111+
}
1112+
}'
1113+
```
1114+
1115+
</TabItem>
1116+
1117+
<TabItem value="eu">
1118+
1119+
```jsx title="Sample Request"
1120+
curl -u "$SAUCE_USERNAME:$SAUCE_ACCESS_KEY" --location \
1121+
--request POST 'https://api.eu-central-1.saucelabs.com/v1/rdc/native-composer/tests' \
1122+
--header 'Content-Type: application/json'
1123+
--data-raw '{
1124+
"test_framework": "XCTEST",
1125+
"test_name": "Your XCTest POC",
1126+
"app_id": "9349d683-5d26-46eb-a943-267cbe8b4deb",
1127+
"xc_test_run_file": "37eef454-a7f8-4a6d-a340-d39865ad6db3",
1128+
"device_query": {
1129+
"type": "DynamicDeviceQuery"
1130+
}
1131+
}'
1132+
```
1133+
1134+
</TabItem>
1135+
</Tabs>
1136+
1137+
#### Responses
1138+
1139+
<table id="table-api">
1140+
<tbody>
1141+
<tr>
1142+
<td><code>200</code></td>
1143+
<td colSpan='2'>Success.</td>
1144+
</tr>
1145+
</tbody>
1146+
<tbody>
1147+
<tr>
1148+
<td><code>400</code></td>
1149+
<td colSpan='2'>Bad Request.</td>
1150+
</tr>
1151+
</tbody>
1152+
</table>
1153+
1154+
```jsx title="Sample Response"
1155+
{
1156+
"test_report": {
1157+
"id": "ef6058735d8d4dfa9e0077d250757aac",
1158+
"url": "https://api.eu-central-1.saucelabs.com/tests/ef6058735d8d4dfa9e0077d250757aac"
1159+
}
1160+
}
1161+
```
1162+
1163+
</details>
1164+
1165+
---
1166+
10171167
### Stop a Job
10181168

10191169
<details>

‎docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-android.md‎

Lines changed: 3 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t
7272
5. Create a directory called `integration_test` in the root of your Flutter project.
7373
6. Create a file called `flutter_integration_test.dart` in the `integration_test` directory.
7474
7. Update your testing dart file `flutter_integration_test.dart` to include the ***tearDownAll***,
75-
The purpose for this is to make sure we close the connection to the driver after the tests have completed.
75+
The purpose for this is to make sure we close the connection to the device after the tests have completed.
7676
```dart
7777
import 'package:flutter/material.dart';
7878
import 'package:flutter_test/flutter_test.dart';
@@ -118,7 +118,7 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t
118118

119119
9. Configure `saucectl` to run the test.
120120
* Create a folder `saucectl` in your project root directory.
121-
* Inside this folder create a `flutter_integration_test.yaml` with the following content:
121+
* Inside this folder create a `flutter_integration_test_android.yaml` with the following content:
122122
```yaml
123123
apiVersion: v1alpha
124124
kind: espresso
@@ -207,12 +207,4 @@ Follow this guide to run [integration tests](https://docs.flutter.dev/cookbook/t
207207
## Example Implementation
208208

209209
For a practical example of how to set up and run integration tests for Flutter apps, you can refer to the [Sauce Labs Flutter demo application](https://github.com/saucelabs/my-demo-app-flutter) repository.
210-
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how everything is configured and run your tests accordingly.
211-
212-
## What's Next
213-
214-
:::info Next step
215-
216-
We're excited to share that Sauce Labs is actively working on expanding support for Flutter integration tests on iOS.
217-
Stay tuned for updates as we continue to develop this capability!
218-
:::
210+
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how everything is configured and run your tests accordingly.

‎docs/mobile-apps/automated-testing/flutter/flutter-integration-testing-ios.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
id: flutter-integration-testing-ios
3+
title: Flutter iOS
4+
sidebar_label: Flutter iOS
5+
description: Run your Flutter integration tests for iOS
6+
---
7+
8+
import useBaseUrl from '@docusaurus/useBaseUrl';
9+
import Tabs from '@theme/Tabs';
10+
import TabItem from '@theme/TabItem';
11+
12+
Flutter compiles iOS [integration tests](https://docs.flutter.dev/cookbook/testing/integration/introduction) into [XCTests](https://developer.apple.com/documentation/xctest) so that they can be executed on Apple devices. The following will explain how to run your XCTests on Sauce Labs infrastructure.
13+
14+
**To run an XCTest (or 'Flutter test') on Sauce Labs, you need to provide two test artifacts:**
15+
1. Your flutter-iOS app compiled as an `.app` or `.ipa` file.
16+
2. The `.xctestrun` file for that app. The [.xctestrun file](https://keith.github.io/xcode-man-pages/xcodebuild.xctestrun.5.html) is the config for your test, this is the same config that Xcode uses when it runs your tests on your development machine.
17+
18+
19+
## Contents
20+
1. [How to build the '.app' and '.xctestrun' files for your Flutter app.](#1-how-to-build-the-app-and-xctestrun-files-for-your-flutter-app)
21+
2. [How to run the flutter-iOS integration test on Sauce Labs infrastructure.](#2-how-to-run-flutter-ios-integration-tests-on-sauce-labs-infrastructure)
22+
3. [Sample Implementation](#example-implementation)
23+
24+
25+
:::info What You'll Need
26+
27+
- A Sauce Labs account ([Log in](https://accounts.saucelabs.com/am/XUI/#login/) or sign up for
28+
a [free trial license](https://saucelabs.com/sign-up))
29+
- Your Sauce Labs [Username and Access Key](https://app.saucelabs.com/user-settings)
30+
- Access to Sauce Labs Real Devices. Sauce Labs only supports XCTests on Real Devices, not virtual.
31+
- Flutter mobile app. If you don't have one, you could use our Flutter Demo App:
32+
- [Sauce Labs Flutter Demo App](https://github.com/saucelabs/my-demo-app-flutter)
33+
- `xcodebuild` tools
34+
- `zip` and/or `saucectl`
35+
:::
36+
37+
38+
## 1. How to build the '.app' and '.xctestrun' files for your Flutter app.
39+
40+
:::note You need to setup your Flutter app for integration tests.
41+
42+
Before you build your app, you must ensure that you correctly set up the `integration_tests` for your flutter-ios app. You can follow the [flutter documentation](https://github.com/flutter/flutter/tree/main/packages/integration_test#integration_test) to do so. The most relevant section is the part on [iOS Device Testing](https://github.com/flutter/flutter/tree/main/packages/integration_test#ios-device-testing). You can stop following Flutter's guide after you have executed the `xcodebuild build-for-testing` command. This command will generate the `.app` and `.xctestrun` files.
43+
:::
44+
45+
To execute your xctest, we require your app (which must be packaged together with your XCTests) in '.app' or '.ipa' format. Additionally we need your your `.xctestrun` file, which is the config for your test.
46+
47+
By default, Xcode will not persist the `.xctestrun` file if you kick off an XCTest on your development machine. To persist the `.xctestrun` file we need to use the `xcodebuild build-for-testing` command. Make sure you are using the correct `scheme` so it includes your integration tests.
48+
49+
```shell
50+
# Example of the xcodebuild command to build the application.
51+
# You will need to adjust the args according to your app.
52+
output="../build/ios_integ"
53+
xcodebuild build-for-testing \
54+
-workspace Runner.xcworkspace \
55+
-scheme Runner \
56+
-xcconfig Flutter/Release.xcconfig \
57+
-configuration Release \
58+
-derivedDataPath \
59+
$output -sdk iphoneos
60+
61+
# The .app and .xctestrun files will now be present in your output directory. In this case: `build/ios_integ/Products/Release-iphoneos`
62+
```
63+
64+
65+
## 2. How to run flutter-iOS integration tests on Sauce Labs infrastructure
66+
67+
To run your Flutter XCTest on Sauce Labs, you have two options: use `saucectl` or integrate with our APIs yourself. If you are unfamiliar with our APIs, we recommend using `saucectl` for ease of use and getting you started quickly.
68+
69+
70+
### Run XCTests via saucectl
71+
72+
First install [saucectl](/docs/dev/cli/saucectl.md#installing-saucectl). **Note: minimum version is `0.192.0`.** Then you can use `saucectl` command to configure and run your test on Sauce Labs infrastructure.
73+
74+
```shell
75+
# If it's the first time you're using saucectl, run:
76+
saucectl configure
77+
78+
# follow the steps to configure your XCTest, with your `.app`/`.ipa` file and the `.xctestrun` test config. Use `Real Device` not `Virtual Device`
79+
saucectl init xctest
80+
81+
# run the newly created XCTest config.
82+
saucectl run
83+
```
84+
85+
For further configuration options and info on how to use `saucectl` visit [/docs/mobile-apps/automated-testing/espresso-xcuitest/xcuitest.md](/docs/mobile-apps/automated-testing/espresso-xcuitest/xcuitest.md)
86+
87+
### Run XCTests without saucectl
88+
89+
If you prefer not to use saucectl, you can directly integrate with our APIs.
90+
91+
**First**, compile your `.app` as an `.ipa` file as described [above](/docs/mobile-apps/automated-testing/ipa-files.md#building-an-ipa-from-an-app-bundle).
92+
93+
**Second**, upload your `.ipa` and `.xctestrun` files to our AppStorage backend, see [AppStorage APIs](/docs/mobile-apps/app-storage.md#upload-apps-via-rest-api).
94+
95+
**Third**, call our native testing API with the AppStorage IDs of the two files you just uploaded. See [RDC native /test API](/docs/dev/api/rdc.md#start-a-xctest-xcuitest-or-espresso-job).
96+
97+
**Fourth**, poll the state of the job and wait until the `status` is `passed|failed|error|complete`. You can do this through the [Jobs API](/docs/dev/api/rdc.md#get-a-specific-real-device-job).
98+
99+
100+
## Example Implementation
101+
102+
For a practical example of how to set up and run integration tests for Flutter apps, you can refer to
103+
the [Sauce Labs Flutter demo application](https://github.com/saucelabs/my-demo-app-flutter) repository.
104+
The steps outlined in this guide have already been implemented in that repository. You can follow along with the demo app to see how
105+
everything is configured and run your tests accordingly.

‎docs/mobile-apps/automated-testing/ipa-files.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,13 @@ In order to disable the resigning process, you must buy your own private devices
3535
3. Compress the `Payload` directory into an archive (.zip file) and give it a new name with .ipa appended to the end of the file name.
3636
4. Your `.ipa` file is now ready for upload to Sauce Labs.
3737

38+
```shell
39+
# example for building an '.ipa' file out of an '.app'
40+
mkdir Payload
41+
cp -r PATH_TO_BUILD_FOLDER/Runner.app Payload
42+
zip -r Runner.ipa Payload
43+
```
44+
3845
### Building an .ipa File
3946

4047
You can use any of the existing methods of distribution for your iOS app, except for the **App Store** type. This means that you can choose any of the three other export methods: **Ad Hoc**, **Enterprise**, or **Development**.

‎sidebars.js‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1438,6 +1438,7 @@ module.exports = {
14381438
items: [
14391439
'mobile-apps/automated-testing/flutter',
14401440
'mobile-apps/automated-testing/flutter/flutter-integration-testing-android',
1441+
'mobile-apps/automated-testing/flutter/flutter-integration-testing-ios',
14411442
],
14421443
},
14431444
'mobile-apps/automated-testing/ipa-files',
@@ -1728,7 +1729,7 @@ module.exports = {
17281729
'visual-testing/integrations/python',
17291730
'visual-testing/integrations/python-robot-framework',
17301731
'visual-testing/integrations/playwright',
1731-
'visual-testing/integrations/espresso'
1732+
'visual-testing/integrations/espresso',
17321733
],
17331734
},
17341735
'visual-testing/cli',

0 commit comments

Comments
 (0)