Skip to content

Commit 65fdd25

Browse files
authored
docs(ai): Adds --analyze flag docs for TestCommand and update Ai Configurations (#110)
* docs(analyze): Adds new --analyze flag docs and assets * docs(analyze): Removes unnecessary note * Update cli/test-suites-and-reports.md * docs(ai): Adjuts ai configuration to robin cloud
1 parent b1ce863 commit 65fdd25

File tree

3 files changed

+81
-25
lines changed

3 files changed

+81
-25
lines changed

Diff for: .gitbook/assets/analyze-report.png

393 KB
Loading

Diff for: api-reference/configuration/ai-configuration.md

+6-25
Original file line numberDiff line numberDiff line change
@@ -16,37 +16,18 @@ commands, additional configuration is required.
1616

1717
The default model is the latest GPT-4o.
1818

19-
You can configure the model to use with the `MAESTRO_CLI_AI_MODEL` env var, for example:
20-
21-
```console
22-
export MAESTRO_CLI_AI_MODEL=claude-3-5-sonnet-20240620
23-
```
24-
25-
Currently supported:
26-
27-
- GPT family of models from OpenAI
28-
- Claude family of models from Anthropic
29-
3019
Support for more models and providers is tracked [in this issue](https://github.com/mobile-dev-inc/maestro/issues/1957).
3120

3221
### API key
3322

34-
To use this command, an API key for the LLM service is required. To set it,
35-
export the `MAESTRO_CLI_AI_KEY` env var.
23+
{% hint style="success" %}
24+
To use these features, you must have an account on [Robin](https://www.robintest.com), the official Maestro cloud platform. Make sure to [create an account](https://signin.robintest.com/sign-up) and [**retrieve your API key**](MAESTRO_CLOUD_API_KEY) to set it up correctly by following this link:
3625

37-
For example, to set the key for OpenAI:
38-
39-
```console
40-
export MAESTRO_CLI_AI_KEY=sk-4NXxdLXY4H9DZW0Vpf4lT3HuBaFJoz1zoL21eLoLRKlyXd69
41-
```
26+
{% endhint %}
4227

43-
or for Anthropic:
28+
To use this command, an API key for the LLM is required. With the latest changes, we have migrated hard-coded API calls and prompts to our cloud backend, allowing us to enhance these features dynamically without requiring new CLI releases. To set it,
29+
export the `MAESTRO_CLOUD_API_KEY` env var.
4430

4531
```console
46-
export MAESTRO_CLI_AI_KEY=sk-ant-api03-U9vWi8GDrxRAvA2RL2RMCImYCQr8BFCbNOq2woeRXLNz2Iy4PbY1X2137leSm92mitI7F9IwxKIrXtXgTIzj7A-2AvgbwAA
32+
export MAESTRO_CLOUD_API_KEY=rb_qikYa7g2y5LcDEwLaEDz258ykjRQW7tIaR9K8jLlz6ijqCLTNfnDla3Nd17JF2ealh8kcsYHYyg35M41obGaz85VJz4uqI1orj
4733
```
48-
49-
{% hint style="info" %}
50-
**You need to bring your own API key.** The keys above are just examples and are
51-
not valid.
52-
{% endhint %}

Diff for: cli/test-suites-and-reports.md

+75
Original file line numberDiff line numberDiff line change
@@ -112,3 +112,78 @@ Note that your Flows should **not** depend on device state and should be treated
112112
#### Configuring part of the Flows to run sequentially
113113

114114
For instance, if you have three Flows, `flowA`, `flowB`, and `flowC`, but you want to run only `flowA` and `flowB` sequentially, don't add `flowC` and `flowD` to the list. Maestro will run these Flows in non-deterministic ordering **after** the Flow sequence has finished executing.
115+
116+
#### Analyze
117+
118+
{% hint style="warning" %}
119+
120+
This is an **experimental** feature powered by LLM technology. All feedback is
121+
welcome.
122+
123+
{% endhint %}
124+
125+
Maestro introduces a new feature that leverages AI to analyze your end-to-end (E2E) mobile tests and provide actionable insights based on your test logs, commands, and screenshots captured during your test runs. The AI-powered analysis identifies potential issues in your app's functionality, UI, and internationalization, helping you improve app quality efficiently.
126+
127+
**Login Requirement:** Before you use the AI analysis feature, ensure you are logged into Maestro. Run the following command:
128+
129+
```bash
130+
maestro login
131+
```
132+
133+
**Analyzing your tests**: To analyze your test flows with AI, use the --analyze flag with the maestro test command:
134+
135+
```bash
136+
maestro test flow-file.yaml --analyze
137+
```
138+
139+
{% hint style="info" %}
140+
141+
While we aim for precision, please note that this is a beta release, and the results should be validated before making critical decision
142+
143+
{% endhint %}
144+
145+
This will enable AI analysis and provide a detailed report based on your test artifacts:
146+
147+
##### Examples
148+
149+
Successful Analysis
150+
151+
Command:
152+
153+
```bash
154+
maestro test flow-file.yaml --analyze
155+
```
156+
157+
Output:
158+
159+
```bash
160+
🔎 Analyzing Flow(s)...
161+
162+
To view the report, open the following link in your browser:
163+
file:///path/to/your/insights-report.html
164+
165+
Analyze support is in Beta. We would appreciate your feedback in our Slack channel: #community-chat
166+
167+
```
168+
169+
<figure><img src="../.gitbook/assets/analyze-report.png" alt=""><figcaption></figcaption></figure>
170+
171+
No Issues Found
172+
173+
If no issues are detected, you will see a message like this:
174+
175+
```bash
176+
Hey, we analyzed your flow for spelling, grammar, and internationalization issues, and good news 🙌 we didn't find any issues!
177+
```
178+
179+
##### Disabling Notifications
180+
181+
To disable the AI analysis notification, set the `MAESTRO_CLI_ANALYSIS_NOTIFICATION_DISABLED` environment variable to true before running Maestro:
182+
183+
```bash
184+
export MAESTRO_CLI_ANALYSIS_NOTIFICATION_DISABLED=true
185+
```
186+
187+
##### Feedback
188+
189+
The Analyze feature is currently in Beta. Share your feedback and suggestions in our Slack channel:[#community-chat](https://mobile-dev-inc.slack.com/archives/C083YB8N42G)

0 commit comments

Comments
 (0)