Skip to content

Commit c83f2e9

Browse files
soumyausoumyau
committed
MQE-1541: Add option to generate:tests for XSD validation on 'merged files'
added debug levels to run:test, run:failed, run:group
1 parent 9150d91 commit c83f2e9

File tree

6 files changed

+52
-21
lines changed

6 files changed

+52
-21
lines changed

‎docs/commands/mftf.md

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -122,19 +122,9 @@ vendor/bin/mftf generate:tests [option] [<test name>] [<test name>] [--remove]
122122
| `--force` | Forces test generation, regardless of the module merge order defined in the Magento instance. Example: `generate:tests --force`. |
123123
| `-i,--time` | Set time in minutes to determine the group size when `--config=parallel` is used. The __default value__ is `10`. Example: `generate:tests --config=parallel --time=15`|
124124
| `--tests` | Defines the test configuration as a JSON string.|
125-
| `--debug=[<developer> or <none>]`| Set debug level to `default` when the option is not specified, to `developer` with `--debug developer` or `--debug`, to `none` with `--debug none` | |
125+
| `--debug or --debug=[<none>]`| Performs schema validations on XML files </br> <br/> DEFAULT: `generate:tests` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test generation fails because of an invalid XML schema. This option takes extra processing time. Use it after test generation has failed once.</br> <br/> NONE: `--debug=none` skips debugging during test generation. Added for backward compatibility, it will be removed in the next MAJOR release.</br>|
126126
| `-r,--remove`| Removes the existing generated suites and tests cleaning up the `_generated` directory before the actual run. For example, `generate:tests SampleTest --remove` cleans up the entire `_generated` directory and generates `SampleTest` only.|
127127

128-
#### Debugging levels
129-
130-
You can run `generate:tests` with any of the following debug levels. XML schema validation errors will be logged as CRITICAL failures.
131-
132-
| Debug level | Description |
133-
| ---| --- |
134-
|`default` `[generate:tests]` | Perform XML schema validation on merged files. Does not indicate the file name where the error is encountered. Use `developer` level for enhanced debugging. |
135-
|`developer` `[generate:tests --debug or generate:tests --debug developer]` | Returns additional debug information (such as the filename where an error occurred) when test generation fails because of an invalid XML schema. Note: This option takes extra processing time. Use it after test generation has failed once. |
136-
|`none` `[generate:tests --debug none]`| Skip debugging during test generation. Added for backward compatibility, it will be removed with the next MAJOR release.|
137-
138128
#### Examples of the JSON configuration
139129

140130
The configuration to generate a single test with no suites:
@@ -305,6 +295,9 @@ vendor/bin/mftf run:group [--skip-generate|--remove] [--] <group1> [<group2>]
305295
| --------------------- | --------------------------------------------------------------------------------------------------------- |
306296
| `-k, --skip-generate` | Skips generating from the source XML. Instead, the command executes previously-generated groups of tests. |
307297
| `-r, --remove` | Removes previously generated suites and tests before the actual generation and run. |
298+
| `--debug or --debug=[<none>]`| <br/>Performs schema validations on XML files </br> <br/> DEFAULT: `run:group` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test run fails because of an invalid XML schema. This option takes extra processing time. Use it after test run has failed once.</br> <br/> NONE: `--debug=none` skips debugging during test run. Added for backward compatibility, it will be removed in the next MAJOR release.</br>|
299+
300+
308301

309302
#### Examples
310303

@@ -327,7 +320,7 @@ Generates and executes tests by name using Codeception.
327320
#### Usage
328321

329322
```bash
330-
vendor/bin/mftf run:test [--skip-generate|--remove] [--] <name1> [<name2>]
323+
vendor/bin/mftf run:test [--skip-generate|--remove] [--] <name1> [<name2>] [--debug|
331324
```
332325
333326
#### Options
@@ -336,6 +329,7 @@ vendor/bin/mftf run:test [--skip-generate|--remove] [--] <name1> [<name2>]
336329
|-----------------------|-----------------------------------------------------------------------------------------------------------|
337330
| `-k, --skip-generate` | Skips generating from the source XML. Instead, the command executes previously-generated groups of tests. |
338331
| `-r, --remove` | Remove previously generated suites and tests. |
332+
| `--debug or --debug=[<none>]`| <br/>Performs schema validations on XML files </br> <br/> DEFAULT: `run:test` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test run fails because of an invalid XML schema. This option takes extra processing time. Use it after test run has failed once.</br> <br/> NONE: `--debug=none` skips debugging during test run. Added for backward compatibility, it will be removed in the next MAJOR release.</br>|
339333
340334
#### Examples
341335
@@ -357,6 +351,12 @@ For more details about `failed`, refer to [Reporting][].
357351
```bash
358352
vendor/bin/mftf run:failed
359353
```
354+
#### Options
355+
356+
| Option | Description |
357+
|-----------------------|-----------------------------------------------------------------------------------------------------------|
358+
| `--debug or --debug=[<none>]`| <br/>Performs schema validations on XML files </br> <br/> DEFAULT: `run:failed` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test run fails because of an invalid XML schema. This option takes extra processing time. Use it after test run has failed once.</br> <br/> NONE: `--debug=none` skips debugging during test run. Added for backward compatibility, it will be removed in the next MAJOR release.</br>|
359+
360360
361361
#### Examples
362362

‎src/Magento/FunctionalTestingFramework/Config/Reader/MftfFilesystem.php

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ public function readFiles($fileList)
4242
$configMerger->merge($content, $fileList->getFilename(), $exceptionCollector);
4343
}
4444
// run per file validation with generate:tests -d
45-
if ($debugLevel == MftfApplicationConfig::LEVEL_DEVELOPER) {
45+
if ($debugLevel === MftfApplicationConfig::LEVEL_DEVELOPER) {
4646
$this->validateSchema($configMerger, $fileList->getFilename());
4747
}
4848
} catch (\Magento\FunctionalTestingFramework\Config\Dom\ValidationException $e) {
@@ -52,7 +52,7 @@ public function readFiles($fileList)
5252
$exceptionCollector->throwException();
5353

5454
//run validation on merged file with generate:tests
55-
if ($debugLevel == MftfApplicationConfig::LEVEL_DEFAULT) {
55+
if ($debugLevel === MftfApplicationConfig::LEVEL_DEFAULT) {
5656
$this->validateSchema($configMerger);
5757
}
5858

‎src/Magento/FunctionalTestingFramework/Console/GenerateTestsCommand.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ protected function configure()
5757
InputOption::VALUE_OPTIONAL,
5858
'Run extra validation when generating tests. Use option \'none\' to turn off debugging --
5959
added for backward compatibility, will be removed in the next MAJOR release',
60-
'default'
60+
MftfApplicationConfig::LEVEL_DEFAULT
6161
);
6262

6363
parent::configure();

‎src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,13 @@ protected function configure()
3737
'f',
3838
InputOption::VALUE_NONE,
3939
'force generation of tests regardless of Magento Instance Configuration'
40+
)->addOption(
41+
'debug',
42+
'd',
43+
InputOption::VALUE_OPTIONAL,
44+
'Run extra validation when running tests. Use option \'none\' to turn off debugging --
45+
added for backward compatibility, will be removed in the next MAJOR release',
46+
MftfApplicationConfig::LEVEL_DEFAULT
4047
);
4148

4249
parent::configure();
@@ -58,6 +65,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int
5865
$skipGeneration = $input->getOption('skip-generate');
5966
$force = $input->getOption('force');
6067
$remove = $input->getOption('remove');
68+
$debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility
6169

6270
if ($skipGeneration and $remove) {
6371
// "skip-generate" and "remove" options cannot be used at the same time
@@ -75,7 +83,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int
7583
]),
7684
'--force' => $force,
7785
'--remove' => $remove,
78-
'--debug' => MftfApplicationConfig::LEVEL_NONE
86+
'--debug' => $debug
7987
];
8088
$command->run(new ArrayInput($args), $output);
8189
}

‎src/Magento/FunctionalTestingFramework/Console/RunTestFailedCommand.php

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
use Symfony\Component\Console\Output\OutputInterface;
1414
use Symfony\Component\Process\Process;
1515
use Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException;
16+
use Symfony\Component\Console\Input\InputOption;
1617

1718
class RunTestFailedCommand extends BaseGenerateCommand
1819
{
@@ -49,7 +50,15 @@ class RunTestFailedCommand extends BaseGenerateCommand
4950
protected function configure()
5051
{
5152
$this->setName('run:failed')
52-
->setDescription('Execute a set of tests referenced via failed file');
53+
->setDescription('Execute a set of tests referenced via failed file')
54+
->addOption(
55+
'debug',
56+
'd',
57+
InputOption::VALUE_OPTIONAL,
58+
'Run extra validation when running failed tests. Use option \'none\' to turn off debugging --
59+
added for backward compatibility, will be removed in the next MAJOR release',
60+
MftfApplicationConfig::LEVEL_DEFAULT
61+
);
5362

5463
parent::configure();
5564
}
@@ -67,12 +76,13 @@ protected function configure()
6776
*/
6877
protected function execute(InputInterface $input, OutputInterface $output): int
6978
{
79+
$debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility
7080
// Create Mftf Configuration
7181
MftfApplicationConfig::create(
7282
false,
7383
MftfApplicationConfig::GENERATION_PHASE,
7484
false,
75-
MftfApplicationConfig::LEVEL_NONE
85+
$debug
7686
);
7787

7888
$testConfiguration = $this->getFailedTestList();
@@ -83,7 +93,11 @@ protected function execute(InputInterface $input, OutputInterface $output): int
8393
}
8494

8595
$command = $this->getApplication()->find('generate:tests');
86-
$args = ['--tests' => $testConfiguration, '--remove' => true];
96+
$args = [
97+
'--tests' => $testConfiguration,
98+
'--remove' => true,
99+
'--debug' => $debug
100+
];
87101

88102
$command->run(new ArrayInput($args), $output);
89103

‎src/Magento/FunctionalTestingFramework/Console/RunTestGroupCommand.php

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,13 @@ protected function configure()
3939
'f',
4040
InputOption::VALUE_NONE,
4141
'force generation of tests regardless of Magento Instance Configuration'
42+
)->addOption(
43+
'debug',
44+
'd',
45+
InputOption::VALUE_OPTIONAL,
46+
'Run extra validation when running tests. Use option \'none\' to turn off debugging --
47+
added for backward compatibility, will be removed in the next MAJOR release',
48+
MftfApplicationConfig::LEVEL_DEFAULT
4249
)->addArgument(
4350
'groups',
4451
InputArgument::IS_ARRAY | InputArgument::REQUIRED,
@@ -64,6 +71,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int
6471
$force = $input->getOption('force');
6572
$groups = $input->getArgument('groups');
6673
$remove = $input->getOption('remove');
74+
$debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility
6775

6876
if ($skipGeneration and $remove) {
6977
// "skip-generate" and "remove" options cannot be used at the same time
@@ -77,7 +85,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int
7785
$force,
7886
MftfApplicationConfig::GENERATION_PHASE,
7987
false,
80-
MftfApplicationConfig::LEVEL_NONE
88+
$debug
8189
);
8290

8391
if (!$skipGeneration) {
@@ -86,7 +94,8 @@ protected function execute(InputInterface $input, OutputInterface $output): int
8694
$args = [
8795
'--tests' => $testConfiguration,
8896
'--force' => $force,
89-
'--remove' => $remove
97+
'--remove' => $remove,
98+
'--debug' => $debug
9099
];
91100

92101
$command->run(new ArrayInput($args), $output);

0 commit comments

Comments
 (0)