Maven rerunFailingTestsCount parameter does not overwrite files written by Cucumber

[NicoIodice]

👓 What did you see?

When using the Maven property rerunFailingTestsCount (value set to 1), files written by Cucumber are not overwritten as stated here.

Our test runner classes have the following configuration:

@Suite
@SuiteDisplayName("Clients Test Runner")
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "pt.cgd.corporateloans.e2e.automation.tests.steps")
@ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "json:target/cucumber-reports/Cucumber.json"
@ExcludeTags({"ignore", "skip", "deprecated"})
@Execution(ExecutionMode.CONCURRENT)
@TestInstance(Lifecycle.PER_CLASS)
public class TestRunnerClients {
}

We use the ClueCumber plugin to generate HTML reports by compiling information from every Cucumber.json file generated by the test runners when running test scenarios from feature files existent in each module. This report can display the result of every run of the same test scenario when rerun was triggered.

✅ What did you expect to see?

We were expecting that the Cucumber.json file would be overwritten but we made some tests, incuding file permissions, and so on, and arrived to the conclusion that the file is never overwritten.

What happens is an empty file (with 0kb) is created before the @BeforeAll JUnit method and right after the @afterall Junit method finishes, all the data is flushed to the file. On the next runs, the file is not accessed anymore and we needed this to happen so we can copy both files to the folder that ClueCumber uses to generate the HTML report with the result of all runs for different test scenarios where the rerun was triggered.

📦 Which tool/library version are you using?

cluecumber-maven v3.6.0
maven-surefire-plugin v3.2.5
selenium-java v4.19.1
cucumber v7.16.1
cucumber-junit-platform-engine v.7.16.1
junit-platform-suite v.5.10.2
junit-jupiter v5.10.2

🔬 How could we reproduce it?

Pré-requisites:

• Add the parameter <rerunFailingTestsCount>1</rerunFailingTestsCount> on the pom inside the project root for maven-surefire plugin
• Make sure there is a failing test scenario within the test coverage

Step 1: Run, via command line, the maven test phase.
Step 2: Validate that the Cucumber.json file is overwritten:

• the fileModified date should be recent and not the same as the first run
• the file content should have a timestamp for the second run of the scenario

📚 Any additional context?

No response

[mpkorstanje]

At a glance this looks like it would be fixed by junit-team/junit5#3693. Could you try with JUnit 5.11.0-M1?

I would recommend using the junit-bom for dependency managment if you aren't using it already.

Alternatively you can use the junit-platform.properties file to configure the plugin.

[NicoIodice]

@mpkorstanje Thank you for your quick response!

Actually we are already using the junit-bom and the cucumber-bom for dependency management.

After a test with the latest JUnit version you pointed out the behavior is different, saving only data for the last rerun.

If we have a module with 5 tests and 2 fail, the next rerun triggers two tests to run. At the end we have a Cucumber.json file with only the 2 tests stored in the file and we loose the info for the other 3 tests.

[mpkorstanje]

That sounds like it works as it should.

[NicoIodice]

I think that was not the expected and even correct behavior.

I mean, was it supposed to loose the data from the first run? This way, we would have a report with 2 test scenarios ran instead of 5...

What is happening now is still not the behavior we expect and desire. Let me explain with a example use case:

• we have 5 test scenarios
• the automation test application is executed and first runs the 5 test scenarios
• an empty Cucumber.json file is created at this point (like before updating the dependency)
• 2 random test fails
• at the end of the module, the 2 random test that failed are triggered to execute again but the test result data now is not flushed to the Cucumber.json (as it was before updating the dependency)
• the 2 test scenarios that failed previously end the execution, as their module
• at the end of the rerun the test result data is flushed to the Cucumber.json but only with the data from the rerun (second run, behavior not reproduced before updating the dependency, at this point on the previous version it only flushed the test result data from the first run)
• now we have a html report that used a file with only test result data from 2 test scenarios and not 5 and does not correspond to all the test coverage executed but only the rerun of failed tests

Please let me know if it was clear and if is this the expected behavior. If that is, neither with a previous version, or an updated one from the indicated dependency we have the problem solved. Do we have an alternative?

[mpkorstanje]

The expected behavior is that any execution of Cucumber overwrites the results of any previous execution. This is on its own is fairly normal behavior.

And in essence, when rerunning failed tests, Surefire runs Cucumber multiple times. As Cucumber does not have any way of knowing that tests are being rerun, it can not behave differently. So the only system that knows, and can report on it correctly is Surefire.

So probably your best short term option would be to use the the JUnit Platform Launcher to programmatically run and rerun your tests. You can configure each execution with its own plugin configuration to avoid overwriting any results.

I would also welcome any attempt to check the viability of a rerun test engine implementation (#2805).

[NicoIodice]

Now I understand. Thank you for your clear feedback @mpkorstanje

We will try this approach on our TestRunners.