Implemented functionality to return non-zero value when no tests available.

[Sanan07]

Description

Introduce new RunConfiguration parameter called TreatNoTestsAsError, which can have values true and false . true means that we want to return non-zero value when no tests are available.

Parameter can be set through runsettings file <RunConfiguration><TreatNoTestsAsError>true</TreatNoTestsAsError> </RunConfiguration> as well as through command line : -- RunConfiguration.TreatNoTestsAsError=true . Command line argument has higher priority, so it will override the value in .runsettings file, if the latter exists.

This works the same for dotnet test command.

Fix #2590
Fix #2262
Fix #2247

Update: renamed FailWhenNoTestsFound to TreatNoTestsAsError as it was shipped.

[nohwnd]

Will the related issues be fixed by this? Is there a reason to not mark them as Fix in the description?

Is there a reason the name from the proposal "TreatNoTestsAsError" is not used? Or is it simply personal preference?

[Sanan07]

All mentioned issues are the same, only one of them related to dotnet test part of the feature. When this change merged, I will mark all of them as fixed. I used FailWhenNoTestsFound which is personal preference and from my perspective is more understandable for end use.

[nohwnd]

Updated the description to automatically close the linked issues, when you merge. It's easier to navigate back to the change that way, and it avoids keeping up with related issues manually.

[Haplois]

LGTM

[Haplois]

Mostly nitpick, but I think we should use "Discovered" instead of found, and use singular "Test" instead of "Tests".

[Sanan07]

I changed the name to TreatNoTestsAsError.

[willdean]

For the benefit of people (like me) who need this feature but don't find it well documented, the magic you need to add to the command line is -- RunConfiguration.TreatNoTestsAsError=true

Note the space after the -- and that the parameter value changed during this issue, so comment at the top is outdated.

[ColbyODonnell]

The documentation here (https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2019) incorrectly lists FailWhenNoTestsFound which does not work. This pull request also makes it seem like FailWhenNoTestsFound is the intended value, but it does not work. I spent 2 hours struggling with this. Whatever name is picked, please follow up with updated documentation.

[nohwnd]

That option was not shipped, updated the docs. MicrosoftDocs/visualstudio-docs#9266

[marchewek]

In a mature project this functionality is sound.
On the other hand, in a new project, this is a blocker - one cannot push a project structure with empty unit test projects because dotnet test will fail on them because there are no tests (what is as designed at this point :')); furthermore <TreatNoTestsAsError>false</TreatNoTestsAsError> seems to have no effect on dotnet test at all...