Make phpunit fail if a --filter does not execute any tests

[staabm]

we don't know yet how to properly escape the --filter value, but we can use --fail-on-empty-test-suite to get a non-zero exit code, so we at least get a failling github action error in case our filter is bogus

found in sebastianbergmann/phpunit#4314

[staabm]

ohh interessting.. we had a test-case, which only runs in PHP 8.3+

phpstan-src/tests/PHPStan/Levels/LevelsIntegrationTest.php

Lines 45 to 47 in f11add3

	if (PHP_VERSION_ID >= 80300) {
	$topics[] = ['constantAccesses83'];
	}

and this case did never run in CI, because level tests only execute in PHP 8.1. that was a problem even before the test-parallelizing PR landed.

this turned into an error in the first commit of this PR, because the tests-levels-matrix job returns a list built on PHP 8.3, but the levels test job is executed on PHP 8.1 and therefore was instructed to run a test, which should skip below 8.3

[staabm]

should we run the level tests in all php versions? or maybe lowest and highest supported version?

[ondrejmirtes]

No it's fine to run it just once.

[ondrejmirtes]

E.g. there's nothing that changes for levels on different PHP versions. If some rule changes behaviour based on PHP version we test in that rule's test.

[ondrejmirtes]

I'd move it to phpunit.xml please - we always want to check this.

Also there are other interesting options that we might add:

  executionOrder="random"
  failOnWarning="true"
  failOnRisky="true"
  failOnEmptyTestSuite="true"
  beStrictAboutOutputDuringTests="true"

From https://backendtea.com/post/phpunit-beyond-basic-config/#tldr

[ondrejmirtes]

Any idea why this does not execute anything? https://github.com/phpstan/phpstan-src/actions/runs/7948332751/job/21698238578 The other executions work.

[staabm]

Any idea why this does not execute anything?

yes, see #2924 (comment)

[ondrejmirtes]

Looks like the random execution order breaks something :) Please look into tests that set environment variables, and if each test sets up env variables it needs, and also if they are properly unset in tearDown.

[ondrejmirtes]

I just think that this test was influenced by some other test that set an env variable.

This diff is a workaround, not a solution to the problem.

[staabm]

yeah, I am still in the process of finding a proper solution. the tests don't fail locally, so I push and look at CI results

[staabm]

I think we are good to go

[ondrejmirtes]

Awesome!