Performance issue running coverage on Windows #1062
Comments
sebastianbergmann
changed the title
|
There have been no changes (that I am aware of) that are specific to branch/path coverage. From your Before I go down the wrong rabbit hole, can you please check whether you also get the "hanging" when branch/path coverage is disabled? |
|
@staabm Can you have a look at https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Target/MapBuilder.php and see if/how this could be optimized? Thanks! |
|
I can have a look after Juliette pushed the phpunit 12 changes for the reproducer |
I think any project can be used to look into the performance of |
|
I was not yet able to reproduce the slow run on https://github.com/Yoast/PHPUnit-Polyfills (I have no experience with branch coverage): do you think we can trigger the slow path without xdebug beeing loaded (e.g. with pre-recorded information)? running blackfire and xdebug in the same process usually doesn't work, as these step onto each others feet. |
|
I think we should wait until Juliette answered my "can you please check whether you also get the "hanging" when branch/path coverage is disabled?" question. I am almost certain that this is not related to branch/path coverage. |
@sebastianbergmann I've just checked and with only line coverage and filtered down to one test class, I see a small, but not a significant slow down. LogsWhen running the full suite with only line coverage, I see no significant difference in the performance. LogsSo, for argument's sake, I checked the performance of path/branch coverage when running the full test suite as well, and the slow down is back, so that confirms it's the path/branch coverage which is the source of the slow-down, not the test filtering. Running the full test suite vs a filtered down test doesn't seem to make a difference in the amount of slow down though. Logs
@staabm The WIP branch I'm using is available, though not yet merged: https://github.com/Yoast/PHPUnit-Polyfills/commits/feature/composer-ci-allow-for-phpunit-12/ Just FYI/I'll figure out some hacky solution - The thing I'm running into is something which was fixed in PHPUnit 11.3.? (non-existent traits not being ignored, but causing a fatal). As this package polyfills a PHPUnit 11.2.0 assertion ( I suspect I'll end up creating PHPUnit version specific wrappers for the tests + adding another config file. Nothing to worry about though as this is an edge-case which should not affect "normal" packages. |
@staabm In case it helps, this is the exact config I used locally: <?xml version="1.0" encoding="UTF-8"?>
<phpunit
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.5/phpunit.xsd"
backupGlobals="true"
bootstrap="./tests/bootstrap.php"
beStrictAboutOutputDuringTests="true"
beStrictAboutTestsThatDoNotTestAnything="true"
colors="true"
cacheDirectory=".cache/phpunit.cache"
requireCoverageMetadata="true"
displayDetailsOnTestsThatTriggerErrors="true"
displayDetailsOnTestsThatTriggerWarnings="true"
displayDetailsOnTestsThatTriggerNotices="true"
displayDetailsOnTestsThatTriggerDeprecations="true"
displayDetailsOnPhpunitDeprecations="true"
failOnWarning="true"
failOnNotice="true"
failOnDeprecation="true"
failOnPhpunitDeprecation="false"
>
<testsuites>
<testsuite name="PHPUnitPolyfills">
<directory>./tests/</directory>
</testsuite>
</testsuites>
<source>
<include>
<file>phpunitpolyfills-autoload.php</file>
<directory suffix=".php">./src/</directory>
</include>
<exclude>
<file>src/Polyfills/AssertClosedResource_Empty.php</file>
</exclude>
</source>
<coverage pathCoverage="true">
<report>
<text outputFile="php://stdout" showOnlySummary="true"/>
<html outputDirectory="build/coverage-html" lowUpperBound="35" highLowerBound="90"/>
</report>
</coverage>
</phpunit> |
|
which I am still unable to reproduce with using |
@staabm This the config I'm using for PHP 8.4.1 (which I used for the above tests): [xdebug]
zend_extension="C:/wamp/bin/php/ext/xdebug/php_xdebug-3.4.1-8.4-ts-vs17-x86_64.dll"
;xdebug.mode allowed are : off develop coverage debug gcstats profile trace
xdebug.mode=coverage
xdebug.output_dir="C:/wamp/tmp"
xdebug.show_local_vars=0
xdebug.log="C:/wamp/logs/xdebug.log"
;xdebug.log_level : 0 Criticals, 1 Connection, 3 Warnings, 5 Communication, 7 Information, 10 Debug Breakpoint
xdebug.log_level=7
xdebug.profiler_output_name=trace.%H.%t.%p.cgrind
xdebug.use_compression=false |
|
For argument's sake just reran the tests with PHP 8.3.14 (same Xdebug config) and got the same results. |
|
@staabm Another thought, though I can't imagine it would make a difference - I'm running the tests with the PHAR file, while it looks l like you're trying to reproduce with a Composer installed version. Update: just tested and to my dismay and surprise - I cannot reproduce with a Composer install either. So the PHAR in combination with path/branch coverage is where things are going wrong... Colour me flabbergasted. |
|
There seems to be something broken with regard to the static analysis cache when the PHAR is used. Although different, sebastianbergmann/phpunit#6124 seems to point into that direction as well. |
|
@jrfnl Can you delete your |
Done (including deleting various other files, like PHPUnit 9 cache file and code coverage result files) and it unfortunately doesn't make a difference. |
|
Please test whether the PHPUnit 12.0.2 PHAR solves this issue for you. Thank you! |
|
@sebastianbergmann Tested and unfortunately, no luck. |
|
Thank you for testing. I will (try to) stop working on this until after the weekend. Besides, I have no idea what might be going on here as I looked into the changes between In general, I can only advise against using branch/path coverage "all the time" (as in configured in your XML configuration file). But, of course, that is not a solution to the problem you discovered. |
|
@sebastianbergmann Sounds like a good plan. Wishing you a relaxing weekend! Let me know if/when you want me to try additional debug steps. Happy to oblige.
Agreed. The default config for projects I manage is generally set to line coverage. It's only my local override config in which I have path/branch coverage enabled and even then, I normally only run coverage locally when making significant changes. Or, like in this case, when debugging an issue with code coverage ;-) |
The only thing I can think of right now would be to use git-bisect to find out which commit in |
|
🤔 I seem to remember setting up making PHARs available as artifacts for the builds in the PHPUnit repo. I might be able to work something out from that. |
|
|
|
Good to know, though I've never managed to get |
|
Yeah, that's not going to work. I've managed to get |
|
Sorry, but you need something UNIXy to build PHARs. |
... for PHPUnit ;-) Generated some PHARs via GH Actions now with PHPUnit at These are the commit hashes I used (from oldest to most recent):
Only the last two commits I tried yield a working PHAR and both of those have the slow down. |
|
Looked at it with fresh eyes and found another cache file. Deleting that one does fix things and gets rid of the slow-down. Also, once it's gone, re-running on PHPUnit 11 (which creates the cache anew) and then PHPUnit 12 without clearing the cache in between, re-introduces the issue. Hope this gets us another step closer. |
|
i encountered the same slowdown issue on a large private codebase running linux :/ i do not use phpunit as phar, so maybe i can try to help with the bisect. |
with upgrading to 12 i'll get the following results |
With PHPUnit 11 and php-code-coverage < 11.0.9 you mean, I guess. |
|
here is my phpunit config for completeness: <?xml version="1.0" encoding="UTF-8"?>
<phpunit
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
bootstrap="./tests/bootstrap.php"
beStrictAboutOutputDuringTests="true"
colors="true"
defaultTestSuite="Legacy,Unit"
cacheDirectory=".phpunit.cache"
displayDetailsOnTestsThatTriggerErrors="true"
displayDetailsOnPhpunitDeprecations="true"
displayDetailsOnTestsThatTriggerDeprecations="true"
displayDetailsOnIncompleteTests="true"
displayDetailsOnTestsThatTriggerWarnings="true"
failOnNotice="true"
failOnWarning="true"
failOnDeprecation="true"
>
<extensions>
<bootstrap class="Zenstruck\Foundry\PHPUnit\FoundryExtension"/>
</extensions>
<coverage />
<testsuites>
<testsuite name="Legacy">
<directory>./tests/legacy</directory>
</testsuite>
<testsuite name="Application">
<directory>./tests/application</directory>
</testsuite>
<testsuite name="Integration">
<directory>./tests/integration</directory>
</testsuite>
<testsuite name="Unit">
<directory>./tests/unit</directory>
</testsuite>
<testsuite name="phpstan">
<directory>./.phpstan/tests</directory>
</testsuite>
<testsuite name="rector">
<directory>./.rector/tests</directory>
</testsuite>
</testsuites>
<php>
<ini name="error_reporting" value="-1" />
<ini name="memory_limit" value="-1" />
<server name="SHELL_VERBOSITY" value="-1" />
<env name="SYMFONY_DEPRECATIONS_HELPER" value="disabled=1" />
</php>
<source>
<include>
<directory>./apps</directory>
<directory>./src</directory>
<directory>./lib</directory>
<directory>./plugins</directory>
</include>
<exclude>
<directory>plugins/sfPropelPlugin/data/generator</directory>
</exclude>
</source>
</phpunit> |
|
@verfriemelt-dot-org That is indeed quite a slowdown. I have not observed something similar in any of my projects. It would certainly help if you could determine which commit in php-code-coverage between It would also be interesting to see where in the code this much more time is spent. This could be done using Xdebug's profiling functionality. Please note that when comparing performance between PHPUnit 11 and PHPUnit 12 that in both cases the static analysis cache is warmed. It would be pointless to compare PHPUnit 11 with warmed static analysis cache to PHPUnit 12 without warmed static analysis cache. For example, you should try the following: While on PHPUnit 11, run this: Then update to PHPUnit 12 and run the above again. |
|
i'll check everything out in a few hours and try to get a profile run. i'll update here as soon as i find something helpful |
i sent you the link to two profile runs via the symfony slack :) |
|
I will have a look, but not today. Thank you. |
|
bisecting from 11.0.8 to main, i ended up with this result: with many runs fail due the following error |
|
The revisions of |
|
i think i found the issue 🤔 maybe... i dismissed the cache comment of yours for the most part, since we do not have a cache setup. ( thus the runtime difference comes imho mostly from the |
|
Performing static analysis while code coverage data collection is active should be avoided. Looks like I may have messed up while working towards PHPUnit 12. I will investigate, thank you for your feedback. That being said, though: configuring a cache directory for PHPUnit cannot be recommended enough. It is, for instance, done in the best practice configuration file that can be generated using |
|
that being said, i setup the cache as recommended and i got still a big memory increase + still timeouts in our pipeline due to extended runtimes. with simple unittests its not too big of a deal, which i used for testing here for the most part. but for application tests touching lots of codepaths, it still is a siginificant runtime increase noticable, But so far i can say that our pipeline used to take about 10-15 minutes and now times out at the 60minutes mark when the run is terminated. i'll collect some data, and append it here in a few hours |
|
Could you create something small which reproduces the problem for us to work with? |
|
here are the promised full runs to illustrate the current situation. this is the same codebase/configuration with cache enabled and warmed for 11.5 and 12.0 vs 12 it seems the problem scales with the size of the codebase and the files analysed. i checked all of our already migrated smaller projects (about 27k loc) and observed there a rough doubling in runtime - from ~15-18 seconds with 11.5 vs ~30s with 12.0. i did not noticed it before, because its more or less negligible. the project with the extreme difference here is about 1.7M lines of code 🤔 i have yet to find a public repository which illustrates this issue, but i think more or less every codebase sufficent in size will have this issue. also the downstream tooling like infections will suffer from this performance hit. i keep my eyes open, if i find a public repo to showcase. if i can help you in any other way with this, let me know, you can also contact me on the sf-slack if you want to try something on our codebase :) |
|
@verfriemelt-dot-org FYI: the repo where I originally noticed the issue is public and it's a relatively small codebase. |
|
@jrfnl How do I run the tests for yoast/phpunit-polyfills with PHPUnit 11 and how with PHPUnit 12? Is the first-party code configured in |
|
@sebastianbergmann There's composer scripts defined to make it easier: Or when running with the PHAR file:
|
|
I must be doing something completely wrong, or my Linux environment behaves very differently from your Windows environment with regards to these performance aspects, but I cannot reproduce a decrease in performance from PHPUnit 11.5 to PHPUnit 12.0 for the yoast/phpunit-polyfills test suite:
You can see below how I ran the test suite using PHPUnit 11.5 and PHPUnit 12.0. PHPUnit 11.5 with PCOVPHPUnit 11.5 with XdebugPHPUnit 12.0 with PCOVPHPUnit 11.5 with Xdebug |
|
i cant reproduce it with PHPUnit-Polyfills either (tried php8.3 and 8.4, running linux) |
(but probably Composer to)Summary
I'm seeing a significant performance slow-down when using PHPUnit 12 for path/branch coverage versus PHPUnit 11 when running on Windows (10).
When running just one test class (filtered down), I see a difference of 3 vs 50 seconds.
The slow-down appears to be at the start, before progress is being shown.
Current behavior
How to reproduce
If I run with
--debugon, it looks like the test suite "hangs" between "Test Passed" and "Test Finished"Reproducible via the https://github.com/Yoast/PHPUnit-Polyfills repo (though I'm still making some updates for PHPUnit 12 compatibility, which will be pulled soonish).
Expected behavior
No significant slow-down in performance.
The text was updated successfully, but these errors were encountered: