-
Notifications
You must be signed in to change notification settings - Fork 7.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unknown memory leak in PHP 8.2 FPM #13775
Comments
Do you use any third party extensions as well? If yes, I wonder if there may be a leak there too. Given the information you gave and the tests you did with Valgrind and massif, it seems indeed persistent allocations are not the problem. I'd say it is worth trying to track request allocations too. To do so, you need to set the environment variable USE_ZEND_ALLOC to 0. This disables Zend's allocator and uses system malloc instead. This will make request allocations visible to Valgrind and massif. Edit: although Zend's memory manager should clean up unfreed memory after request shutdown... |
We are using php-redis, other than that I would say it's vanilla/core extensions.
I have now made a 6,5 hour Valgrind test with
massif.out.zip <-- download logs here I read in Generating a valgrind log that you can enable |
I don't really notice anything weird, but that data . I'm sorry, but I don't really have an idea what might be going on.
The allocations will be tracked in any case, but the advantage of using this environment variable is that modules will not be unloaded and therefore symbol names will be visible in stack traces of third party modules. |
Thanks for taking a look! I haven't been able to find a reproducer, so next step will be set up a test environment to learn how to make our production environment co-exist with a manually built PHP. It looks like it's only 10 steps to find the correct commit, but we will need to run the good one for a while, so will take some time for me to set it up. AFAIK, this is the correct way to start testing it:
|
That looks right. It's possible you may need to re-run buildconf and configure too when building PHP, during bisection, because internal APIs and build files changed between PHP 8.1 and PHP 8.2. |
https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Leak-Checking tried that ? should work if you preload jemalloc AND USE_ZEND_ALLOC=0 |
That doesn't really work properly, see #10670 for why |
No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you. |
Hi! It took me a while but I have now successfully manage to find where our issue started. Don't know which extension that triggers it yet, but it's one of these;
I have been running PHP 8.2.19 with the above code removed (and it's modifications) for 3 days without any sign of memory leaks. I'm using a minimum build for our CDN right now, haven't build it with all extensions we require for www. I did all testing with the following configuration and settings.
I will let this PHP 8.2.19 test running over the weekend and after that try to remove one extension at the time to pinpoint which of |
Thanks for the reporting! |
I haven't done a test where opcache has been disabled or not loaded. Going to do a test on Monday. You want me not to load the extension?
Or load it and not enable it?
|
Thanks! |
I have now done a 30 hours test with the branch |
Thanks a lot for the info, it's very peculiar. |
Thanks for taking the time trying to find the bug! I made two new tests today where i first removed |
Thanks, that narrows it down even more but at this point I'm convinced it's somehow in opcache. Of course more data is better to help diagnose this! |
Did you manage to find out that the reason is and if there is some fix? We transitioned from php 7.4 to 8.2 and now all of a sudden our server crashes for no reason - and the "no reason" seems to be we are running out of RAM |
Could you share the contents of What is the value of the One possible cause may be that we override an existing mapping due to a race condition or a mistake in 17aa81a. To check this, could you replace
|
Not yet. I have disabled @arnaud-lb Thanks for the code suggestion, I will run a test tomorrow and get back to you. |
No dice. We got memory leak with the new build.
|
Would turning opcache off fix the issue temporary, while this is being looked at? |
I have no knowledge of your application(s) or settings. For our specific issue you can solve it by disabling OPcache. PHP 8.1 reaches end of life 2025-12-31 and is a viable option until the bug or coding error is resolved. |
Thank you @jreklund. One thing noticeable in the memory mapping is that the opcache shm is allocated just next to the heap:
from experimentation, this will not prevent malloc() from working, however it will use a separate memory mapping, not labelled I'm not sure why the opcache SHM is allocated so close to the heap. This should not happen without a hint, so we may be setting a wrong hint here. |
Looking at the current code, the mmap location may be caused by this workaround: https://github.com/php/php-src/blob/ec277ce7fe1e7b7f98e270198826fc1432cdcb3d/ext/opcache/shared_alloc_mmap.c#L68C1-L79C4. This is unexpectedly causing the function to return a hint when it would otherwise return no hint. Edit: The workaround is not an issue. The end of the heap is less than 4GiB away from the .text segment, so the free segment after the heap is a good candidate. |
Could you test with this patch applied? https://patch-diff.githubusercontent.com/raw/php/php-src/pull/14793.diff This is a backport of a2af8ac from 8.3 + a fix to prevent placing the opcache shm segment just after the heap. Please also share the content of |
Hi! Your latest patch seem stable. Have been running it for 8 hours now. Will leave it over night.
|
Great! I will see if we can merge this, then. |
Great job. I know that PHP 8.1 is out of active support, but is there any chance that being merged for 8.1 too as being fundamental memory issue? Will be great to get it merged at least in 8.1 too :) |
@kanevbg This specific problem don't exists in PHP 8.1 or earlier. The code that caused the problem (for us) was introduced in PHP 8.2. |
Was this released with PHP Version 8.2.21 04 Jul 2024(yesterday) or will it be on a later stage? |
No, it will be part of a future release of 8.2.x. |
#14793 has been merged. This prevents the opcache SHM from being allocated too close to the heap in most cases, which is deemed to trigger the memory leak. Non-PIE builds may still allocate the opcache SHM just after the heap when JIT is enabled, but it may be possible to fix that as well if we get more insights regarding the root cause. |
Thanks! Will try out the official patch in 3 weeks, when I get back to work. Hopefully I or someone else will find the real cause of the problem in the future. |
@arnaud-lb #14793 does not seem to have been merged, but only closed. The code does not appear in the PHP-8.2.2 tag : https://github.com/php/php-src/blob/PHP-8.2.22/ext/opcache/shared_alloc_mmap.c#L48 Do you have more information ? |
@cpernot-hf The 8.2.22 version was tagged before the fix was merged in to the 8.2.x branch. The fix will be included in 8.2.23 which will be released in september. See Lines 32 to 33 in dc670cb
|
I'm back from vacation and have been evaluated the new fix available in PHP 8.2 branch and it works without issues. Have been running it for over 3 days. |
Description
This is an follow up on the issue hi-jacking I did earlier (sorry!). I haven't been able to replicate the issue in development or on our test servers during this time. So I tried a memory profiler on the production server, without any luck. Now I need some new advice from some experts!
#8646 (comment)
After running our application for about an hour we start to see an unhealthy amount of RAM usage. Fast forward a couple of hours and our sever will run out of RAM and start writing to disk instead.
PHP versions tested: 8.2.17, 8.2.14, 8.2.12, 8.2.9, 8.2.8, 8.2.7, 8.2.5, 8.2.4
Configuration: #8646 (comment)
12 days with PHP 8.1.27 and we utilize 1.3 GB of RAM. Same utilization can be seen using PHP 5.6 - 8.1.
Around 3 hours with PHP 8.2.17 and we utilize 4.5 GB of RAM. This will build up to use all available memory.
I tried to use valgrind massif tool to locate the issue, but it's stable for 7 hours.
I have made two runs, using these settings, they give us the same behavior.
sudo valgrind --tool=massif /usr/sbin/php-fpm8.2
sudo valgrind --tool=massif /usr/sbin/php-fpm8.2 --nodaemonize --fpm-config /etc/php/8.2/fpm/php-fpm.conf
@nielsdos You helped me out before and stated there was another way of using a memory profiler. I didn't see any memory leak using Valgrid thought. But these servers fail 100% of time not using Valgrid.
If not, I will start testing our application with PHP 8.2.0 to see if it's compatible before going the
git-bisect
route.PHP Version
8.2.17
Operating System
Ubuntu 22.04.4
The text was updated successfully, but these errors were encountered: