Memory leak PHP FPM 8.1 with class entry instances

[davtur19]

Description

Description

A few months ago I tried to migrate to php8.1 but it had a memory leak, so I stayed at 8.0, the memory leak is still present today, while on version 8.0 no problem, the memory remains stable.

Info

Armbian (Ubuntu 20.04.4 LTS)
Linux roc-rk3399-pc 5.15.25-rockchip64
Docker version 20.10.16, build aa7e414

Docker image not working: php8.1-fpm
Docker image working fine: php8.0-fpm

My dockerfile:

FROM php:8.1-fpm

RUN pecl install igbinary
RUN pecl bundle redis && cd redis && phpize && ./configure --enable-redis-igbinary && make && make install
RUN docker-php-ext-enable opcache igbinary redis

RUN mv "$PHP_INI_DIR/php.ini-production" "$PHP_INI_DIR/php.ini"

HEALTHCHECK CMD curl -s 172.17.0.1:8080/ping | grep pong

My php.ini

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_get_handler,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,pcntl_async_signals,pcntl_unshare,exec,passthru,shell_exec,system,popen,curl_multi_exec,parse_ini_file,show_source,
expose_php = Off
max_execution_time = 60
max_input_time = 60
error_reporting = E_ALL
display_errors = Off
post_max_size = 51M
file_uploads = On
upload_max_filesize = 50M
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
open_basedir = "/code/:/tmp/"
cgi.force_redirect = 1

[opcache]
; Determines if Zend OPCache is enabled
opcache.enable=1
; The OPcache shared memory storage size.
opcache.memory_consumption=128
; The amount of memory for interned strings in Mbytes.
opcache.interned_strings_buffer=8
; The maximum number of keys (scripts) in the OPcache hash table.
; Only numbers between 200 and 1000000 are allowed.
opcache.max_accelerated_files=10000
; The maximum percentage of "wasted" memory until a restart is scheduled.
opcache.max_wasted_percentage=5
; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
opcache.validate_timestamps=1
; Enables or disables file search in include_path optimization
opcache.revalidate_path=1
; If disabled, all PHPDoc comments are dropped from the code to reduce the
; size of the optimized code.
opcache.save_comments=0
; Allow file existence override (file_exists, etc.) performance feature.
opcache.enable_file_override=1
; Check the cache checksum each N requests.
; The default value of "0" means that the checks are disabled.
opcache.consistency_checks=100

#opcache.enable_cli=1
#opcache.jit_buffer_size=100MB
#opcache.jit=1235

php8.1-fpm memory usage (purple line):

It had gone up to 800MB in a relatively short time without stopping

php8.0-fpm memory usage (purple line):

I don't know what other information to give you because I don't know how to use profiling tools, and I can't test disabling extensions.

PHP Version

8.1

Operating System

Armbian (Ubuntu 20.04.4 LTS)

[iluuu1994]

Well, most likely there's some code in your application that makes PHP leak memory. Is it possible that you isolate the code causing the problem and then provide a snippet that reproduces the leak?

[davtur19]

All the code runs on the webserver and doesn't run for more than 60 seconds, could it still be due to my code?

It's a little tricky trying to isolate everything, isn't there a way to do it without going crazy?
Do you have any ideas or guides to link to me?
Could functions like memory_get_usage() help me?

A Telegram bot runs on the webserver, so it's mostly curl requests and some redis.

[iluuu1994]

@davtur19 Normally any memory that PHP allocates per request gets automatically deallocated after the request ends. It's possible that some code in PHP incorrectly allocates memory (persistently instead of request-bound) which does not automatically get freed and thus the memory keeps creeping up.

Assuming you can't provide the source code, it's probably easiest to add a die; statement in different places of your application, run a couple hundred times and check if the memory keeps increasing. If not, move the die; back and check again. There are other ways to check if PHP leaks (valgrind or ASAN) but that's not quite as easy to set up.

[staabm]

You might use https://github.com/BitOne/php-meminfo to analyze memory leaks

[github-actions]

No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you.

[bckp]

Maybe this will help anyone. We face same issue, for us, issue was in file descriptors, our app often open/write/close files, as we have by accident used file cache instead of memoryone (bug on our side), and looks like PHP8.1 work differently with open files, as even terminating PHP script did not free ram consumed by file descriptors. Not sure if this is PHP bug or not...
Just FYI: @staabm @iluuu1994

[davtur19]

[global]
emergency_restart_threshold=3
emergency_restart_interval=30s
process_control_timeout=5s

[www]
pm.max_requests=1000
request_terminate_timeout=60s

I tried adding these parameters to the fpm configuration, it keeps leaking memory despite restarting the workers.

[bukka]

I'm going to re-open this as there are some things to investigate. I will get to it after the feature freeze and few other things. Are you actually talking about master process memory?

[davtur19]

It does not seem to be the master to have the leak, it is the pools that increase the memory usage, even with the emergency restart

htop php8.1 inside container [time: 01:46]

htop php8.1 inside container [time: 01:56]

php-fpm.log

[ItsReddi]

@bukka i am not sure if it feature freeze is a little bit late.
Change from PHP 8.0 -> 8.1

[bukka]

@davtur19 @ItsReddi @bckp Just to be able to better isolate this, could you try to temporarily disable opcache (opcache.enable=0) and confirm if you see still see the memory increase?

[ItsReddi]

@bukka i cannot answer that to 100% because we can only reproduce the issue in production with many concurrent requests at the moment. But from what i see with an apache benchmark test, the memory is chilling on ~120mb without op cache.

Some More info, JIT was not enabled at all time.
If the host hits max_memory, the php-fpm childs, restarting every 2 seconds with "out of memory error" an no memory is freed.

[davtur19]

@bukka the problem also occurs with opcache to 0

emergency_restart seems to help, but it's not really a good solution (ram decreased when all workers restarted)

My config file:

php.ini

disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_get_handler,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,pcntl_async_signals,pcntl_unshare,exec,passthru,shell_exec,system,popen,curl_multi_exec,parse_ini_file,show_source,
expose_php = Off
max_execution_time = 60
max_input_time = 60
error_reporting = E_ALL
display_errors = Off
post_max_size = 51M
file_uploads = On
upload_max_filesize = 50M
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
open_basedir = "/code/:/tmp/"
cgi.force_redirect = 1

[opcache]
; Determines if Zend OPCache is enabled
opcache.enable=0
; The OPcache shared memory storage size.
opcache.memory_consumption=128
; The amount of memory for interned strings in Mbytes.
opcache.interned_strings_buffer=8
; The maximum number of keys (scripts) in the OPcache hash table.
; Only numbers between 200 and 1000000 are allowed.
opcache.max_accelerated_files=10000
; The maximum percentage of "wasted" memory until a restart is scheduled.
opcache.max_wasted_percentage=5
; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
opcache.validate_timestamps=1
; Enables or disables file search in include_path optimization
opcache.revalidate_path=1
; If disabled, all PHPDoc comments are dropped from the code to reduce the
; size of the optimized code.
opcache.save_comments=0
; Allow file existence override (file_exists, etc.) performance feature.
opcache.enable_file_override=1
; Check the cache checksum each N requests.
; The default value of "0" means that the checks are disabled.
opcache.consistency_checks=100

#opcache.enable_cli=1
#opcache.jit_buffer_size=100MB
#opcache.jit=1235

fpm.conf

[global]
error_log = /var/log/php-fpm.log

emergency_restart_threshold=3
emergency_restart_interval=30s
process_control_timeout=5s

[www]
listen = /run/php/php-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
pm = dynamic
pm.max_children = 100
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 35
pm.status_path = /status
ping.path = /ping
access.log = /dev/null

pm.max_requests=1000
request_terminate_timeout=60s

[ItsReddi]

After more testing, i think we are hitting an other issue in our end.
#8065

our leak comes from opcache oom restarts.
the opcache is invalidating hundreds of scripts on each request.

Have to test that in production tomorrow.

[bukka]

@davtur19 Just to clarify, are you saying that the emergency restart helps without opcache but does not help with opcache?

[davtur19]

@bukka it probably also helped with opcache just I hadn't left it running for long enough before triggering a restart.
I can try, but I'm pretty sure I had left it running briefly in this comment.

[davtur19]

actually "helping" with the ram is the pm.max_requests parameter

fpm.conf

[global]
error_log = /var/log/php-fpm.log

;emergency_restart_threshold=3
;emergency_restart_interval=30s
process_control_timeout=5s

[www]
listen = /run/php/php-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
pm = dynamic
pm.max_children = 100
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 35
pm.status_path = /status
ping.path = /ping
access.log = /dev/null

pm.max_requests=100
request_terminate_timeout=60s

[davtur19]

I tried just emergency_restarts instead of just using max_requests

[global]
error_log = /var/log/php-fpm.log

emergency_restart_threshold=3
emergency_restart_interval=30s
process_control_timeout=5s

[www]
listen = /run/php/php-fpm.sock
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
pm = dynamic
pm.max_children = 100
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 35
pm.status_path = /status
ping.path = /ping
access.log = /dev/null

;pm.max_requests=1000
request_terminate_timeout=60s

[bukka]

@davtur19 Are you able to somehow recreate the issue on non prod (e.g. with some testing redis server and load generated by wrk or similar tools)? If you could provide a minimal app (the smallest version where you can see the issue), that would be also great. If you want, you can send me privately whatever version of the app with installation and recreation steps. I think we somehow need a way to recreate it otherwise it might be really tricky to have an idea what's going on.

[nielsdos]

@jreklund Yes that works, thanks! I can open the files and I see a trace in time for the memory allocations. Nothing yet stands out to me but it's only 5 minutes of course.

Traces like this will show us the persistent allocations, which is where the leak should come from.

For future reference: it's possible to also include request allocations by using export USE_ZEND_ALLOC=0 before running the command. But I don't believe this is necessary right now, after all those allocations should disappear after a request.

[jreklund]

This round I had valgrind running for 9 hours before sending sudo kill -TERM <pidid>. Did 10k request around every 60-90 min, so hopefully that did it. :)

phpmemory.zip

[nielsdos]

Thank you @jreklund I had a look at the graphs (You can use massif-visualizer to do so). This is how they look

PID 1681

PID 1682

Note that the x-axis is not entirely the same for both PIDs. This is because PID 1681 is the leader process, and 1682 is the worker, and once the leader has started, no more persistent allocations on requests happened in the leader. The peak in the two graphs is the same one, so you can take that as a reference point on the x-axis. Everything left of the peak is startup, and everything on the right is from the memory consumption throughout your run.
We can also see that the memory consumption of the follower (1682) stays stable, so I don't see a memory leak there.

Did you observe memory leaking behaviour of this run? If yes, that means the problem is somehow not originating from a persistent allocation, and we can give instructions on how to include request memory inside the graphs. If no, then the experiment needs to be run again in a situation where the leak occurs.

[jreklund]

That's a bummer! Valgrind runs in serial and not parallell so it combined both PHP-FPM memory with it's own, so I have no way of knowing if I had any problems in this run, by looking at the processes. Like you stated before it's super slow, so don't really wanna run it in production. 😅

I can make a longer test on our testserver (don't have a custom built PHP there). PHP 8.2 have been running for a week and started to eat up memory there as well. We do have other applications running on that server as well, and some of them require at least PHP 8.1 to run.

I did 150k request on PHP 8.0 and 250k request on PHP 8.2 and then left it as is for a week, just serving around 100 request a day.

2023-05-02 - After stress testing both PHP-FPM

2023-05-10 - After waiting a week (do note that PHP 8.0 haven't been used in any application under this week)

● php8.2-fpm.service - The PHP 8.2 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php8.2-fpm.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-05-02 13:19:53 CEST; 1 weeks 0 days ago
       Docs: man:php-fpm8.2(8)
   Main PID: 3650824 (php-fpm8.2)
     Status: "Processes active: 0, idle: 5, Requests: 265383, slow: 0, Traffic: 0.2req/sec"
      Tasks: 6 (limit: 76733)
     Memory: 609.0M
     CGroup: /system.slice/php8.2-fpm.service
             ├─3650824 php-fpm: master process (/etc/php/8.2/fpm/php-fpm.conf)
             ├─3650836 php-fpm: pool www
             ├─3650837 php-fpm: pool www
             ├─3650838 php-fpm: pool www
             ├─3650839 php-fpm: pool www
             └─3650840 php-fpm: pool www

● php8.0-fpm.service - The PHP 8.0 FastCGI Process Manager
     Loaded: loaded (/lib/systemd/system/php8.0-fpm.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-05-02 12:09:10 CEST; 1 weeks 0 days ago
       Docs: man:php-fpm8.0(8)
   Main PID: 3648340 (php-fpm8.0)
     Status: "Processes active: 0, idle: 5, Requests: 154306, slow: 0, Traffic: 0req/sec"
      Tasks: 6 (limit: 76733)
     Memory: 89.1M
     CGroup: /system.slice/php8.0-fpm.service
             ├─3648340 php-fpm: master process (/etc/php/8.0/fpm/php-fpm.conf)
             ├─3648352 php-fpm: pool www
             ├─3648353 php-fpm: pool www
             ├─3648354 php-fpm: pool www
             ├─3648355 php-fpm: pool www
             └─3648356 php-fpm: pool www

[nielsdos]

Weird, so stress testing did not cause a notable increase in memory usage, but just letting natural traffic flow in did cause an increase?
In that case your issue is not the same one as the original report in this github issue page.
I am aware of one leak in php-fpm: #11088 but I expect that leak to be fairly small to be honest, and I think older versions are also affected by that bug, so then you should've seen the issue on 8.0 too.
Please remind me: did you see the leak on 8.1 too, or don't you know that?
Also quick question: are you showing us the virtual memory usage, or the actual memory usage in RAM+swap?

[jreklund]

I crawled parts of our forum and articles, not all of the websites functionality. Only GET request and with no open session (guest). We have the same problem in our CDN (only guests there) and limited amounts of routes make me think it's something in the core routing, so I didn't make a global crawler or make it sign in.

Stress testing only made a small difference and just sitting there waiting did all the work. Haven't looked at it all during the week, so don't know when it jumped.

We haven't used PHP 8.1 in production / teserserver, only on dev-machines, so I don't know. We usually jump a version 5.6 to 7.1, 7.3, 8.0 and now 8.2.

I'm showing the RES column from htop

This is 15 million request on a production server (WEB), after a couple of DDoS attacks. So it can climb higher in PHP 8.0 as well. Naturally it don't build up as fast thought.

[jreklund]

Long time no see! We have now replaced our entire production infrastructure, as our operating system where reaching end of life.

I started off by trying PHP 8.2.9 on our four production servers, to see if it where our old servers fault, but after around 12 hours I had to restart the service on our CDN-servers as it started to write to disk, due to lack of RAM. Later on I had to restart our Web-servers as well.

I thought it might have something to do with all DEPRECATED messages that where thrown in set_error_handler(), as we failed to specify that DEPRECATED should not be logged in production. We have a filter in that function that silently hide "passing null to string", as that needs an complete application overhaul. Still only been running PHP 8.0 and 8.2 in production.

After fixing that logging error, the issue still persisted, so I downgraded to PHP 8.1.22 and that version works perfectly. 🎉

We have tested these versions in production:
8.2.9
8.2.8
8.2.7
8.2.5
8.2.4

It may be a GD problem, as the CDN-servers dies first. Or it's fopen/fwrite as they read stored images on disk. Images are only processed/verified on upload on our Web-servers, so just a gut feeling. Don't have a proof of concept code as I haven't been able to replicate it at this time. I have been busy replacing servers, so haven't made any new efforts since my last post.

We don't use any new features from PHP 8.1 and PHP 8.2, as it's compatible with PHP 8.0.

We may need to setup a production run with Valgrind after all. 😓

---

php -m

[PHP Modules]
Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
hash
iconv
igbinary
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
random
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zlib

[Zend Modules]
Zend OPcache

PHP.ini

[PHP]
engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = -1
disable_functions =
disable_classes =
zend.enable_gc = On
zend.exception_ignore_args = On
zend.exception_string_param_max_len = 0
expose_php = Off
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 2M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
[CLI Server]
cli_server.color = On
[Date]
[filter]
[iconv]
[imap]
[intl]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = Off
mail.mixed_lf_and_crlf = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.save_handler = files
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.cookie_samesite =
session.serialize_handler = php
session.gc_probability = 0
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 26
session.trans_sid_tags = "a=href,area=href,frame=src,form="
session.sid_bits_per_character = 5
[Assertion]
zend.assertions = -1
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[dba]
[opcache]
[curl]
[openssl]
[ffi]

conf.d/*.ini

extension=mysqlnd.so
zend_extension=opcache.so
opcache.jit=off
extension=pdo.so
extension=xml.so
extension=ctype.so
extension=curl.so
extension=dom.so
extension=exif.so
extension=fileinfo.so
extension=ftp.so
extension=gd.so
extension=iconv.so
extension=igbinary.so
igbinary.compact_strings=On
extension=intl.so
extension=mbstring.so
extension=mysqli.so
extension=pdo_mysql.so
extension=readline.so
extension=redis.so
extension=shmop.so
extension=simplexml.so
extension=sockets.so
extension=sysvmsg.so
extension=sysvsem.so
extension=sysvshm.so
extension=tokenizer.so
extension=xmlreader.so
extension=xmlwriter.so
extension=xsl.so
# General
date.timezone = Europe/Stockholm
# File upload
file_uploads = Off
# Exec
disable_functions = system, exec, shell_exec, passthru, phpinfo, show_source, highlight_file, popen, proc_open, fopen_with_path, dbmopen, dbase_open, putenv, chdir, rmdir, rename, filepro, filepro_rowcount, filepro_retrieve, posix_mkfifo

www.conf

[www]
user = www-data
group = www-data
listen = /run/php/php8.2-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = static
pm.max_children = 20
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

[nielsdos]

Without a reproducer or logs it's not possible for us to guess what the problem may be.
The leak could be in any extension.

Be careful with Valgrind in production, it's gonna slow things down incredibly...

An alternative is running a bisect between 8.1 and 8.2 in production. Also tedious, but might be a little easier and more reliable. This also has the advantage of finding the commit that supposedly introduced the issue for you.

[jreklund]

Yeah, I figured as much. Just wanted to give you guys an update on the status; that PHP 8.1 works, that's something you inquired on before.

By bisect do you mean git-bisect? That's something I haven't used before as well. After marking a commit good/bad we need to rebuild PHP and start the testing procedure again?

git bisect start php-8.2.4 php-8.1.0 or git bisect start php-8.2.4 php-8.1.22?
Assuming bisect looks for tags. Don't know how you merging between versions affect things.

[nielsdos]

Huh, I thought I replied to this but apparently I didn't.

By bisect do you mean git-bisect? That's something I haven't used before as well. After marking a commit good/bad we need to rebuild PHP and start the testing procedure again?

Yes, I mean a git bisect. After marking a commit good/bad you need to rebuild PHP and test again yes. Note that you might need to do a ./buildconf and ./configure again as well if you jump between merge commits.

git bisect start php-8.2.4 php-8.1.0 or git bisect start php-8.2.4 php-8.1.22? Assuming bisect looks for tags. Don't know how you merging between versions affect things.

I've done bisecting many times before, it works fine even with the merges. Just sometimes needs a ./buildconf and ./configure again.

[Xmister]

I'm not a 100% sure that it's related, however pcntl_fork() seems to leak php-fpm pools on 8.2.
A simple code like this added to a drupal site's index.php:

$pid = pcntl_fork();
    if ( $pid !== 0  ) {
        // we are the parent process
        return;
    }
    sleep(10);

Will leave the forks running in the background, and eventually they detach themselves from the master process and just keep running:

root     21194  0.0  0.8 464184 16820 ?        Ss   13:59   0:00 php-fpm: master process (/var/www/site-fpm/drush12test.prod/php-fpm-8.2.conf)
drush12+ 21277  0.0  1.1 464248 23332 ?        S    13:59   0:00 php-fpm: pool drush12test
drush12+ 21285  0.0  1.1 464248 22972 ?        S    13:59   0:00 php-fpm: pool drush12test
drush12+ 21287  0.0  1.1 464248 22972 ?        S    13:59   0:00 php-fpm: pool drush12test
drush12+ 21289  0.0  1.2 464248 24696 ?        S    13:59   0:00 php-fpm: pool drush12test

[nielsdos]

@Xmister That's a different issue than this one. Can you please open a new issue for that bug so we can track it in a separate thread (to avoid cluttering this one)? Thanks.

[bilogic]

@davtur19

how are you measuring your php FPM like in the OP? I'm facing something similar (filamentphp/filament#9637) and trying to invesstigate

[davtur19]

how are you measuring your php FPM like in the OP? I'm facing something similar (filamentphp/filament#9637) and trying to invesstigate

I was using: https://github.com/andrew-ld/docker-stats

In a few comments below there were also netdata screens: #8646 (comment)

[bilogic]

@davtur19 thanks, unfortunately I don't run my FPM in a docker.

Have you guys tried pm = ondemand? Or is that a nono because of performance considerations?

[davtur19]

@bilogic after the fixes that had been made, I no longer encountered memory leaks.
On version 8.2.12, I do not experience any problems.

[bilogic]

@davtur19 thanks, let me try 8.2.12 and can I confirm you are using pm = dynamic?

[sneakyimp]

I think I'm experiencing this memory leak with the very latest PHP 8.2.12 on our production server. We were running PHP 7 in PHP-FPM configuration and memory consumption was stable for years. We migrated our app to another server (with only tiny changes to avoid deprecation warnings about dynamic class properties) and we now see the per-process RAM consumption for our pool of PHP-FPM processes creep up over the course of several days until our server is completely out of memory. Restarting the PHP-FPM pool appears to get all the memory back. Our web app is pretty elaborate so I doubt I can replicate it for anyone, but I do have a bunch of data, including an OOM/Killed dump from the syslog, and some lists of processes were you can see the bloated RAM consumption after about a week versus the lean RAM consumption immediately following a PHP-FPM pool restart.

I was pretty alarmed to see that PHP 8.1 is going to be closed for bug fixes and this memory leak may not be resolved?

[nielsdos]

I'm going to clarify something: the original reported issue, and some related ones have been long fixed.
In fact, the original issue wasn't even fpm-specific!

But this issue report has become a general "report your memleak here" issue, which is kinda ok but confusing for people.

Without a reproducer or dump it is not clear if the leak is even in fpm, or in an extension. It could even be a leak in a third party extension.

@sneakyimp whatever issue you're having is different from OP's. A dump is better than nothing but since it comes from a production system it has no debug symbols so it will be hard to figure out what exactly happened.

And indeed, 8.1 is closed for bugfixes.

If anyone wants to share a dump, please do so privately via mail. Reminder: never share the dump in public here because a dump usually contains sensitive data like passwords/credentials/private keys.
Please also include the output of phpinfo() when sending a dump.

[sneakyimp]

@nielsdos Thanks for your response.

Without a reproducer or dump it is not clear if the leak is even in fpm, or in an extension. It could even be a leak in a third party extension.

@sneakyimp whatever issue you're having is different from OP's. A dump is better than nothing but since it comes from a production system it has no debug symbols so it will be hard to figure out what exactly happened.

I'm happy to provide any detail I have. Thanks for the warning about the dump possibly containing sensitive info. I was aware of this. That said, I don't think the dump i'm talking about is the same thing as the dump you are referring to. I'm no expert but the information doesn't look very sensitive. I just copied it from the syslog. It goes something like this:

Nov 14 16:52:44 redacted-hostname kernel: [959137.367888] php invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Nov 14 16:52:44 redacted-hostname kernel: [959137.367900] CPU: 6 PID: 1496058 Comm: php Not tainted 5.15.0-88-generic #98-Ubuntu
Nov 14 16:52:44 redacted-hostname kernel: [959137.367903] Hardware name: Xen HVM domU, BIOS 4.7 09/01/2021
Nov 14 16:52:44 redacted-hostname kernel: [959137.367905] Call Trace:
Nov 14 16:52:44 redacted-hostname kernel: [959137.367908]  <TASK>
Nov 14 16:52:44 redacted-hostname kernel: [959137.367911]  show_stack+0x52/0x5c
...etc

In the meantime, these two charts should demonstrate first the stable RAM consumption under PHP 7 and after about Nov 5 the increasing RAM consumption that led to the OOM situation.
![php-per-process](https://github.com/php/php-src/assets/1297828/68090b3a-7b69-42e5-8c91-6ad81a4d6584)
![php-overall-ram](https://github.com/php/php-src/assets/1297828/0041c10e-45db-4a17-a2bf-d577f2b4373c)

EDIT: I tried posting the charts here but they are not showing up. I posted them to imgur here.

[nielsdos]

Right, I indeed thought about a different kind of dump. The one you posted is not sensitive indeed.
From the charts it seems like the memory usages spikes shortly after a memory drop. So could it be that it spikes after an opcache restart?

[sneakyimp]

@nielsdos It looks like the adjustment to pm.max_requests is working, and memory consumption once again seems stable. Perhaps the default php-fpm config should have a nonzero value for this setting?

[nielsdos]

Interesting workaround. It probably hides the real issue, or somehow some children aren't properly terminated? I don't know.
Anyway, cc @bukka perhaps for ideas

[wizardist]

Version 8.2.5 changelog states:

Fixed bug GH-8646 (Memory leak PHP FPM 8.1).

Should this be closed then?

[nielsdos]

Version 8.2.5 changelog states:

Fixed bug GH-8646 (Memory leak PHP FPM 8.1).

Should this be closed then?

The original bug reported in this issue was fixed. It was reopened because people use this issue as a generic leak issue thread.
Perhaps this should be renamed and closed because this thread became a bit messy.
cc @iluuu1994 wdyt?

[iluuu1994]

Sure! If the original problem is solved we should close this and create new issues for anything distinctly unrelated.