crashes with zend_mm_heap corrupted

[ralfbecker]

To test I used a Kubernetes cluster running two memcached servers and the following image:
https://quay.io/repository/egroupware/php_ius/image/0a841b570d7cdd5be81dc83dea8552e26291a670d08aaeb6feae2c14a8d13470

I run the following test with memcached session-handler and default files:

bash-3.2$ kubectl run test --image=quay.io/egroupware/php_ius:7.0.15 -it --restart=Never
Waiting for pod default/test to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.
[root@test /]# php -a
Interactive shell

php > var_dump(ini_get('session.save_handler'));
string(9) "memcached"
php > var_dump(ini_get('session.save_path'));
string(33) "memcached1:11211,memcached2:11211"
php > var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning:  session_start(): Cannot send session cache limiter - headers already sent (output started at php shell code:1) in php shell code on line 1
bool(true)
# The 2 Warnings are expected running the test on cli!
php > $_SESSION['test'] = true;
php > var_dump($_SESSION['test']);
bool(true)
php > var_dump($id = session_id());
string(26) "16qkk8kn469288365p2egg7bk6"
php > session_write_close();
# now reopening the session
php > session_id($id); var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
zend_mm_heap corrupted
[root@test /]#

Reopening the session fails and crashes php. As a test I run the same code with default session-handler files:

[root@test /]# php -a
Interactive shell
php > ini_set('session.save_handler', 'files');
php > ini_set('session.save_path', '/tmp');
php > var_dump(session_start());
bool(true)
php > var_dump($id=session_id());
string(26) "i31fegutibh77478jhrb92paj3"
php > $_SESSION['test']=true;
php > session_write_close();
# reopening the session
php > var_dump(session_id($id));
string(26) "i31fegutibh77478jhrb92paj3"
php > var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning:  session_start(): Cannot send session cache limiter - headers already sent (output started at php shell code:1) in php shell code on line 1
bool(true)
php > var_dump($_SESSION['test']);
bool(true)

With default session-handler files, sessions work as expected.

I got the same result with a single memcached server, if php-memcached would be - in contrary what it's developer says - affected by the php-memcache (no d!) bug with multiple servers allocating 4GB of ram.

Could it be that lowering/easing the libmemcached dependency caused that?
0f1cf6d
I remember someone mentioned the new libmemcached version was required and that was also what Remi required.

I believe you wanted to build a newer version, thought I could not find it and used the stock one coming with current CentOS7 (1.0.16).

I'm happy to test with a current libmemcached, if you build it.
Or do you have any other idea what I could test or what's going wrong.

Ralf

[carlwgeorge]

Lowering the libmemcached requirement was intentional in hopes of using stock libmemcached on EL7. I discussed this here and linked to it from both the php70u-pecl-memcached and the php71u-pecl-memcached requests. I'm under the impression that upstream would like to maintain compatibility with libmemcached 1.0.16. Is that fair to say @sodabrew? Would the above be more appropriate as an issue on your project? Or is it time to finally just set the minimum libmemcached version to 1.0.18 (which I'm fine with)?

[ralfbecker]

Setting session.lazy_write = false does NOT fix the issue:

bash-3.2$ kubectl run test --image=quay.io/egroupware/php_ius:7.0.15 -it --restart=Never
Waiting for pod default/test to be running, status is Pending, pod ready: false
If you don't see a command prompt, try pressing enter.
[root@test /]# php -a
Interactive shell
php >  ini_set('session.lazy_write', false);
php > session_start();
php > var_dump($id=session_id());
string(26) "rg9t28us4nuev3ukqn2u564lp6"
php > $_SESSION['test']=true;
php > session_write_close();
php > var_dump(session_id($id));
string(26) "rg9t28us4nuev3ukqn2u564lp6"
php > var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
zend_mm_heap corrupted

[sodabrew]

The minimum libmemcached with the memcached 3.x extension is 1.0.2. I know EL7 has 1.0.16, and that had a particular bug with binary protocol that causes a warning to switch to ascii protocol. Hence the recommendation to use 1.0.18.

[sodabrew]

Heap corruption is bad news! Feel free to bring this bug report upstream to php-memcached if needed.

[ralfbecker]

@sodabrew Sure I can post it at php-memcached too, question if that makes sense at the current state?

My initial post contained our memcached session configuration in a link:

session.save_handler = memcached
session.save_path = "memcached1:11211,memcached2:11211"
session.gc_maxlifetime = 14400
session.serialize_handler = igbinary
memcached.sess_persistent = On
memcached.sess_number_of_replicas = 2
memcached.sess_connect_timeout = 50
memcached.sess_lock_wait_min = 250
memcached.sess_lock_wait_max = 2000
memcached.sess_lock_retries = 20

(last 3 entries are used as we have many ajax requests, so we want an initial small timeout, but we also made the experience (with old php-memcache extension) that it's default lock-timeout of 15s, which is close to yours of 20s, caused not verifyable sessions, which a higher lock-timeout fixed)

I'm happy to test with different settings, thought I don't want to test all permutations.

[ralfbecker]

Following config works (listing only changes from above not working config):

session.save_path = "memcached1:11211"
session.serialize_handler = php
memcached.sess_persistent = Off
memcached.sess_number_of_replicas = 0

So it's either the 2 memcached servers, or persistent connection or the serialize handler ;-)

[ralfbecker]

Using 2 memcached servers (listing only changes from above):

session.save_path = "memcached1:11211,memcached2:11211"
memcached.sess_number_of_replicas = 2

Still works too:

[root@mserver-2036111592-1z87c /]# php -a
Interactive shell
php > session_start();
php > var_dump($id=session_id());
string(26) "0p6dd2bh4p3mb3sgvm7a7mdh52"
php > $_SESSION['test']=true;
php > session_write_close();
php > var_dump(session_id($id));
string(26) "0p6dd2bh4p3mb3sgvm7a7mdh52"
php > var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning:  session_start(): Cannot send session cache limiter - headers already sent (output started at php shell code:1) in php shell code on line 1
bool(true)
php > var_dump($_SESSION);
array(1) {
  ["test"]=>
  bool(true)
}

Down to either persisten connection or session.serialize_handler = igbinary ...

[ralfbecker]

Seems to be the persistent connection:

memcached.sess_persistent = On

[root@mserver-2036111592-1z87c /]# php -a
Interactive shell
php > session_start();
php > var_dump($id=session_id());
string(26) "mjbk9fph9tj0t28pg55sskcli4"
php > $_SESSION['test']=true;
php > session_write_close();
php > var_dump(session_id($id));
string(26) "mjbk9fph9tj0t28pg55sskcli4"
php > session_start();
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
zend_mm_heap corrupted

--> testing my original config without the persistent connection

[ralfbecker]

Ok, it comes down to persistent connection:

memcached.sess_persistent = Off
session.serialize_handler = igbinary

[root@mserver-2036111592-1z87c /]# php -a
Interactive shell
php > session_start();
php > var_dump($id=session_id());
string(26) "p34sndhhiv6r67j5jndi6a0mf6"
php > $_SESSION['test']=true;
php > session_write_close();
php > var_dump(session_id($id));
string(26) "p34sndhhiv6r67j5jndi6a0mf6"
php > session_start();
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning:  session_start(): Cannot send session cache limiter - headers already sent (output started at php shell code:1) in php shell code on line 1
php > var_dump($_SESSION);
array(1) {
  ["test"]=>
  bool(true)
}

--> so persistent connection seems to cause the zend_mm_heap corrupted (maybe in combination with any of the above, I'll try with only one server, so basically default config with just persistent connections on)

[ralfbecker]

An other observation session.lazy_write = On, which is the default (and not in IUS /etc/php.ini!) gives terrible performance with libmemcached 1.0.16, seems like some kind of timeout.
@carlwgeorge if you keep going with libmemcached 1.0.16, this only works reasonable with session.lazy_write off. Maybe put that in /etc/php.d/50-memcached.ini, if that's possible ...

[ralfbecker]

@sodabrew As a final test I tried with a single memcached:

[root@mserver-2036111592-1z87c /]# php -a
Interactive shell
php > ini_set('memcached.sess_persistent', true);
php > ini_set('session.save_path', 'memcached1:11211');
php > ini_set('memcached.sess_number_of_replicas', 0);
php > session_start(); $id=session_id(); $_SESSION['test']=true; session_write_close();
php >  session_id($id); session_start(); var_dump($_SESSION); session_write_close();
zend_mm_heap corrupted

So I conclude:
1.) memcached.sess_persistent = 1 is causing zend_mm_heap corrupted in 3.0.1 (at least with libmemcached 1.0.16)
2.) session.lazy_write = On with libmemcache 1.0.16 (at least) results in terrible performance (probably a timeout somewhere)

@sodabrew please let me know if you need more information

[b-harper]

@ralfbecker, Thanks for providing your findings. I edited your posts just to add some formatting so they are easier to read.

[carlwgeorge]

@ralfbecker When you file this issue upstream, can you link it here so I can subscribe to it?

[sodabrew]

Great information! There is an upstream bug describing a known issue with libmemcached 1.0.16 and the binary protocol for timeouts. Upstream bug report php-memcached-dev/php-memcached#310 has further details.

There's another upstream report about segfaults at php-memcached-dev/php-memcached#265 that seemed to point at special characters being the proximate cause of the problem, but I don't have a complete backtrace yet.

Getting a C-level backtrace with all of the symbols will be super helpful if you can post one here.