python 3.9 segfault in libssl SSL_get_peer_certificate

[xeron]

Provide environment information

OS:
Darwin C4Q7YDHK6G 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:11 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6020 arm64

SHELL:
GNU bash, version 5.2.37(1)-release (aarch64-apple-darwin23.4.0)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

BASH VERSION:
5.2.37(1)-release

ASDF VERSION:
v0.14.1

ASDF INTERNAL VARIABLES:
ASDF_DEFAULT_TOOL_VERSIONS_FILENAME=.tool-versions
ASDF_DATA_DIR=/Users/ilarionov/.asdf
ASDF_DIR=/opt/homebrew/opt/asdf/libexec
ASDF_CONFIG_FILE=/Users/ilarionov/.asdfrc

ASDF INSTALLED PLUGINS:
nodejs                       https://github.com/asdf-vm/asdf-nodejs.git master 93bd217
python                       https://github.com/danhper/asdf-python.git master a3a0185
ruby                         https://github.com/asdf-vm/asdf-ruby.git master 976e383
terraform                    https://github.com/asdf-community/asdf-hashicorp.git master 22eb1c4

To Reproduce

asdf install python 3.9.21
mkdir test-py3.9
cd test-py3.9
asdf local python 3.9.21
pip3.9 install --upgrade pip

Describe the Bug

I'm getting segfaults in python 3.9 on macOS 15.1.1 with openssl 3.4.0 from homebrew:

$ pip3.9 install --upgrade pip
Requirement already satisfied: pip in /Users/ilarionov/.asdf/installs/python/3.9.21/lib/python3.9/site-packages (23.0.1)
/Users/ilarionov/.asdf/plugins/python/shims/pip: line 53: 33991 Segmentation fault: 11  "$pip" "$@"

$ lldb -- ~/.asdf/installs/python/3.9.21/bin/python3.9 -m pip install pip
(lldb) target create "/Users/ilarionov/.asdf/installs/python/3.9.21/bin/python3.9"
Current executable set to '/Users/ilarionov/.asdf/installs/python/3.9.21/bin/python3.9' (arm64).
(lldb) settings set -- target.run-args  "-m" "pip" "install" "pip"
(lldb) run
Process 32537 launched: '/Users/ilarionov/.asdf/installs/python/3.9.21/bin/python3.9' (arm64)
Requirement already satisfied: pip in /Users/ilarionov/.asdf/installs/python/3.9.21/lib/python3.9/site-packages (23.0.1)
Process 32537 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x180f9f2b8fbab048)
    frame #0: 0x000000026de10614 libssl.48.dylib`SSL_get_peer_certificate + 28
libssl.48.dylib`SSL_get_peer_certificate:
->  0x26de10614 <+28>: ldr    x19, [x8, #0x80]
    0x26de10618 <+32>: cbz    x19, 0x26de10640 ; <+72>
    0x26de1061c <+36>: add    x0, x19, #0x1c
    0x26de10620 <+40>: adrp   x3, 34
Target 0: (python3.9) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x180f9f2b8fbab048)
  * frame #0: 0x000000026de10614 libssl.48.dylib`SSL_get_peer_certificate + 28
    frame #1: 0x000000010073e764 _ssl.cpython-39-darwin.so`_ssl__SSLSocket_getpeercert + 108

$ otool -L ~/.asdf/installs/python/3.9.21/lib/python3.9/lib-dynload/_ssl.cpython-39-darwin.so
/Users/ilarionov/.asdf/installs/python/3.9.21/lib/python3.9/lib-dynload/_ssl.cpython-39-darwin.so:
	/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)

Not sure if related but SSL_get_peer_certificate was deprecated recently: openssl/openssl#24296

Reported to cpython here but not reproducible with python 3.9 from homebrew or when compiled manually from sources. Could be asdf-python or pyenv specific bug.

Expected Behaviour

pip install works and doesn't segfault

[xeron]

Found a possible issue and a workaround. This fixes the segfault for me:

PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" asdf install python 3.9.21

I think the problem comes from pyenv trying to use openssl 1.1 but python ssl lib linking with openssl 3:

https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/3.9.21#L1

$ otool -L ~/.asdf/installs/python/3.9.21/lib/python3.9/lib-dynload/_ssl.cpython-39-darwin.so
/Users/ilarionov/.asdf/installs/python/3.9.21/lib/python3.9/lib-dynload/_ssl.cpython-39-darwin.so:
	/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)

[wojexe]

I ran into the same issue - I've discovered that the error stops happening when using Python >= 3.11, since 3.11 switched to OpenSSL3.

The workaround provided by @xeron works for me.

[xeron]

This was fixed in pyenv/pyenv#3223.