Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JITServer tests failing #21066

Open
pshipton opened this issue Feb 4, 2025 · 14 comments · May be fixed by #21085
Open

JITServer tests failing #21066

pshipton opened this issue Feb 4, 2025 · 14 comments · May be fixed by #21085
Labels
blocker comp:jitserver Artifacts related to JIT-as-a-Service project test failure

Comments

@pshipton
Copy link
Member

pshipton commented Feb 4, 2025

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/972/ - rh7-390-3
testJITServer_1

04:02:20  Expected an exit value of 0 or 143, got 1 instead.
04:02:20  Dumping the contents of log file: /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/testServer.server.out
04:02:20  ////////////////////////////////////////////////////////////////////////
04:02:20  //// JITServer is currently a technology preview. Its use is not yet supported.
04:02:20  //// 
04:02:20  //// JITServer is ready to accept incoming requests
04:02:20  //// JVMDUMP039I Processing dump event "abort", detail "" at 2025/02/04 04:01:19 - please wait.
04:02:20  //// JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/core.20250204.040119.11180.0001.dmp' in response to an event
04:02:20  //// JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/core.20250204.040119.11180.0001.dmp
04:02:20  //// JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/javacore.20250204.040119.11180.0002.txt' in response to an event
04:02:20  //// JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/javacore.20250204.040119.11180.0002.txt
04:02:20  //// JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/Snap.20250204.040119.11180.0003.trc' in response to an event
04:02:20  //// JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/Snap.20250204.040119.11180.0003.trc
04:02:20  //// JVMDUMP032I JVM requested JIT dump using '/home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/jitdump.20250204.040119.11180.0004.dmp' in response to an event
04:02:20  //// JVMDUMP051I JIT dump occurred in 'SIGABRT Thread' thread 0x00000000001C9200
04:02:20  //// JVMDUMP010I JIT dump written to /home/jenkins/workspace/Test_openjdk11_j9_sanity.functional_s390x_linux_Nightly_testList_0/aqa-tests/TKG/output_17386587791782/testJITServer_1/jitdump.20250204.040119.11180.0004.dmp
04:02:20  //// JVMDUMP013I Processed dump event "abort", detail "".
04:02:20  ////////////////////////////////////////////////////////////////////////

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_x86-64_linux_Nightly_testList_0/760/ - cent7-x64-3
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_Nightly_testList_0/908/ - rh7-390-2
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_x86-64_linux_Nightly_testList_0/694 - cent7-x64-3
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_s390x_linux_Nightly_testList_0/410 - rh7-390-4
https://openj9-jenkins.osuosl.org/job/Test_openjdk24_j9_sanity.functional_x86-64_linux_Nightly_testList_0/8/ - cent7-x64-5

also in OMR acceptance
https://openj9-jenkins.osuosl.org/job/Pipeline-OMR-Acceptance/824/
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_OMR_testList_0/565 - rh7-390-4
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_s390x_linux_OMR_testList_0/208/ - rh7-390-1
https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/598/ - cent6-x64-4
https://openj9-jenkins.osuosl.org/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_1/598/ - cent6-x64-2
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_x86-64_linux_OMR_testList_0/605/ - cent7-x64-5
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_x86-64_linux_OMR_testList_0/502/ - cent7-x64-1

@pshipton pshipton added comp:jitserver Artifacts related to JIT-as-a-Service project test failure labels Feb 4, 2025
@pshipton
Copy link
Member Author

pshipton commented Feb 4, 2025

OpenSSL changed on Linux/AIX to be bundled, not sure if it's related.

7a12bae...64a7502
eclipse-openj9/openj9-omr@8afd5c6...987df34
ibmruntimes/openj9-openjdk-jdk11@d874dd5...a16320b

All the failures seem to be on rh/centos 6/7.

@pshipton
Copy link
Member Author

pshipton commented Feb 4, 2025

@mpirvu
Copy link
Contributor

mpirvu commented Feb 4, 2025

00:03:00 [ERR] 0037560CDD7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:322:
I have a centos build container and I will see if I can reproduce it locally.

@pshipton
Copy link
Member Author

pshipton commented Feb 4, 2025

It's likely a problem related to the libssl.so.3 library we are now bundling with the JVM.

@mpirvu
Copy link
Contributor

mpirvu commented Feb 4, 2025

I cannot reproduce this problem locally. I used all the latest sources and built in a centos7 container. The host OS is RHEL9 though.

@pshipton
Copy link
Member Author

pshipton commented Feb 5, 2025

You need to build with configure option --enable-openssl-bundling, also --with-openssl=fetched.
Getting the source you need openssl, we use these options with getsource.sh -openssl-repo=https://github.com/ibmruntimes/openssl.git -openssl-branch=openssl-3.0.15+CVEs2

@mpirvu
Copy link
Contributor

mpirvu commented Feb 5, 2025

After configuring with --enable-openssl-bundling --with-openssl=fetched I see that the server does not find the bundled openssl3 dll unless I set "LD_LIBRARY_PATH" to JDK/lib. The client correctly uses the embedded OpenSSL3. Investigating...

@mpirvu
Copy link
Contributor

mpirvu commented Feb 6, 2025

Both server and client use JNI_CreateJavaVM_impl to initialize a JVM and at first glance I don't see what happens differently in the client case so that it manages to find the libssl.3 in the JDK/lib directory while the server does not find it.
dlopen open doc mentions something about DT_RPATH tag and DT_RUNPATH tag contained by the shared library that calls dlopen, but again I don't see the difference between the client and the server because the calling point of dlopen is the same.

On the side note, the openssl lib is recompiled from scratch on every 'make images' command (even though it has not been changed) which makes the "code-build-test" cycle very painful.

@pshipton
Copy link
Member Author

pshipton commented Feb 6, 2025

@keithc-ca fyi the following from the previous comment.

On the side note, the openssl lib is recompiled from scratch on every 'make images' command (even though it has not been changed) which makes the "code-build-test" cycle very painful.

@keithc-ca
Copy link
Contributor

I have wondered whether we can do better, but I'm not sure that we can detect whether the OpenSSL configuration is up-to-date. I think that configure step creates one or more new header files that are used pervasively, leading to rebuilding essentially all of OpenSSL.

My practice is to (re-)run configure after OpenSSL is built. Then a jdk build will use the existing OpenSSL artifacts rather than rebuild them.

@mpirvu
Copy link
Contributor

mpirvu commented Feb 6, 2025

Using the LD_DEBUG=libs env var I found the following:

libssl.so3 is found both by client and server because the RUNPATH from libj9jit29.so is the same in both cases:
Client:

   365884:     find library=libssl.so.3 [0]; searching
    365884:      search path=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/..:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default              
   (RUNPATH from file /team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/libj9jit29.so)
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../x86_64/libssl.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../libssl.so.3

Server:

    365786:     find library=libssl.so.3 [0]; searching
    365786:      search path=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../x86_64:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/..:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default              
    (RUNPATH from file /team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/libj9jit29.so)
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/avx512_1/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/haswell/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/avx512_1/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../tls/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/avx512_1/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../haswell/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../avx512_1/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../x86_64/libssl.so.3
    365786:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/../libssl.so.3

However, what the server cannot find is libcrypto.so.3 which is probably a dependency of openssl. In this case the client uses the RUNPATH from java while the server does not have a RUNPATH.
Client:

    365884:     find library=libcrypto.so.3 [0]; searching
    365884:      search path=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/bin:/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/bin/../lib           
   (RPATH from file ./build/linux-x86_64-server-release/images/jdk/bin/java)
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/bin/libcrypto.so.3
    365884:       trying file=/team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/bin/../lib/libcrypto.so.3

Server:

    365786:     find library=libcrypto.so.3 [0]; searching
    365786:      search cache=/etc/ld.so.cache
    365786:      search path=/lib/x86_64-linux-gnu/tls/haswell/avx512_1/x86_64:/lib/x86_64-linux-gnu/tls/haswell/avx512_1:/lib/x86_64-linux-gnu/tls/haswell/x86_64:/lib/x86_64-linux-gnu/tls/haswell:/lib/x86_64-linux-gnu/tls/avx512_1/x86_64:/lib/x86_64-linux-gnu/tls/avx512_1:/lib/x86_64-linux-gnu/tls/x86_64:/lib/x86_64-linux-gnu/tls:/lib/x86_64-linux-gnu/haswell/avx512_1/x86_64:/lib/x86_64-linux-gnu/haswell/avx512_1:/lib/x86_64-linux-gnu/haswell/x86_64:/lib/x86_64-linux-gnu/haswell:/lib/x86_64-linux-gnu/avx512_1/x86_64:/lib/x86_64-linux-gnu/avx512_1:/lib/x86_64-linux-gnu/x86_64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/tls/haswell/avx512_1/x86_64:/usr/lib/x86_64-linux-gnu/tls/haswell/avx512_1:/usr/lib/x86_64-linux-gnu/tls/haswell/x86_64:/usr/lib/x86_64-linux-gnu/tls/haswell:/usr/lib/x86_64-linux-gnu/tls/avx512_1/x86_64:/usr/lib/x86_64-linux-gnu/tls/avx512_1:/usr/lib/x86_64-linux-gnu/tls/x86_64:/usr/lib/x86_64-linux-gnu/tls:/usr/lib/x86_64-linux-gnu/haswell/avx512_1/x86_64:/usr/lib/x86_64-linux-gnu/haswell/avx512_1:/usr/lib/x86_64-linux-gnu/haswell/x86_64:/usr/lib/x86_64-linux-gnu/haswell:/usr/lib/x86_64-linux-gnu/avx512_1/x86_64:/usr/lib/x86_64-linux-gnu/avx512_1:/usr/lib/x86_64-linux-gnu/x86_64:/usr/lib/x86_64-linux-gnu:/lib/tls/haswell/avx512_1/x86_64:/lib/tls/haswell/avx512_1:/lib/tls/haswell/x86_64:/lib/tls/haswell:/lib/tls/avx512_1/x86_64:/lib/tls/avx512_1:/lib/tls/x86_64:/lib/tls:/lib/haswell/avx512_1/x86_64:/lib/haswell/avx512_1:/lib/haswell/x86_64:/lib/haswell:/lib/avx512_1/x86_64:/lib/avx512_1:/lib/x86_64:/lib:/usr/lib/tls/haswell/avx512_1/x86_64:/usr/lib/tls/haswell/avx512_1:/usr/lib/tls/haswell/x86_64:/usr/lib/tls/haswell:/usr/lib/tls/avx512_1/x86_64:/usr/lib/tls/avx512_1:/usr/lib/tls/x86_64:/usr/lib/tls:/usr/lib/haswell/avx512_1/x86_64:/usr/lib/haswell/avx512_1:/usr/lib/haswell/x86_64:/usr/lib/haswell:/usr/lib/avx512_1/x86_64:/usr/lib/avx512_1:/usr/lib/x86_64:/usr/lib      
   (system search path)
    365786:       trying file=/lib/x86_64-linux-gnu/tls/haswell/avx512_1/x86_64/libcrypto.so.3
    365786:       trying file=/lib/x86_64-linux-gnu/tls/haswell/avx512_1/libcrypto.so.3
   .....
    365786:       trying file=/lib/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/haswell/avx512_1/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/haswell/avx512_1/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/haswell/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/haswell/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/avx512_1/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/avx512_1/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/tls/libcrypto.so.3
    365786:       trying file=/usr/lib/haswell/avx512_1/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/haswell/avx512_1/libcrypto.so.3
    365786:       trying file=/usr/lib/haswell/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/haswell/libcrypto.so.3
    365786:       trying file=/usr/lib/avx512_1/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/avx512_1/libcrypto.so.3
    365786:       trying file=/usr/lib/x86_64/libcrypto.so.3
    365786:       trying file=/usr/lib/libcrypto.so.3

As I understand, the RUNPATH is set with -runpath linker option. It's likely that such a linker option exist for java but not for jitserver

Correction: I see that libcrypto is using RPATH and not RUNPATH. Therefore we should be looking for -rpath linker options.

@pshipton
Copy link
Member Author

pshipton commented Feb 6, 2025

I think we need to bundle libcrypto if we are bundling libssl. Otherwise there will be a mismatch trying to use the bundled libssl but libcrypto from the OS.
nm, they are both there, I guess I missed seeing libcrypto when I looked earlier.

@pshipton
Copy link
Member Author

pshipton commented Feb 7, 2025

Would it help to explicitly load libcrypto before loading libssl?

@mpirvu
Copy link
Contributor

mpirvu commented Feb 7, 2025

Explicitly trying to load libcrypto (before libssl) appears to use

RUNPATH from file /team/mpirvu/Build/FullJava17/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/lib/default/libj9jit29.so

as opposed to the RPATH from jitserver.
This works because the RUNPATH from libj9jit29.so includes jdk/lib/default/..

One thing to worry is whether libssl will add other dependencies in the future.
Currently I only see these:

        linux-vdso.so.1 (0x00007fff32f24000)
        libcrypto.so.3 => not found
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1ff5277000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1ff5085000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1ff535d000)

mpirvu added a commit to mpirvu/openj9 that referenced this issue Feb 7, 2025
When libssl3 (or higher) is bundled with the JDK, we want to load that version
in preference over the one present on the system. `dlopen` will do that because
the RUNPATH for the JIT dll (from which `dlopen` is invoked) includes the
"JDK/lib" path where libssl is bundled. However, as part of loading libssl3,
`dlopen` will also attempt to load libcrypto3 (because it is a dependency).
This is searched in the RPATH of the jitserver executable for a server,
or in the RPATH of the java executable for a client. Currently, the jitserver
executable does not include an RPATH, so libcrypto3 is searched on the system
and this may fail on systems that do not have version 3 installed. This problem
can be circumvented by performing an explicit `dlopen` for the crypto library,
in which case the RUNPATH for the JIT is going to be used as search path.

Fixes: eclipse-openj9#21066

Signed-off-by: Marius Pirvu <[email protected]>
@mpirvu mpirvu linked a pull request Feb 7, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker comp:jitserver Artifacts related to JIT-as-a-Service project test failure
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants