JIT crash for java/lang/StringConcatHelper.stringOf(Ljava/lang/Object;)Ljava/lang/String; #19369
Comments
|
fyi @r30shah |
|
@beikov Looking at the segmentation fault, I see that the failure is happening while executing a JIT compiled method. From the output, it seems like it has generated diagnostic files (core dump, javacore, jitdump, etc). |
|
Pls give the 17.0.10 Milestone 2 build a try to see if the problem still occurs. https://github.com/ibmruntimes/semeru17-binaries/releases/tag/jdk-17.0.11%2B7_openj9-0.44.0-m2 |
pshipton
added this to the Release 0.46 (Java 8, 11, 17, 21, 22) July refresh milestone
|
No more JIT error with that milestone, but now we have a stackoverflow that makes no sense. The JIT seems to produce wrong code: https://ci.hibernate.org/job/hibernate-orm-pipeline/job/wip%2F7.0/21/testReport/junit/org.hibernate.orm.test.bootstrap.jpa/PersistenceUnitOverridesTests/Build___s390x_h2___Test___testCfgXmlBaseline/ It reports this SO which makes no sense: The methods in |
|
It may be the same issue so upgrading the build did not work. Looking at the method StringConcatHelper.toString, I see that it will call Object.toString() method, that is what Identifier.toString() call is. |
|
I can reproduce this on my personal VM and now I do have a core-dump to analyze. Following is the failing stack, Looking at the disassembly of |
|
Any updates on this? I see you released a new version recently, but since you didn't respond here, I wanted to know if that version is supposed to fix this problem. |
|
@beikov This is still under investigation, will post more detailed analysis on the investigation soon and will share which release would contain the fix, but to answer your question, the new version (0.44) that was released does not contain the fix - That would be more or less same driver that you tried in #19369 (comment). |
|
@r30shah Any new news on this one? |
|
@vij-singh I have been trying to reproduce this on latest nightly build - but seems like lot of changes went in https://github.com/hibernate/hibernate-orm/tree/wip/7.0 and seems like the build is failing with error in compiling java code hence it does not even reach to the point where it executes the test. I will try to rollback the repo to older commits to see if I can get the tests to execute. |
|
I don't understand why nobody cares about this issue. This makes OpenJ9 totally unusable on s390x, maybe also on other architectures. We are going to stop testing with s390x in Hibernate ORM 7.0 and discourage people from using OpenJ9 JDK17+ with Hibernate ORM altogether. |
|
@beikov I am working on the issue on background and as issue is tagged fro 0.48 release, aiming to fix it by then. I do apologize it took bit long for me, but as I mentioned in #19369 (comment), with recent changes that went in hibernate I could not reproduce the failure as I started hitting following error Which I saw on x86 linux as well. I had to find the commit to roll back reproduce this and continue investigation. |
|
Our CI builds the |
|
@beikov @r30shah I tried building hibernate with IBM Semeru java version |
|
We tried upgrading Hibernate's s390x CI runner to jdk 17.0.12 (openj9-0.46.0), but the errors still persist, see for example this build scan. |
|
I am able to reproduce this with latest nightly build. I needed to comment this part to be able to run the test with latest OpenJ9 nightly build. Posting my analysis from one of the core-dump here. This is the stderr when it fails. Looking at the disassembly of Seems like it fails trying to load from the function table from what it thinks is J9Class pointer. Looking at the value of The call stack looks like following, I have looked at the other frames after the Interpreter frame (0x000003ffafb4c45d {org/hibernate/persister/entity/AbstractEntityPersister.internalInitSubclassPropertyAliasesMap} INT [0x3f095a0-0x3f096a0]) to chase from where the corrupt object is coming from. First JIT method after that interpreter frame looks like following, Looking at the values on the Java Stack when we enter this method, Seems like we are calling this JIT compiled method from interpreter, where the first argument does not point to valid object. I have uploaded the result files on Linux on Z system in internal farm in |
|
@TobiAjila Can I get some help in nailing this one down? I am trying to find out from where the corrupt object is coming from and can not see the point where it gets corrupted. |
|
Looking at the last interpreter frame It appears that we are doing an invokedynamic at bytecode 77 If so the args should be: and |
|
Looking at the failing stack @TobiAjila you pointed - It does fail in the Looking at the JIT compiled code starting from top method, For Looking at the value in GPR1 - Now checking out second method in the stack, In above frame object was loaded from GPR8 (which was unchanged in top method and I can confirm the bad value in GPR8. In the beginning of this Looking into the next method which is Now checking the next method ( I do see that GPR2 is loaded from GPR8 which contains the first argument with which this method was called. Now looking at the values on the Java stack in this method's frame, The frame below this is the interpreted I am going to take a look at the JIT compiled code for |
|
Hi, I'm working on what looks like a similar case on x86. In the example I've been looking at it looks like OSR is involved - any sign of OSR in your cases? Quoting from my last update:
|
|
Hi @JamesKingdon I do think that some sort of redefinition / class unloading is causing the call to interpreted call, but can not say for sure, but can checkout to see if I am seeing similar behavior. |
|
I reproduced the failure again, this time with verbose option to dump OSR details. In this iteration of the failure, I am seeing the test to fail with following stack trace Same issue where object we should be coming into StringConcatHelper is corrupted. In the JIT verbose log, I can see the OSR details matching the bytecode (That in Identifier.render(), calls String concatenation helper), This suggests that at some point, we invalidated the compiled code Identifier.render() and executed interpreted version and messed up with the preparing the frame. As I am seeing consistently this happening around String Concatenation, going to attempt and create a unit test to make it easier to work with. @JamesKingdon Thanks for pointing out, I do now think this is same issue. |
|
I've opened #20232 to fix this |
|
Reopening since the fix still needs to go into 0.48 |
|
Issue Number: 19369 |
|
Is the fix part of the 0.48-m1 release? |
|
Yes, the PR #20242 is included in 0.48 M1. |
|
Looks like the new version fixes the problem. Thanks! |
Java -version output
openjdk version "17.0.10" 2024-01-16
IBM Semeru Runtime Open Edition 17.0.10.0 (build 17.0.10+7)
Eclipse OpenJ9 VM 17.0.10.0 (build openj9-0.43.0, JRE 17 Linux s390x-64-Bit Compressed References 20240116_630 (JIT enabled, AOT enabled)
OpenJ9 - 2c3d78b
OMR - ea8124dbc
JCL - 2aad089841f based on jdk-17.0.10+7)
Summary of problem
Running the Hibernate ORM testsuite triggers a JVM crash. Also see https://ci.hibernate.org/job/hibernate-orm-pipeline/job/wip%2F7.0/20/execution/node/66/log/
The problem can be reproduced by running
./gradlew check -PexcludeTests=**/KotlinProjectTests* -Plog-test-progress=true --stacktracewith thewip/7.0branch of Hibernate ORM.Diagnostic files
javacore.20240423.090841.2949679.0002.txt
jitdump.20240423.090841.2949679.0004.dmp
Snap.20240423.090841.2949679.0003.trc.txt
The text was updated successfully, but these errors were encountered: