J9vmTest_FAILED j9mm.316 ASSERTION FAILED at openj9/runtime/gc_vlhgc/GlobalMarkingScheme.cpp:957: ((uintptr_t)0x99669966 == (clazz)->eyecatcher) & Segmentation error vmState=0x00020002

[JasonFengJ9]

Failure link

From an internal build(ubu22s390x-rt-1):

openjdk version "23-beta" 2024-09-17
IBM Semeru Runtime Open Edition 23+37-202409100108 (build 23-beta+37-202409100108)
Eclipse OpenJ9 VM 23+37-202409100108 (build master-b1e27b8afc, JRE 23 Linux s390x-64-Bit Compressed References 20240910_30 (JIT enabled, AOT enabled)
OpenJ9   - b1e27b8afc
OMR      - 10fdf657a
JCL      - 980a836ab based on jdk-23+37)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-09-10T06:09:54.517Z] variation: Mode501
[2024-09-10T06:09:54.517Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs 

[2024-09-10T06:11:35.451Z] +++ j9vm.test.jni.CriticalRegionTest: +++
[2024-09-10T06:11:35.451Z] command: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/jdkbinary/j2sdk-image/bin/java  -Xjit -Xgcpolicy:balanced -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m -classpath /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/../../testDependency/lib/asm-all.jar j9vm.test.jni.CriticalRegionTest 
[2024-09-10T06:11:35.451Z] 
[2024-09-10T06:11:47.055Z] 06:11:46.812 0x3ff08002400    j9mm.316    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk23/jdk23-linux-s390x-openj9/workspace/build/src/openj9/runtime/gc_vlhgc/GlobalMarkingScheme.cpp:957: ((uintptr_t)0x99669966 == (clazz)->eyecatcher)
[2024-09-10T06:11:47.055Z] JVMDUMP039I Processing dump event "traceassert", detail "" at 2024/09/10 07:11:46 - please wait.

[2024-09-10T06:11:52.428Z] non-zero exit value: 255
[2024-09-10T06:11:52.428Z] *** Test FAILED *** (j9vm.test.jni.CriticalRegionTest)

[2024-09-10T06:12:58.078Z] 71 passed / 1 failed
[2024-09-10T06:12:58.078Z] -----------------------------------
[2024-09-10T06:12:58.078Z] J9vmTest_3_FAILED

[2024-09-10T06:12:58.078Z] variation: Mode551
[2024-09-10T06:12:58.078Z] JVM_OPTIONS:  -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced 

[2024-09-10T06:14:23.586Z] +++ j9vm.test.jni.CriticalRegionTest: +++
[2024-09-10T06:14:23.586Z] command: /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/jdkbinary/j2sdk-image/bin/java  -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced  -Xdump -Xms64m -Xmx64m -classpath /home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/aqa-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/../../testDependency/lib/asm-all.jar j9vm.test.jni.CriticalRegionTest 
[2024-09-10T06:14:23.586Z] 
[2024-09-10T06:14:37.361Z] Unhandled exception
[2024-09-10T06:14:37.361Z] Type=Segmentation error vmState=0x00020002
[2024-09-10T06:14:37.361Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
[2024-09-10T06:14:37.361Z] Handler1=000003FFADACB980 Handler2=000003FFAD9B1A60 InaccessibleAddress=00000000636F6000
[2024-09-10T06:14:37.361Z] gpr0=0000000000000190 gpr1=0000000099669966 gpr2=000003FFA80A6F40 gpr3=000003FF44001D78
[2024-09-10T06:14:37.361Z] gpr4=0000000000000003 gpr5=0000000000000001 gpr6=00000000FFF88000 gpr7=00000000636F6D00
[2024-09-10T06:14:37.361Z] gpr8=000003FF44001D78 gpr9=0000000000000001 gpr10=000003FFA80A6F40 gpr11=00000000FFF88000
[2024-09-10T06:14:37.361Z] gpr12=000003FFAE7B0F90 gpr13=000003FFACC04890 gpr14=000003FFACA18ADC gpr15=000003FFADA7E700
[2024-09-10T06:14:37.361Z] psw=000003FFACA16B84 mask=0705200180000000 fpc=00080000 bea=000003FFACA18AD6
[2024-09-10T06:14:37.361Z] fpr0=0000000000000001 (f: 1.000000, d: 4.940656e-324)
[2024-09-10T06:14:37.361Z] fpr1=411fff0000000000 (f: 0.000000, d: 5.242240e+05)
[2024-09-10T06:14:37.361Z] fpr2=000003ffa85297a4 (f: 2823985152.000000, d: 2.172197e-311)
[2024-09-10T06:14:37.361Z] fpr3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-09-10T06:14:37.361Z] fpr4=411fff0000000000 (f: 0.000000, d: 5.242240e+05)
[2024-09-10T06:14:37.361Z] fpr5=000003ffadecbeb8 (f: 2917973760.000000, d: 2.172243e-311)
[2024-09-10T06:14:37.361Z] fpr6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-09-10T06:14:37.361Z] fpr7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-09-10T06:14:37.361Z] fpr8=000003ffada3f000 (f: 2913202176.000000, d: 2.172241e-311)
[2024-09-10T06:14:37.361Z] fpr9=0000000000041000 (f: 266240.000000, d: 1.315400e-318)
[2024-09-10T06:14:37.361Z] fpr10=000003ffa8528940 (f: 2823981312.000000, d: 2.172197e-311)
[2024-09-10T06:14:37.361Z] fpr11=000003ff85efd6c8 (f: 2247087872.000000, d: 2.171912e-311)
[2024-09-10T06:14:37.361Z] fpr12=000003ffada7f900 (f: 2913466624.000000, d: 2.172241e-311)
[2024-09-10T06:14:37.361Z] fpr13=000003ff802c2ff8 (f: 2150379520.000000, d: 2.171864e-311)
[2024-09-10T06:14:37.361Z] fpr14=000003ffada3f000 (f: 2913202176.000000, d: 2.172241e-311)
[2024-09-10T06:14:37.362Z] fpr15=000003ff8033c808 (f: 2150877184.000000, d: 2.171864e-311)
[2024-09-10T06:14:37.362Z] Module=/home/jenkins/workspace/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/jdkbinary/j2sdk-image/lib/default/libj9gc29.so
[2024-09-10T06:14:37.362Z] Module_base_address=000003FFAC900000
[2024-09-10T06:14:37.362Z] Target=2_90_20240910_30 (Linux 5.15.0-119-generic)
[2024-09-10T06:14:37.362Z] CPU=s390x (4 logical CPUs) (0x1f5201000 RAM)
[2024-09-10T06:14:37.362Z] ----------- Stack Backtrace -----------
[2024-09-10T06:14:37.362Z] _ZN22MM_GlobalMarkingScheme10scanObjectEP19MM_EnvironmentVLHGCP8J9ObjectNS_10ScanReasonE+0x4c (0x000003FFACA16B84 [libj9gc29.so+0x116b84])
[2024-09-10T06:14:37.362Z] _ZN22MM_GlobalMarkingScheme19markLiveObjectsScanEP19MM_EnvironmentVLHGC+0x84 (0x000003FFACA18ADC [libj9gc29.so+0x118adc])
[2024-09-10T06:14:37.362Z] _ZN25MM_ParallelGlobalMarkTask3runEP18MM_EnvironmentBase+0x1e8 (0x000003FFACA1A0B0 [libj9gc29.so+0x11a0b0])
[2024-09-10T06:14:37.362Z] _ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x22e (0x000003FFACA55DE6 [libj9gc29.so+0x155de6])
[2024-09-10T06:14:37.362Z] _Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x138 (0x000003FFACA548E8 [libj9gc29.so+0x1548e8])
[2024-09-10T06:14:37.362Z] omrsig_protect+0x3d8 (0x000003FFAD9B2BB0 [libj9prt29.so+0x32bb0])
[2024-09-10T06:14:37.362Z] dispatcher_thread_proc+0x5c (0x000003FFACA54584 [libj9gc29.so+0x154584])
[2024-09-10T06:14:37.362Z] thread_wrapper+0xf6 (0x000003FFAD908A46 [libj9thr29.so+0x8a46])
[2024-09-10T06:14:37.362Z]  (0x000003FFAE396296 [libc.so.6+0x96296])
[2024-09-10T06:14:37.362Z]  (0x000003FFAE40FF8E [libc.so.6+0x10ff8e])
[2024-09-10T06:14:37.362Z] ---------------------------------------
[2024-09-10T06:14:37.362Z] JVMDUMP039I Processing dump event "gpf", detail "" at 2024/09/10 07:14:35 - please wait.

[2024-09-10T06:14:38.941Z] non-zero exit value: 255
[2024-09-10T06:14:38.941Z] *** Test FAILED *** (j9vm.test.jni.CriticalRegionTest)

[2024-09-10T06:15:45.269Z] 71 passed / 1 failed
[2024-09-10T06:15:45.269Z] -----------------------------------
[2024-09-10T06:15:45.269Z] J9vmTest_4_FAILED

50x internal grinder - all failed

Also seen on JDK17 s390x_linux
JDK11 s390x_linux

[dmitripivkine]

It might be regression due #20132
Looking.

[dmitripivkine]

Seems it is not a regression from #20132, at least it's GC part.

This crash occur due heap corruption. An attempt to scan class object !j9object 0xFFF88000 for class !j9class 0x1137b00 ends with crash due heap corruption. An address 0xFFF88000 happen to be first address or the region. Close look to the heap shows there is an object !j9indexableobject 0x00000000FFF87FF0 0 20 crossing region border. This object is a [B value for String object !j9object 0xFFF87FD8:

0xFFF87FD0 :  fff96f18 fff87fd8 00f81600 fff87ff0 [ ..o............. ] <--- String object
0xFFF87FE0 :  00000000 8b4a4fc6 00000000 00000000 [ .....JO......... ]
0xFFF87FF0 :  00f7c500 00000014 00000000 fff88000 [ ................ ] <--- object allocated across region border
=========== Next Region ==========
0xFFF88000 :  636f6d2e 69626d2e 6770752e 696e7465 [ com.ibm.gpu.inte ] <-----
0xFFF88010 :  726e616c 00000000 00fe0200 00000000 [ rnal............ ]

> !j9object 0xFFF87FD8
J9VMJavaLangString at 0x00000000FFF87FD8 {
struct J9Class* clazz = !j9class 0xF81600 // java/lang/String
Object flags = 0x00000000;
[B value = !fj9object 0xfff87ff0 (offset = 0) (java/lang/String)
B coder = 0x00000000 (offset = 4) (java/lang/String)
I hash = 0x8B4A4FC6 (offset = 8) (java/lang/String)
Z hashIsZero = 0x00000000 (offset = 12) (java/lang/String)
"com.ibm.gpu.internal"
}

> !j9indexableobject 0x00000000FFF87FF0 0 20
!J9IndexableObject 0x00000000FFF87FF0 {
    struct J9Class* clazz = !j9arrayclass 0xF7C500   // [B
    Object flags = 0x00000000;
    U_32 size = 0x00000014;
	[0] =  99, 0x63
	[1] = 111, 0x6f
	[2] = 109, 0x6d
	[3] =  46, 0x2e
	[4] = 105, 0x69
	[5] =  98, 0x62
	[6] = 109, 0x6d
	[7] =  46, 0x2e
	[8] = 103, 0x67
	[9] = 112, 0x70
	[10] = 117, 0x75
	[11] =  46, 0x2e
	[12] = 105, 0x69
	[13] = 110, 0x6e
	[14] = 116, 0x74
	[15] = 101, 0x65
	[16] = 114, 0x72
	[17] = 110, 0x6e
	[18] =  97, 0x61
	[19] = 108, 0x6c
}

Balanced has individual Memory Pool for each region, so there is no way to allocate object out-of-line across region border (and there were no recent changes in this area). Also (by the same reason) TLH memory can not cross region's border.
My guess there is inlined object allocation in JIT (String + correspondent byte array) which does not obey TLH borders.

@r30shah @hzongaro Would you please take a look?

[r30shah]

Taking a look

[keithc-ca]

Also seen in external acceptance builds:

• https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.functional_s390x_linux_OpenJDK17/93
• https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.functional_s390x_linux_OpenJDK21/33

[r30shah]

Looked into the core-dump from https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk23_j9_sanity.functional_s390x_linux_testList_1/21/ to see if I can figure out which JIT code created the the object in question.
There were only ~250 JIT compiled methods, and looking into the one that has the code generated for inlined allocation, I can not spot anything obvious (And none of those methods were in the stack at the time of failure).

Given the internal grinder Jason has posted, has 100% failure rate, I replicated the test on my Linux on Z fyre image. I can reproduce the failure without JIT as well with same failing stack.

$ /root/20150/jdk-23+37/bin/java -Xshareclasses:none  -Xint  -Xgcpolicy:balanced -Xnocompressedrefs  -Xdump -Xms64m -Xmx64m -classpath /root/20150/openj9/test/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar:/root/20150/openj9/test/TKG/../TKG/lib/asm-all.jar j9vm.test.jni.CriticalRegionTest
Unhandled exception
Type=Segmentation error vmState=0x00020002
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF9BCCB980 Handler2=000003FFA02B1A60 InaccessibleAddress=0000000000000000
gpr0=0000000000000138 gpr1=0000000099669966 gpr2=000003FF9C0A4BA0 gpr3=000003FF64004398
gpr4=000003FF00000003 gpr5=0000000000000001 gpr6=000003FF9B08A000 gpr7=0000000000000000
gpr8=000003FF64004398 gpr9=0000000000000001 gpr10=000003FF9C0A4BA0 gpr11=000003FF9B08A000
gpr12=000003FFA0DB0F90 gpr13=000003FF9B681890 gpr14=000003FF9B496E1C gpr15=000003FF9BC7E700
psw=000003FF9B494F14 mask=0705200180000000 fpc=00080000 bea=000003FF9B496E16
fpr0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr1=41028bc000000000 (f: 0.000000, d: 1.519280e+05)
fpr2=000003ff9c89bef8 (f: 2626273024.000000, d: 2.172099e-311)
fpr3=000003ff9c000d48 (f: 2617249024.000000, d: 2.172095e-311)
fpr4=41028bc000000000 (f: 0.000000, d: 1.519280e+05)
fpr5=1b0d77bb370f81b8 (f: 923763136.000000, d: 2.272473e-178)
fpr6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr8=000003ff9bc3f000 (f: 2613309440.000000, d: 2.172093e-311)
fpr9=0000000000041000 (f: 266240.000000, d: 1.315400e-318)
fpr10=000003ff9c89ce58 (f: 2626276864.000000, d: 2.172099e-311)
fpr11=000002aa2809eeb0 (f: 671739584.000000, d: 1.447533e-311)
fpr12=000003ff9bc7f900 (f: 2613573888.000000, d: 2.172093e-311)
fpr13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr14=000003ff9bc3f000 (f: 2613309440.000000, d: 2.172093e-311)
fpr15=000003ffe62fe108 (f: 3861897472.000000, d: 2.172710e-311)
Module=/root/20150/jdk-23+37/lib/default/libj9gc_full29.so
Module_base_address=000003FF9B380000
Target=2_90_20240910_30 (Linux 5.15.0-119-generic)
CPU=s390x (16 logical CPUs) (0xfb8674000 RAM)
----------- Stack Backtrace -----------
_ZN22MM_GlobalMarkingScheme10scanObjectEP19MM_EnvironmentVLHGCP8J9ObjectNS_10ScanReasonE+0x4c (0x000003FF9B494F14 [libj9gc_full29.so+0x114f14])
_ZN22MM_GlobalMarkingScheme19markLiveObjectsScanEP19MM_EnvironmentVLHGC+0x84 (0x000003FF9B496E1C [libj9gc_full29.so+0x116e1c])
_ZN25MM_ParallelGlobalMarkTask3runEP18MM_EnvironmentBase+0x1e8 (0x000003FF9B4983C8 [libj9gc_full29.so+0x1183c8])
_ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x22e (0x000003FF9B4D38AE [libj9gc_full29.so+0x1538ae])
_Z23dispatcher_thread_proc2P14OMRPortLibraryPv+0x138 (0x000003FF9B4D23B0 [libj9gc_full29.so+0x1523b0])
omrsig_protect+0x3d8 (0x000003FFA02B2BB0 [libj9prt29.so+0x32bb0])
dispatcher_thread_proc+0x5c (0x000003FF9B4D204C [libj9gc_full29.so+0x15204c])
thread_wrapper+0xf6 (0x000003FFA0208A46 [libj9thr29.so+0x8a46])
 (0x000003FFA0996296 [libc.so.6+0x96296])
 (0x000003FFA0A0FF8E [libc.so.6+0x10ff8e])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2024/09/13 20:19:59 - please wait.
JVMDUMP032I JVM requested System dump using '/root/20150/openj9/test/TKG/output_17262582023330/core.20240913.201959.1960024.0001.dmp' in response to an event
JVMDUMP010I System dump written to /root/20150/openj9/test/TKG/output_17262582023330/core.20240913.201959.1960024.0001.dmp
JVMDUMP032I JVM requested Java dump using '/root/20150/openj9/test/TKG/output_17262582023330/javacore.20240913.201959.1960024.0002.txt' in response to an event
JVMDUMP010I Java dump written to /root/20150/openj9/test/TKG/output_17262582023330/javacore.20240913.201959.1960024.0002.txt
JVMDUMP032I JVM requested Snap dump using '/root/20150/openj9/test/TKG/output_17262582023330/Snap.20240913.201959.1960024.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /root/20150/openj9/test/TKG/output_17262582023330/Snap.20240913.201959.1960024.0003.trc
JVMDUMP013I Processed dump event "gpf", detail "".

Loaded core-dump into jdmpview and ran GC Check, although bit different stack, I am seeing following corruption,

> !gccheck
Starting GC Check
Checking HEAP...done (680 ms).
Checking CLASS HEAP...  <gc check (1): from debugger: CLASS HEAP: slots slot 3ff9ca41400(3ff9ca41430) -> 3ff9b08a000: class pointer is null>
done (197 ms).
Checking REMEMBERED SET...done (0 ms).
Checking UNFINALIZED...done (3 ms).
Checking FINALIZABLE...done (2 ms).
Checking OWNABLE_SYNCHRONIZER...done (1 ms).
Checking STRING TABLE...done (222 ms).
Checking CLASS LOADERS...done (2 ms).
Checking JNI GLOBAL REFS...done (1 ms).
Checking JNI WEAK GLOBAL REFS...done (7 ms).
Checking JVMTI OBJECT TAG TABLES...done (7 ms).
Checking VM CLASS SLOTS...done (1 ms).
Checking MONITOR TABLE...done (8 ms).
Checking VM THREAD SLOTS...done (373 ms).
Checking THREAD STACKS...done (84 ms).
Done (1606ms)

Given that this is reproducible with Xint as well, this may not be JIT issue.