Repository navigation
Closed
Labels
arch:zcomp:jitsegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Description
Failure link
https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal/4
- test category: sanity.openjdk
- OS/architecture: s390x_linux
- java -version info:
Eclipse OpenJ9 VM (build master-0027b819341, JRE 23 Linux s390x-64-Bit Compressed References 20240930_8 (JIT enabled, AOT enabled)
OpenJ9 - 0027b819341
OMR - c7b33483737
JCL - 47ae995d04f based on jdk-23+37)
vmState [0x5ff04]: {J9VMSTATE_JIT} {InstructionSelection}
Failure output (captured from console output)
[2024-09-30T19:22:09.443Z] STARTED GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[2] Config[streamSize=10, parallel=true]'
[2024-09-30T19:22:09.443Z] #0: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xae5a5e) [0x3ff7ce65a5e]
[2024-09-30T19:22:09.444Z] #1: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaf3a00) [0x3ff7ce73a00]
[2024-09-30T19:22:09.444Z] #2: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1798bc) [0x3ff7c4f98bc]
[2024-09-30T19:22:09.444Z] #3: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x31c36) [0x3ff7d8b1c36]
[2024-09-30T19:22:09.444Z] #4: [0x3ff559f66e0]
[2024-09-30T19:22:09.444Z] #5: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x68829c) [0x3ff7ca0829c]
[2024-09-30T19:22:09.444Z] #6: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbc0856) [0x3ff7cf40856]
[2024-09-30T19:22:09.444Z] #7: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6885f6) [0x3ff7ca085f6]
[2024-09-30T19:22:09.444Z] #8: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbbacc2) [0x3ff7cf3acc2]
[2024-09-30T19:22:09.444Z] #9: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6885f6) [0x3ff7ca085f6]
[2024-09-30T19:22:09.444Z] #10: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x14d47a) [0x3ff7c4cd47a]
[2024-09-30T19:22:09.444Z] #11: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6998fc) [0x3ff7ca198fc]
[2024-09-30T19:22:09.444Z] #12: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x69a74e) [0x3ff7ca1a74e]
[2024-09-30T19:22:09.444Z] #13: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x695b88) [0x3ff7ca15b88]
[2024-09-30T19:22:09.444Z] #14: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x6c785c) [0x3ff7ca4785c]
[2024-09-30T19:22:09.444Z] #15: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x195eee) [0x3ff7c515eee]
[2024-09-30T19:22:09.444Z] #16: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x197018) [0x3ff7c517018]
[2024-09-30T19:22:09.444Z] #17: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b68) [0x3ff7d8b2b68]
[2024-09-30T19:22:09.444Z] #18: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19467a) [0x3ff7c51467a]
[2024-09-30T19:22:09.444Z] #19: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x194c04) [0x3ff7c514c04]
[2024-09-30T19:22:09.444Z] #20: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19368a) [0x3ff7c51368a]
[2024-09-30T19:22:09.444Z] #21: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193b7c) [0x3ff7c513b7c]
[2024-09-30T19:22:09.444Z] #22: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x193c14) [0x3ff7c513c14]
[2024-09-30T19:22:09.444Z] #23: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32b68) [0x3ff7d8b2b68]
[2024-09-30T19:22:09.444Z] #24: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x194056) [0x3ff7c514056]
[2024-09-30T19:22:09.444Z] #25: /home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a46) [0x3ff7d808a46]
[2024-09-30T19:22:09.444Z] #26: /lib64/libpthread.so.0(+0x8312) [0x3ff7e308312]
[2024-09-30T19:22:09.444Z] #27: /lib64/libc.so.6(+0x10e232) [0x3ff7e20e232]
[2024-09-30T19:22:09.444Z] Unhandled exception
[2024-09-30T19:22:09.444Z] Type=Segmentation error vmState=0x0005ff04
[2024-09-30T19:22:09.444Z] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=06b6fed5 Signal_Code=00000001
[2024-09-30T19:22:09.444Z] Handler1=000003FF7D9CBA48 Handler2=000003FF7D8B1A18 InaccessibleAddress=00001FCCCBA40000
[2024-09-30T19:22:09.444Z] gpr0=0000000000000000 gpr1=000003FF7C971908 gpr2=000003FF1CD562F0 gpr3=000003FF7D2993A0
[2024-09-30T19:22:09.444Z] gpr4=0000000000005E40 gpr5=0000000000000030 gpr6=000003FECB7D2BC0 gpr7=000003FEC9D00960
[2024-09-30T19:22:09.444Z] gpr8=00001FCCCBA40810 gpr9=000003FF36104CA0 gpr10=000003FF1CD562F0 gpr11=000003FEC9D007D0
[2024-09-30T19:22:09.444Z] gpr12=000003FF7D295000 gpr13=000003FF7D0EA4F8 gpr14=000003FF7CA0816E gpr15=000003FF559F6B68
[2024-09-30T19:22:09.444Z] psw=000003FF7CA0829C mask=0705200180000000 fpc=0008fe00 bea=000003FF7CA0817A
[2024-09-30T19:22:09.444Z] fpr0=401924407ab0e074 (f: 2058412160.000000, d: 6.285402e+00)
[2024-09-30T19:22:09.444Z] fpr1=3fb93d4bb7e327a9 (f: 3085117440.000000, d: 9.859155e-02)
[2024-09-30T19:22:09.444Z] fpr2=bfd24407ab0e073a (f: 2869823232.000000, d: -2.854022e-01)
[2024-09-30T19:22:09.444Z] fpr3=3fd999f1abd2dada (f: 2882722560.000000, d: 4.000210e-01)
[2024-09-30T19:22:09.444Z] fpr4=3f7ab2e26ca4c962 (f: 1822738816.000000, d: 6.518254e-03)
[2024-09-30T19:22:09.444Z] fpr5=3fcc723e7dcde3fb (f: 2110645248.000000, d: 2.222365e-01)
[2024-09-30T19:22:09.444Z] fpr6=3fcb2a68ec9ad9b5 (f: 3969571328.000000, d: 2.122317e-01)
[2024-09-30T19:22:09.444Z] fpr7=3f83e83bb8b32a19 (f: 3098749440.000000, d: 9.720294e-03)
[2024-09-30T19:22:09.444Z] fpr8=000003ff65a141a8 (f: 1705066880.000000, d: 2.171644e-311)
[2024-09-30T19:22:09.444Z] fpr9=00000000005cd458 (f: 6083672.000000, d: 3.005733e-317)
[2024-09-30T19:22:09.444Z] fpr10=000003ff65a28d50 (f: 1705151872.000000, d: 2.171644e-311)
[2024-09-30T19:22:09.444Z] fpr11=000003ffa5ffe3d8 (f: 2785010688.000000, d: 2.172178e-311)
[2024-09-30T19:22:09.454Z] fpr12=0006235ae58e315c (f: 3851301120.000000, d: 8.536087e-309)
[2024-09-30T19:22:09.454Z] fpr13=000003ff080537d8 (f: 134559712.000000, d: 2.170868e-311)
[2024-09-30T19:22:09.454Z] fpr14=000003ffdcd7aa94 (f: 3705121536.000000, d: 2.172632e-311)
[2024-09-30T19:22:09.454Z] fpr15=000003ff0804fe98 (f: 134545056.000000, d: 2.170868e-311)
[2024-09-30T19:22:09.454Z] Module=/home/jenkins/workspace/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/jdkbinary/j2sdk-image/lib/default/libj9jit29.so
[2024-09-30T19:22:09.454Z] Module_base_address=000003FF7C380000
[2024-09-30T19:22:09.454Z]
[2024-09-30T19:22:09.454Z] Method_being_compiled=java/util/stream/Gatherers$3State.integrate(Ljava/lang/Object;Ljava/util/stream/Gatherer$Downstream;)Z
[2024-09-30T19:22:09.454Z] Target=2_90_20240930_8 (Linux 3.10.0-1160.118.1.el7.s390x)
[2024-09-30T19:22:09.454Z] CPU=s390x (4 logical CPUs) (0x1ec1b1000 RAM)
[2024-09-30T19:22:09.454Z] ----------- Stack Backtrace -----------
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[2] Config[streamSize=10, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[3] Config[streamSize=33, parallel=false]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[3] Config[streamSize=33, parallel=false]'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[4] Config[streamSize=33, parallel=true]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[4] Config[streamSize=33, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[5] Config[streamSize=99, parallel=false]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[5] Config[streamSize=99, parallel=false]'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[6] Config[streamSize=99, parallel=true]'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::cancelsStartedTasksIfShortCircuited '[6] Config[streamSize=99, parallel=true]'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[1] true'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[1] true'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[2] false'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::rethrowsRuntimeExceptionsUnwrapped '[2] false'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[1] -2147483648'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[1] -2147483648'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[2] -999'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[2] -999'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[3] -1'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[3] -1'
[2024-09-30T19:22:09.454Z] STARTED GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[4] 0'
[2024-09-30T19:22:09.454Z] SUCCESSFUL GatherersMapConcurrentTest::throwsIAEWhenConcurrencyLevelIsLowerThanOne '[4] 0'
Metadata
Metadata
Assignees
Labels
arch:zcomp:jitsegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Activity
github-actions commentedon Sep 30, 2024
Issue Number: 20267
Status: Open
Recommended Components: comp:test, comp:gc, comp:vm
Recommended Assignees: jasonfengj9, pshipton, llxia
[-]JIT failure in GatherersMapConcurrentTest[/-][+]JIT failure in GatherersMapConcurrentTest vmState=0x0005ff04[/+]pshipton commentedon Sep 30, 2024
@hzongaro fyi
hzongaro commentedon Oct 1, 2024
@r30shah, could someone on your team take an initial look at this? It looks like it's crashing in a call to
OMR::CodeGenerator::decReferenceCount, but I didn't seem to be able to do much more than that with the core file. . . .r30shah commentedon Oct 1, 2024
@ehsankianifar Can you take a look at this one ? I have launched a grinder of 10x to see if it is easily reproducible. We should take a look at the core-dump (You can use the debug-images and see if we can find out which node it is complaining of).
r30shah commentedon Oct 1, 2024
@ehsankianifar Link for the 10x grinder : https://openj9-jenkins.osuosl.org/job/Grinder/3861 - 1/10 failed with BBEnd node which is same as seen in https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_s390x_linux_Personal_testList_1/4. May be generated JITdump can show what it is complaining about.
pshipton commentedon Oct 15, 2024
Another seg fault with 0x0005ff04 compiling test material.
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.openjdk_ppc64_aix_OMR_testList_0/132
ehsankianifar commentedon Oct 15, 2024
Thanks @pshipton for sharing it. I am investigating this issue and will provide the fix as soon as I find the root cause.
pshipton commentedon Nov 14, 2024
A couple more here on xmac jdk21
#20531 (comment)
hzongaro commentedon Dec 10, 2024
@ehsankianifar has been making progress investigating this, but I don't think a fix can be ready for 0.49. Moving this out to 0.51.
JasonFengJ9 commentedon Dec 10, 2024
openjdk11_j9_sanity.openjdk_s390x_linux
50x grinder - this problem wasn't repeated. One failure, which is #17135
pshipton commentedon Jan 6, 2025
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_special.system_ppc64_aix_Personal_testList_3/69
pshipton commentedon Jan 8, 2025
https://openj9-jenkins.osuosl.org/job/Test_openjdk21_j9_sanity.system_ppc64_aix_Nightly_testList_1/364
Prevent moving node above check nodes
ehsankianifar commentedon Jan 21, 2025
Investigating
GatherersMapConcurrentTestfailure reveals thatLocal Reorderingoptimization was moving a node before a null check of one of its children! This was causing a wrong tree structure and failure in rare cases. This PR was opened to fix this issue. I'll keep checking jit compilation after this fix was merged to see if there are more dimensions to this issue.