Repository navigation
Closed
Labels
blockercomp:jitsegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Description
This also occurred on AIX with a 6/30 failure rate.
#20536 (comment)
Internal build
[Linux PPC] 80 Load_Level_2.harmony.5mins.Mode112 -Xgcpolicy:gencon -Xjit:count=0 -Xnocompressedrefs
30x grinder failed 4/30, one with vmstate=0x0005ffff #20546
j> 16:22:09 #INFO: No threads were activated following a resume all compilation threads call - FYI this also shows up in passing tests
j> 16:22:10 Unhandled exception
j> 16:22:10 Type=Segmentation error vmState=0x00020015
j> 16:22:10 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
j> 16:22:10 Handler1=0F860060 Handler2=0F698FA0
j> 16:22:10 R0=0E20CAE0 R1=AC7FC940 R2=AC806930 R3=9DFFA280
j> 16:22:10 R4=FFFFFFFD R5=B19B111C R6=F74DFC98 R7=00000001
j> 16:22:10 R8=FFFF963D R9=AC7FF4F4 R10=AD7F39E4 R11=0E205D40
j> 16:22:10 R12=84004882 R13=AC804D00 R14=B1940FCF R15=0FB3CC98
j> 16:22:10 R16=000000B4 R17=00000000 R18=0DFF3AA0 R19=F744336C
j> 16:22:10 R20=F744336C R21=F7442EF0 R22=F7442EEC R23=A0EDE270
j> 16:22:10 R24=FFFFFFFD R25=9D6F77C8 R26=00000054 R27=00000015
j> 16:22:10 R28=0F42C3A8 R29=00000000 R30=0F4BA804 R31=9DFFA280
j> 16:22:10 NIP=0E205D74 MSR=0280F032 ORIG_GPR3=0E20CADC CTR=0E205D40
j> 16:22:10 LINK=0E205D64 XER=20000000 CCR=42004822 MQ=00000001
j> 16:22:10 TRAP=00000300 DAR=FFFFFFF9 dsisr=40000000 RESULT=00000000
j> 16:22:10 Module=/bluebird/builds/bld_81426/sdk/xp3280/jre/lib/ppc/default/libj9jit29.so
j> 16:22:10 Module_base_address=0DE00000
j> 16:22:10 Target=2_90_20241111_81426 (Linux 3.10.0-1160.119.1.el7.ppc64)
j> 16:22:10 CPU=ppc (16 logical CPUs) (0x17c2e0000 RAM)
j> 16:22:10 ----------- Stack Backtrace -----------
j> 16:22:12 protectedBacktrace+0x2c (0x0F6CF64C [libj9prt29.so+0x6f64c])
j> 16:22:12 omrsig_protect+0x5bc (0x0F696FBC [libj9prt29.so+0x36fbc])
j> 16:22:12 omrintrospect_backtrace_thread_raw+0x120 (0x0F6CFDC0 [libj9prt29.so+0x6fdc0])
j> 16:22:12 protectedIntrospectBacktraceThread+0x30 (0x0F6CF150 [libj9prt29.so+0x6f150])
j> 16:22:12 omrsig_protect+0x5bc (0x0F696FBC [libj9prt29.so+0x36fbc])
j> 16:22:12 omrintrospect_backtrace_thread+0x98 (0x0F6CF218 [libj9prt29.so+0x6f218])
j> 16:22:12 generateDiagnosticFiles+0xec (0x0F860A4C [libj9vm29.so+0xd0a4c])
j> 16:22:12 omrsig_protect+0x5bc (0x0F696FBC [libj9prt29.so+0x36fbc])
j> 16:22:12 structuredSignalHandler+0x134 (0x0F860194 [libj9vm29.so+0xd0194])
j> 16:22:12 mainSynchSignalHandler+0x1f0 (0x0F699190 [libj9prt29.so+0x39190])
j> 16:22:12 __kernel_sigtramp_rt32+0x0 (0x001003C0)
j> 16:22:12 _Z25ppcCreateMethodTrampolinePvS_S_+0xc4 (0x0E205E04 [libj9jit29.so+0x405e04])
j> 16:22:12 _ZN3OMR9CodeCache19syncTempTrampolinesEv+0x240 (0x0E20CAE0 [libj9jit29.so+0x40cae0])
j> 16:22:12 _ZN3OMR16CodeCacheManager22synchronizeTrampolinesEv+0xa4 (0x0E210924 [libj9jit29.so+0x410924])
j> 16:22:12 _Z18jitHookGlobalGCEndPP15J9HookInterfacejPvS2_+0x80 (0x0DFF3B20 [libj9jit29.so+0x1f3b20])
j> 16:22:12 _Z14J9HookDispatchPP15J9HookInterfacejPv+0x20c (0x0F720D8C [libj9hookable29.so+0xd8c])
j> 16:22:12 _ZN19MM_ParallelGlobalGC11reportGCEndEP18MM_EnvironmentBase+0x21c (0x0DAF107C [libj9gc29.so+0x21107c])
j> 16:22:12 _ZN19MM_ParallelGlobalGC19internalPostCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpace+0x1d0 (0x0DAEFF50 [libj9gc29.so+0x20ff50])
j> 16:22:12 _ZN15MM_ConcurrentGC19internalPostCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpace+0x138 (0x0DB2C758 [libj9gc29.so+0x24c758])
j> 16:22:12 _ZN32MM_ConcurrentGCIncrementalUpdate19internalPostCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpace+0x24 (0x0DB3AAA4 [libj9gc29.so+0x25aaa4])
j> 16:22:12 _ZN12MM_Collector11postCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpace+0xb0 (0x0DA60AB0 [libj9gc29.so+0x180ab0])
j> 16:22:12 _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0x4a0 (0x0DA612C0 [libj9gc29.so+0x1812c0])
j> 16:22:12 _ZN17MM_MemorySubSpace14garbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionj+0x1a0 (0x0DA894C0 [libj9gc29.so+0x1a94c0])
j> 16:22:12 _ZN17MM_MemorySubSpace23percolateGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionj+0xb0 (0x0DA89110 [libj9gc29.so+0x1a9110])
j> 16:22:12 _ZN12MM_Scavenger23percolateGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription15PercolateReasonj+0x94 (0x0DB18754 [libj9gc29.so+0x238754])
j> 16:22:12 _ZN12MM_Scavenger22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0x117c (0x0DB1845C [libj9gc29.so+0x23845c])
j> 16:22:12 _ZN12MM_Collector14garbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescriptionjP28MM_ObjectAllocationInterfaceS3_P20MM_AllocationContext+0x368 (0x0DA61188 [libj9gc29.so+0x181188])
j> 16:22:12 _ZN26MM_MemorySubSpaceSemiSpace23allocationRequestFailedEP18MM_EnvironmentBaseP22MM_AllocateDescriptionN17MM_MemorySubSpace14AllocationTypeEP28MM_ObjectAllocationInterfacePS4_S8_+0x4d8 (0x0DB949B8 [libj9gc29.so+0x2b49b8])
j> 16:22:12 _ZN24MM_MemorySubSpaceGeneric11allocateTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP28MM_ObjectAllocationInterfaceP17MM_MemorySubSpaceS7_b+0x21c (0x0DB922BC [libj9gc29.so+0x2b22bc])
j> 16:22:12 _ZN23MM_TLHAllocationSupport7refreshEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x880 (0x0DA9E840 [libj9gc29.so+0x1be840])
j> 16:22:12 _ZN23MM_TLHAllocationSupport15allocateFromTLHEP18MM_EnvironmentBaseP22MM_AllocateDescriptionb+0x80 (0x0DA9EFA0 [libj9gc29.so+0x1befa0])
j> 16:22:12 _ZN25MM_TLHAllocationInterface14allocateObjectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x450 (0x0DA9D330 [libj9gc29.so+0x1bd330])
j> 16:22:12 _ZN25MM_TLHAllocationInterface21allocateArrayletSpineEP18MM_EnvironmentBaseP22MM_AllocateDescriptionP14MM_MemorySpaceb+0x34 (0x0DA9D3D4 [libj9gc29.so+0x1bd3d4])
j> 16:22:12 _Z21OMR_GC_AllocateObjectP12OMR_VMThreadP25MM_AllocateInitialization+0x274 (0x0DAE60F4 [libj9gc29.so+0x2060f4])
j> 16:22:12 J9AllocateIndexableObject+0xf80 (0x0D946820 [libj9gc29.so+0x66820])
j> 16:22:12 newBaseTypeArray+0x74 (0x0F877CF4 [libj9vm29.so+0xe7cf4])
j> 16:22:12 _Z12newByteArrayP7JNIEnv_i+0x28 (0x0F86A528 [libj9vm29.so+0xda528])
j> 16:22:12 Java_com_ibm_crypto_plus_provider_icc_NativeInterface_PBE_1doFinal+0x324 (0x0D2D8F00 [libjgskit.so+0x18f00])
j> 16:22:12 (0x9B2E7A6C [<unknown>+0x0])
j> 16:22:12 runJavaThread+0x1d0 (0x0F841E70 [libj9vm29.so+0xb1e70])
j> 16:22:12 _Z23javaProtectedThreadProcP13J9PortLibraryPv+0x100 (0x0F8B3940 [libj9vm29.so+0x123940])
j> 16:22:12 omrsig_protect+0x5bc (0x0F696FBC [libj9prt29.so+0x36fbc])
j> 16:22:12 javaThreadProc+0x70 (0x0F8B37F0 [libj9vm29.so+0x1237f0])
j> 16:22:12 thread_wrapper+0x314 (0x0F757534 [libj9thr29.so+0x7534])
j> 16:22:12 start_thread+0x10c (0x0FFB7AEC [libpthread.so.0+0x7aec])
j> 16:22:12 clone+0x84 (0x0FE31610 [libc.so.6+0x111610])
j> 16:22:12 ---------------------------------------
Metadata
Metadata
Assignees
Labels
blockercomp:jitsegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Activity
github-actions commentedon Nov 11, 2024
Issue Number: 20567
Status: Open
Recommended Components: comp:gc, comp:vm, comp:test
Recommended Assignees: pshipton, linhu2016, chengjin01
pshipton commentedon Nov 11, 2024
@dmitripivkine @hzongaro another problem which seems recently introduced, and is repeatable. Unfortunately we're not getting diagnostic upload on AIX, but only on Linux PPC (automation/issues/116).
dmitripivkine commentedon Nov 11, 2024
I will check what I can see, but this one is JIT related most likely. The crash occur in the GC End Hook event.
vij-singh commentedon Nov 12, 2024
@zl-wang FYI
IBMJimmyk commentedon Nov 13, 2024
I tried looking at the core file for http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=95773976 (internal link)
https://github.com/eclipse-openj9/openj9-omr/blob/3da49aa3aaed9217fb02d5ab3d069aa7142f89ba/compiler/runtime/OMRCodeCache.cpp#L711-L716
What happens is
startPCthere returns -3 which is treated as a real start PC and leads to a crash.The J9Method passed into the
startPClooks like this:openj9/runtime/compiler/runtime/Trampoline.cpp
Line 178 in 4b8e6ef
The
trampPtrpassed intoppcCreateMethodTrampolinelooks to be0x9dffa280.The code starting from
0x9dffa280looks like this:EncryptedPrivateKeyInfo.<init>was compiled at warm but looks like it was invalidated.zl-wang commentedon Nov 13, 2024
a reasonable fix is: at the point to re-create the permanent trampoline, testing newPC. if it is not compiled (meaning trials of compilation all failed), recreate with oldPC (at oldPC location, it is already patched to go back to interpreter).
zl-wang commentedon Nov 13, 2024
this should happen very very rarely ... never run into this situation before.
pshipton commentedon Nov 14, 2024
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=95916557
[Linux PPC] 80 Load_Level_2.harmony.5mins.Mode112
pshipton commentedon Nov 14, 2024
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=95953865
[AIX] 80 Load_Level_2.harmony.5mins.Mode112
pshipton commentedon Nov 15, 2024
Different state vmState=0x00020012
vmState [0x20012]: not a valid vmStatehttp://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96052043
[AIX] 80 Load_Level_2.harmony.5mins.Mode112
dmitripivkine commentedon Nov 15, 2024
just curious why
vmState=0x00020012is not recognized as valid vmState. It is defined in the code obviously:pshipton commentedon Nov 15, 2024
The JIT doesn't print GC states. I shouldn't have tried to look it up like I do for JIT states.
pshipton commentedon Nov 19, 2024
AIX core files can be found under http://vmfarm.rtp.raleigh.ibm.com/etc/cores/tmp/
IBMJimmyk commentedon Nov 19, 2024
I created a PR for a fix here:
eclipse-omr/omr#7550
I was able to create a small test case to reproduce the failure. The verbose log output looks like this:
callBis compiled.callAis compiled.stressTrampolinesis set so it uses a trampoline to callcallB.callBis re-compiled at profiled very-hot.callBis attempted to be re-compiled at scorching but intentionally fails.callBis attempted to be re-compiled at warm but intentionally fails and no further attempts will be made.After that, I get a global GC to happen which triggers
syncTempTrampolinesand without my fix a crash occurs. With my fix, everything is okay.Crash example:
pshipton commentedon Nov 20, 2024
fix not merged or promoted yet
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96342718
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96388471
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96528590
http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96852496
IBMJimmyk commentedon Nov 28, 2024
Fixes for the
syncTempTrampolinesversion of this problem have been merged in:#20657
eclipse-omr/omr#7550
This is not expected to fix #20546 but I am moving on to looking at that one next since the problem looks similar.
IBMJimmyk commentedon Nov 29, 2024
0.49.0 version of the above PRs:
#20720
eclipse-openj9/openj9-omr#217
github-actions commentedon Nov 29, 2024
Issue Number: 20567
Status: Closed
Actual Components: comp:jit, test failure, blocker, segfault
Actual Assignees: No one :(
PR Assignees: IBMJimmyk
pshipton commentedon Dec 2, 2024
Pls create PRs for IBM 25_01 as well.