Repository navigation
Description
Java -version output
openjdk version "1.8.0_312"
IBM Semeru Runtime Open Edition (build 1.8.0_312-b07)
Eclipse OpenJ9 VM (build openj9-0.29.0, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20211022_234 (JIT enabled, AOT enabled)
OpenJ9 - e1e72c4
OMR - 299b6a2d2
JCL - 8860d39588 based on jdk8u312-b07)
Summary of problem
We use a proprietary native agent which takes stack samples using AsyncGetCallTrace() API, from the context of a signal handler. This works reliably in HotSpot VM (in the sense that the JVM is stable, AsyncGetCallTrace has its known sampling problems).
When used with a recent-enough OpenJ9 VM, which supports AGCT, the VM either hangs or crashes, typically with an OutOfMemoryError message. The crash doesn't happen with our agent in the stack trace, but it happens only if the AGCT functionality is being used. Hopefully something can be learned from the crash dumps.
Perhaps it's useful to mention that we use per-thread timers to invoke the signal, that is each thread is receiving a signal when its CPU time has crossed a threshold, and that means we can be executing AGCT in more than one thread by more than one core in parallel.
Diagnostic files
https://drive.google.com/file/d/1emXVh3avEdckSXXMOMOoI6Gwhff4YGQC/view?usp=sharing
The tarball contains two core files because two were generated (maybe by different threads). core1 is the earlier one.
The standard output/error is:
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/10/30 22:44:44 - please wait.
JVMDUMP032I JVM requested System dump using '/home/uri/oms-data/core.20211030.224444.69054.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.69403.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 69403 was not found. Expected to find core file with name "/home/uri/oms-data/core.69403"
JVMDUMP032I JVM requested Heap dump using '/home/uri/oms-data/heapdump.20211030.224444.69054.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /home/uri/oms-data/heapdump.20211030.224444.69054.0002.phd
JVMDUMP032I JVM requested Java dump using '/home/uri/oms-data/javacore.20211030.224444.69054.0003.txt' in response to an event
JVMDUMP010I Java dump written to /home/uri/oms-data/javacore.20211030.224444.69054.0003.txt
JVMDUMP032I JVM requested Snap dump using '/home/uri/oms-data/Snap.20211030.224444.69054.0004.trc' in response to an event
JVMDUMP010I Snap dump written to /home/uri/oms-data/Snap.20211030.224444.69054.0004.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/10/30 22:44:53 - please wait.
JVMDUMP032I JVM requested Heap dump using '/home/uri/oms-data/heapdump.20211030.224453.69054.0005.phd' in response to an event
JVMDUMP010I Heap dump written to /home/uri/oms-data/heapdump.20211030.224453.69054.0005.phd
JVMDUMP032I JVM requested Java dump using '/home/uri/oms-data/javacore.20211030.224453.69054.0006.txt' in response to an event
JVMDUMP010I Java dump written to /home/uri/oms-data/javacore.20211030.224453.69054.0006.txt
JVMDUMP032I JVM requested Snap dump using '/home/uri/oms-data/Snap.20211030.224453.69054.0007.trc' in response to an event
JVMDUMP010I Snap dump written to /home/uri/oms-data/Snap.20211030.224453.69054.0007.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2021/10/30 22:44:54 - please wait.
JVMDUMP032I JVM requested Heap dump using '/home/uri/oms-data/heapdump.20211030.224454.69054.0008.phd' in response to an event
JVMDUMP010I Heap dump written to /home/uri/oms-data/heapdump.20211030.224454.69054.0008.phd
JVMDUMP032I JVM requested Java dump using '/home/uri/oms-data/javacore.20211030.224454.69054.0009.txt' in response to an event
JVMDUMP010I Java dump written to /home/uri/oms-data/javacore.20211030.224454.69054.0009.txt
JVMDUMP032I JVM requested Snap dump using '/home/uri/oms-data/Snap.20211030.224454.69054.0010.trc' in response to an event
JVMDUMP010I Snap dump written to /home/uri/oms-data/Snap.20211030.224454.69054.0010.trc
JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "http-nio-8080-exec-5" java.lang.OutOfMemoryError: native memory exhausted
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F7E7B5CA330 Handler2=00007F7E7B327AE0 InaccessibleAddress=FFFFFFFFF7CDBAF0
RDI=0000000000000000 RSI=00000000012BC000 RAX=00000000002557C0 RBX=0000000000010000
RCX=FFFFFFFFFED43F9D RDX=FFFFFFFFF7CDBAF0 R8=FFFFFFFFFED43F9D R9=FFFFFFFFF6A1FCF0
R10=0000000000000000 R11=0000000000000000 R12=000000000063FFFF R13=00000000012BBE60
R14=000000000001C883 R15=00000000012BBE00
RIP=00007F7E7B31F3DB GS=0000 FS=0000 RSP=00007F7E4B56CA40
EFlags=0000000000010206 CS=0033 RBP=00007F7E7B585860 ERR=0000000000000007
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=FFFFFFFFF7CDBAF0
xmm0 ffff000000000000 (f: 0.000000, d: -nan)
xmm1 000039326367396a (f: 1667709312.000000, d: 3.107109e-310)
xmm2 00007f7e4b56c870 (f: 1263978624.000000, d: 6.925832e-310)
xmm3 00007f7e4a6c1c60 (f: 1248599168.000000, d: 6.925832e-310)
xmm4 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
xmm5 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
xmm6 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
xmm7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm8 6f772f396a6e6570 (f: 1785619840.000000, d: 8.787673e+228)
xmm9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/uri/jdks/jdk8u312-b07-openj9/jre/lib/amd64/default/libj9prt29.so
Module_base_address=00007F7E7B2FE000
Target=2_90_20211022_234 (Linux 5.14.13-200.fc34.x86_64)
CPU=amd64 (8 logical CPUs) (0x7ca1a6000 RAM)
----------- Stack Backtrace -----------
(0x00007F7E7B31F3DB [libj9prt29.so+0x213db])
(0x00007F7E7B31ED24 [libj9prt29.so+0x20d24])
(0x00007F7E7B320AD6 [libj9prt29.so+0x22ad6])
(0x00007F7E7B5DDF5C [libj9vm29.so+0x52f5c])
(0x00007F7E7B616B8D [libj9vm29.so+0x8bb8d])
(0x00007F7E7B61A8AA [libj9vm29.so+0x8f8aa])
(0x00007F7E7B328843 [libj9prt29.so+0x2a843])
(0x00007F7E7B616A5A [libj9vm29.so+0x8ba5a])
(0x00007F7E7B0F14F6 [libj9thr29.so+0xe4f6])
(0x00007F7E80FA6299 [libpthread.so.0+0x9299])
clone+0x43 (0x00007F7E80CAF353 [libc.so.6+0x100353])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2021/10/30 22:44:54 - please wait.
JVMDUMP032I JVM requested System dump using '/home/uri/oms-data/core.20211030.224454.69054.0011.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.69420.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 69420 was not found. Expected to find core file with name "/home/uri/oms-data/core.69420"
JVMDUMP032I JVM requested Java dump using '/home/uri/oms-data/javacore.20211030.224454.69054.0012.txt' in response to an event
JVMDUMP010I Java dump written to /home/uri/oms-data/javacore.20211030.224454.69054.0012.txt
JVMDUMP032I JVM requested Snap dump using '/home/uri/oms-data/Snap.20211030.224454.69054.0013.trc' in response to an event
JVMDUMP010I Snap dump written to /home/uri/oms-data/Snap.20211030.224454.69054.0013.trc
JVMDUMP032I JVM requested JIT dump using '/home/uri/oms-data/jitdump.20211030.224454.69054.0014.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'http-nio-8080-exec-5' thread 0x0000000000FA2E00
JVMDUMP010I JIT dump written to /home/uri/oms-data/jitdump.20211030.224454.69054.0014.dmp
JVMDUMP013I Processed dump event "gpf", detail "".
Activity
pshipton commentedon Nov 1, 2021
@urisimchoni can you pls run
jpackcore -x <core>on the first core file and make the result available as well. It collects the OS libraries into a zip file so we can look at the core on other machines and we don't need to use the same OS where the crash occurred.https://www.eclipse.org/openj9/docs/tool_jextract/
pshipton commentedon Nov 1, 2021
@tajila @gacholio
urisimchoni commentedon Nov 1, 2021
Here's a link for core1 with jpackcore -x:
https://drive.google.com/file/d/1g5oDs5zuxeqV7LFYu9nGnixPc0WjswkF/view?usp=sharing
tajila commentedon Nov 1, 2021
Core1
Native stack
Java Stack
tajila commentedon Nov 1, 2021
Core 2
Native stack
pshipton commentedon Nov 1, 2021
The first javacore shows
Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create stack trace (most likely due to lack of OS memory)" receivedpshipton commentedon Nov 1, 2021
There was an attempt to allocate a ridiculous amount of memory, the trace history shows
gacholio commentedon Nov 2, 2021
One scenario that might cause this is if the thread is interrupted during a malloc/free, and the ASGCT call uses malloc/free, resulting in corruption of the malloc heap.
ASGCT is careful not to allocate anything, but I will take another pass over the code to be sure.
gacholio commentedon Nov 3, 2021
The huge memory allocation attempt is the stack walker cache used by
fillInStackTrace. Generally, huge cache sizes indicate a corrupt stack.(sp - stack start) < 0results in a huge positive value.In the core, the stack is completely walkable, which makes it look like the
spvalue was incorrect at the time the cache computation took place. ASGCT does modify thesptemporarily in the case where a JIT frame is on stack, but it's restored before exit.This makes it appear as if the thread is running java while the ASGCT stack modification is in place, which is clearly impossible.
I'll need to think more on the fact that multiple ASGCT are running.
@urisimchoni Is it possible that an ASGCT is interrupted and another ASGCT takes place on the same thread? I'm not sure this is even a problem if it is the case.
urisimchoni commentedon Nov 3, 2021
ASGCT is invoked from a signal handler, and the signal is blocked while the signal handler is executing (default POSIX behavior), so no - an ASGCT cannot be interrupted by agent code that calls ASGCT.
gacholio commentedon Nov 5, 2021
I've taken a look at the code and I don't see any allocations. The signal handler code does some stuff which might not stack well, but I can't blame it yet.
@urisimchoni Can you provide something I can run to reproduce the problem? I need to be able to strip pieces out of ASGCT to see what's causing the issue.
urisimchoni commentedon Nov 5, 2021
Sure, I'll make a stripped-down version of the agent.
Thanks for looking into this!
46 remaining items
gacholio commentedon Jan 14, 2022
async_profilerappears to follow the same pattern as the provided example.urisimchoni commentedon Jan 14, 2022
Re-GetEnv, I think it's possible to make the agent behavior dependent on JVM type. For example we use the presence of "com.ibm.GetOSThreadID" extension to determine the JVM is J9. I don't know what other profilers would think of it but for me it's not an issue.
However, commenting-out GetEnv doesn't solve the issue for me. The If I plant a "return" statement at the beginning of the signal handler, the crash still happens, and If I comment-out the entire signal handler, the crash disappears. This lead me to think that the issue is with allocating lots of memory on the native stack - the signal handler allocates the AGCT buffer on the stack - and indeed if I use a smaller buffer size (max 2 frames instead of 640) the crash disappears.
@gacholio Can you confirm that there might be a problem with allocating a large amount on native stack (640 frames * 16 bytes per frame, 10K I believe)? I think it should be easy to use a thread-local var that points to the buffer, allocate it on thread init and deallocate on thread termination.
gacholio commentedon Jan 15, 2022
10K doesn't sound like very much, but your investigation would suggest it's the issue (even though calling ASGCT will use a bunch of stack as well). The TLS solution sounds safe regardless of which VM you're targetting.
Looks like the VM defaults to a native stack size of 256k. If the native stack is the issue, then adding
-Xmso1Mto the command line should also fix it.I'm not sure how signal handlers work on the various platforms. I was under the impression that they might run on their own stack sometimes. On x86 platforms, when running JIT compiled code, the
RSPregister points into the java stack rather than the native stack. If the signal handler simply runs on top of whereverRSPpoints, then we might need to increase the "slush" area of the java stack (a 32K area beyond the point of java stack overflow used to handle the overflow or possible signal handler execution).OpenJ9 allocates small java stacks initially and grows them as need be (the slush area is the same regardless of the java stack size). The growing behaviour can be disabled by setting the intial stack size to the maximum (
-Xiss1Massuming you're using the default sizes).gacholio commentedon Jan 15, 2022
Any chance you can confirm whether the
JNIEnvis an input or output parameter? If it is needed as input, it would also be simple to stash this in TLS during thread startup to avoid theGetEnvcall in the handler.urisimchoni commentedon Jan 16, 2022
@gacholio it's pretty clear from the HotSpot sources (https://github.com/openjdk/jdk/blob/master/src/hotspot/share/prims/forte.cpp#L564) that this is an input parameter, at least for HotSpot.
By default a signal runs on the stack of the thread that receives it, and probably the stack in X86 is identified by RSP. In any case it's not difficult to move the AGCT buffer to a dynamically-allocated buffer, pointed-to by a thread-local variable.
As far as I'm concerned we can close this issue, I just like to know whether obtaining the JNIEnv will be async-safe.
Thanks!
Uri
gacholio commentedon Jan 18, 2022
I will update GetEnv to avoid using tracepoints, which should make it async-safe for fetching the JNIEnv. I will also update ASGCT to accept the JNIEnv as an incoming parameter, though I may restrict actual use of the call to the current thread while I consider what extra effects inspecting other threads might have.
For additional safety, if you're caching the buffers in TLS during the JVMTI thread start event, you could also cache the JNIEnv and avoid the call entirely in the signal handler (this would work on all VMs).
urisimchoni commentedon Jan 18, 2022
From the hotspot code (see my previous link) it appears that hotspot too requires that the JNIEnv is that of the interrupted thread.
gacholio commentedon Jan 18, 2022
Which makes the parameter rather pointless, but I am updating the call to check it and return failure. All changes should be committed later today.
gacholio commentedon Jan 18, 2022
#14308
gacholio commentedon Jan 27, 2022
All of the OpenJ9 issues have been resolved, so closing.
gacholio commentedon Jun 29, 2022
@urisimchoni Where did you make the TLS buffer change? We're seeing a similar issue with what looks to be async profiler, so I'm wondering if the fix made it into the main repo.
urisimchoni commentedon Jun 29, 2022
No, we kind of dropped the ball on this one, we're avoiding AGCT when using OpenJ9. It's really a small change that we should implement!