Repository navigation
Description
Background Context
Through a customer engagement we've identified a bottleneck in OpenJ9's stack walking implementation. Using the apache/logging-log4j2 framework we've built the following simple JMH benchmark which is representative bottleneck in the customer workload:
package org.openj9;
import org.apache.logging.log4j.LogManager;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import java.util.concurrent.TimeUnit;
@Fork(value = 1)
@Warmup(iterations = 10, time = 2000, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 3, time = 1000, timeUnit = TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class Log4jGetLoggerBenchmark {
public static org.apache.logging.log4j.Logger log4jLogger;
@Benchmark
public void testGetLoggerBasic() {
log4jLogger = LogManager.getLogger("JJ.510845011882325196");
}
}
At a high level the LogManager.getLogger API invokes java/lang/StackWalker.walk API to apply a filter which I'll inline here:
public Class<?> getCallerClass(final String fqcn, final String pkg) {
return walker.walk(s -> s.dropWhile(f -> !f.getClassName().equals(fqcn))
.dropWhile(f -> f.getClassName().equals(fqcn))
.dropWhile(f -> !f.getClassName().startsWith(pkg))
.findFirst())
.map(StackWalker.StackFrame::getDeclaringClass).orElse(null);
}
This filter effectively looks for the name of the caller class of the LogManager.getLogger API call. In the above JMH benchmark it is looking for the class name "Log4jGetLoggerBenchmark".
Java 8 vs. Java 11
According to the documentation due to the deprecation of the getCallerClass(int) API in Java 8, the log4j library had to roll out an alternative way of extracting the class name of the caller. It rolled out the StackLocator implementation which is the source of a Java 8 vs. Java 11 regression.
A quick measurement on Linux x86-64 reproduces the delta:
JDK 1.8.0_302, Eclipse OpenJ9 VM, openj9-0.27.0:
Benchmark Mode Cnt Score Error Units
Log4jGetLoggerBenchmark.testGetLoggerBasic thrpt 3 129065.089 ± 2370.504 ops/s
JDK 11.0.12, Eclipse OpenJ9 VM, openj9-0.27.0:
Benchmark Mode Cnt Score Error Units
Log4jGetLoggerBenchmark.testGetLoggerBasic thrpt 3 90105.024 ± 35967.100 ops/s
Profiling
Grabbing a perf record profile with call-graph information of the Java 11 case shows clearly where the bottleneck is:
Samples: 5K of event 'cycles', Event count (approx.): 1803867237
Children Self Command Shared Object Symbol
- 67.15% 4.34% org.openj9.Log4 libjclse29.so [.] Java_java_lang_StackWalker_getImpl
- Java_java_lang_StackWalker_getImpl
- 36.66% j9gc_createJavaLangString
- 16.01% MM_StringTable::hashAtUTF8
- MM_StringTable::hashAt
- hashTableFind
- 14.32% comparatorToEqualFn
stringComparatorFn
+ 3.13% monitor_exit.part.10
+ 2.79% monitor_enter_three_tier.constprop.17
+ 11.57% J9AllocateObject
+ 5.17% walkStackFrames
+ 2.98% getLineNumberForROMClassFromROMMethod
2.62% J9WriteBarrierStore
+ 1.36% getOriginalROMMethod
1.13% MM_GenerationalAccessBarrierComponent::postObjectStore
0.59% MM_StandardAccessBarrier::postObjectStoreImpl
+ 36.66% 13.83% org.openj9.Log4 libj9gc29.so [.] j9gc_createJavaLangString
+ 16.01% 0.05% org.openj9.Log4 libj9gc29.so [.] MM_StringTable::hashAtUTF8
+ 15.97% 0.23% org.openj9.Log4 libj9gc29.so [.] MM_StringTable::hashAt
+ 15.74% 1.21% org.openj9.Log4 libj9gc29.so [.] hashTableFind
+ 14.32% 0.34% org.openj9.Log4 libj9gc29.so [.] comparatorToEqualFn
+ 14.00% 1.78% org.openj9.Log4 libj9gc29.so [.] J9AllocateObject
+ 13.98% 13.96% org.openj9.Log4 libj9gc29.so [.] stringComparatorFn
+ 11.63% 2.57% org.openj9.Log4 libj9gc29.so [.] OMR_GC_AllocateObject
+ 7.96% 0.46% org.openj9.Log4 libj9vm29.so [.] walkStackFrames
+ 7.71% 0.39% org.openj9.Log4 libjclse29.so [.] Java_java_lang_StackWalker_walkWrapperImpl
+ 7.38% 0.75% org.openj9.Log4 libj9jit29.so [.] jitWalkStackFrames
+ 5.31% 0.62% org.openj9.Log4 libj9gc29.so [.] MM_TLHAllocationInterface::allocateObject
The perf report is telling us pretty much to focus on the children inside the Java_java_lang_StackWalker_getImpl function:
| stringObject = utfToStringObject(env, J9ROMMETHOD_NAME(romMethod), J9_STR_INTERN); |
More specifically these calls to utfToStringObject which end up calling j9gc_createJavaLangString. Looking at just that function it has 1/3 of time spent in itself and 2/3 spent in its children (MM_StringTable::hashAt* being the biggest offenders).
Looking further into the annotation of j9gc_createJavaLangString we can see we are interning all of these strings and getting hits in the hash table as there are no ticks following the call to hashAtUTF8 until the rest of the method.
This means we are skipping this entire if statement:
openj9/runtime/gc_base/StringTable.cpp
Lines 637 to 838 in 8c2d799
| if (NULL == result) { | |
| UDATA unicodeLength = 0; | |
| UDATA allocateFlags = J9_ARE_ANY_BITS_SET(stringFlags, J9_STR_INSTRUMENTABLE) | |
| ? J9_GC_ALLOCATE_OBJECT_INSTRUMENTABLE | |
| : J9_GC_ALLOCATE_OBJECT_NON_INSTRUMENTABLE; | |
| if (J9_ARE_ANY_BITS_SET(stringFlags, J9_STR_TENURE | J9_STR_INTERN)) { | |
| allocateFlags |= J9_GC_ALLOCATE_OBJECT_TENURED; | |
| } | |
| J9Class *stringClass = J9VMJAVALANGSTRING_OR_NULL(vm); | |
| result = J9AllocateObject(vmThread, stringClass, allocateFlags); | |
| if (result == NULL) { | |
| goto nomem; | |
| } | |
| if (isUnicode) { | |
| unicodeLength = length / 2; | |
| } else { | |
| if (isASCII) { | |
| unicodeLength = length; | |
| } else { | |
| UDATA tempLength = length; | |
| U_8 *tempData = data; | |
| while (tempLength != 0) { | |
| U_16 unicode = 0; | |
| UDATA consumed = VM_VMHelpers::decodeUTF8CharN(tempData, &unicode, tempLength); | |
| Assert_GC_true_with_message2(MM_EnvironmentBase::getEnvironment(vmThread->omrVMThread), 0 != consumed, "Invalid UTF8 character at %p, %zu", tempData, tempLength); | |
| tempData += consumed; | |
| tempLength -= consumed; | |
| unicodeLength += 1; | |
| } | |
| } | |
| } | |
| PUSH_OBJECT_IN_SPECIAL_FRAME(vmThread, result); | |
| if (J9_ARE_ANY_BITS_SET(vm->runtimeFlags, J9_RUNTIME_STRING_BYTE_ARRAY)) { | |
| if (compressStrings && isASCIIorLatin1) { | |
| charArray = J9AllocateIndexableObject(vmThread, vm->byteArrayClass, (U_32) unicodeLength, allocateFlags); | |
| } else { | |
| charArray = J9AllocateIndexableObject(vmThread, vm->byteArrayClass, (U_32) unicodeLength * 2, allocateFlags); | |
| } | |
| } else { | |
| if (compressStrings && isASCII) { | |
| charArray = J9AllocateIndexableObject(vmThread, vm->charArrayClass, (U_32) (unicodeLength + 1) / 2, allocateFlags); | |
| } else { | |
| charArray = J9AllocateIndexableObject(vmThread, vm->charArrayClass, (U_32) unicodeLength, allocateFlags); | |
| } | |
| } | |
| result = POP_OBJECT_IN_SPECIAL_FRAME(vmThread); | |
| if (charArray == NULL) { | |
| goto nomem; | |
| } | |
| if (isUnicode) { | |
| U_16 * unicodeData = (U_16 *) data; | |
| if (compressStrings && isASCIIorLatin1) { | |
| for (UDATA i = 0; i < unicodeLength; ++i) { | |
| J9JAVAARRAYOFBYTE_STORE(vmThread, charArray, i, (I_8)unicodeData[i]); | |
| } | |
| } else { | |
| for (UDATA i = 0; i < unicodeLength; ++i) { | |
| J9JAVAARRAYOFCHAR_STORE(vmThread, charArray, i, unicodeData[i]); | |
| } | |
| } | |
| } else { | |
| if (compressStrings && isASCIIorLatin1) { | |
| UDATA lastSlash = 0; | |
| if (translateSlashes) { | |
| if (isASCII) { | |
| for (UDATA i = 0; i < length; ++i) { | |
| U_8 c = data[i]; | |
| if ('/' == c) { | |
| lastSlash = i; | |
| c = '.'; | |
| } | |
| J9JAVAARRAYOFBYTE_STORE(vmThread, charArray, i, c); | |
| } | |
| } else { | |
| lastSlash = storeLatin1ByteArrayhelper(vmThread, data, length, charArray, true); | |
| } | |
| } else { | |
| if (isASCII) { | |
| for (UDATA i = 0; i < length; ++i) { | |
| J9JAVAARRAYOFBYTE_STORE(vmThread, charArray, i, data[i]); | |
| } | |
| } else { | |
| lastSlash = storeLatin1ByteArrayhelper(vmThread, data, length, charArray, false); | |
| } | |
| } | |
| /* Anonymous classes have the following name format [className]/[ROMADDRESS], so we have to fix up the name | |
| * because the previous loops have converted '/' to '.' already. | |
| */ | |
| if (anonClassName && (0 != lastSlash)) { | |
| J9JAVAARRAYOFBYTE_STORE(vmThread, charArray, lastSlash, (U_8)'/'); | |
| } | |
| } else { | |
| UDATA lastSlash = 0; | |
| if (isASCII) { | |
| if (translateSlashes) { | |
| for (UDATA i = 0; i < length; ++i) { | |
| U_8 c = data[i]; | |
| if ('/' == c) { | |
| lastSlash = i; | |
| c = '.'; | |
| } | |
| J9JAVAARRAYOFCHAR_STORE(vmThread, charArray, i, c); | |
| } | |
| } else { | |
| for (UDATA i = 0; i < length; ++i) { | |
| J9JAVAARRAYOFCHAR_STORE(vmThread, charArray, i, data[i]); | |
| } | |
| } | |
| } else { | |
| UDATA writeIndex = 0; | |
| UDATA tempLength = length; | |
| U_8* tempData = data; | |
| while (tempLength > 0) { | |
| U_16 unicode = 0; | |
| UDATA consumed = VM_VMHelpers::decodeUTF8Char(tempData, &unicode); | |
| if (translateSlashes) { | |
| if ((U_16)'/' == unicode) { | |
| lastSlash = writeIndex; | |
| unicode = (U_16)'.'; | |
| } | |
| } | |
| J9JAVAARRAYOFCHAR_STORE(vmThread, charArray, writeIndex, unicode); | |
| writeIndex += 1; | |
| tempData += consumed; | |
| tempLength -= consumed; | |
| } | |
| } | |
| /* Anonymous classes have the following name format [className]/[ROMADDRESS], so we have to fix up the name | |
| * because the previous loops have converted '/' to '.' already. | |
| */ | |
| if (anonClassName && (0 != lastSlash)) { | |
| J9JAVAARRAYOFCHAR_STORE(vmThread, charArray, lastSlash, (U_16)'/'); | |
| } | |
| } | |
| } | |
| J9VMJAVALANGSTRING_SET_VALUE(vmThread, result, charArray); | |
| if (compressStrings) { | |
| if (isASCIIorLatin1) { | |
| if (J2SE_VERSION(vm) >= J2SE_V11) { | |
| J9VMJAVALANGSTRING_SET_CODER(vmThread, result, 0); | |
| } else { | |
| J9VMJAVALANGSTRING_SET_COUNT(vmThread, result, (I_32)unicodeLength); | |
| } | |
| } else { | |
| if (J2SE_VERSION(vm) >= J2SE_V11) { | |
| J9VMJAVALANGSTRING_SET_CODER(vmThread, result, 1); | |
| } else { | |
| J9VMJAVALANGSTRING_SET_COUNT(vmThread, result, (I_32)unicodeLength | (I_32)0x80000000); | |
| } | |
| if (J9VMJAVALANGSTRING_COMPRESSIONFLAG(vmThread, stringClass) == 0) { | |
| /* | |
| * jitHookClassPreinitialize will process initialization events for String compression sideEffectGuards | |
| * so we must initialize the class if this is the first time we are loading it | |
| */ | |
| J9Class* flagClass = vmFuncs->internalFindKnownClass(vmThread, J9VMCONSTANTPOOL_JAVALANGSTRINGSTRINGCOMPRESSIONFLAG, J9_FINDKNOWNCLASS_FLAG_INITIALIZE); | |
| if (NULL == flagClass) { | |
| goto nomem; | |
| } else { | |
| j9object_t flag = J9AllocateObject(vmThread, flagClass, allocateFlags); | |
| if (NULL == flag) { | |
| goto nomem; | |
| } | |
| J9VMJAVALANGSTRING_SET_COMPRESSIONFLAG(vmThread, stringClass, flag); | |
| } | |
| } | |
| } | |
| } else { | |
| if (J2SE_VERSION(vm) >= J2SE_V11) { | |
| J9VMJAVALANGSTRING_SET_CODER(vmThread, result, 1); | |
| } else { | |
| J9VMJAVALANGSTRING_SET_COUNT(vmThread, result, (I_32)unicodeLength); | |
| } | |
| } | |
| MM_AtomicOperations::writeBarrier(); | |
| /* Intern the String if requested */ | |
| if (internString) { | |
| result = stringTable->addStringToInternTable(vmThread, result); | |
| if (NULL == result) { | |
| goto nomem; | |
| } | |
| } | |
| } |
All this makes sense.
The bottleneck appears to be from attempting to create the strings representing the method name and signature. Because previous invocations have interned such strings we are doing full string comparisons and matching entries in the interned string hash table. This is why we see lots of ticks coming from stringComparatorFn. Additionally we are recomputing the hash values of the UTF8 strings so we can carry out a hash table lookup every time.
Previous Work
We do cache the class name in the java/lang/Class object for fast lookup:
openj9/jcl/src/java.base/share/classes/java/lang/Class.java
Lines 177 to 178 in d3ef784
| /* Cache filename on Class to avoid repeated lookups / allocations in stack traces */ | |
| private transient String fileNameString; |
As can be seen here in the getImpl stack walker code:
openj9/runtime/jcl/common/java_lang_StackWalker.cpp
Lines 202 to 217 in d3ef784
| stringObject = J9VMJAVALANGCLASS_CLASSNAMESTRING(vmThread, J9VM_J9CLASS_TO_HEAPCLASS(ramClass)); | |
| if (stringObject == NULL) { | |
| UDATA flags = J9_STR_XLAT; | |
| if (J9_ARE_ANY_BITS_SET(romClass->extraModifiers, J9AccClassAnonClass | J9AccClassHidden)) { | |
| flags |= J9_STR_ANON_CLASS_NAME; | |
| } | |
| J9UTF8 *nameUTF = J9ROMCLASS_CLASSNAME(romClass); | |
| stringObject = utfToStringObject(env, nameUTF, flags); | |
| if (VM_VMHelpers::exceptionPending(vmThread)) { | |
| goto _pop_frame; | |
| } | |
| /* Class name was interned so it's safe to write it back to the Class Object */ | |
| Assert_JCL_false(J9CLASS_IS_ARRAY(ramClass)); | |
| J9VMJAVALANGCLASS_SET_CLASSNAMESTRING(vmThread, J9VM_J9CLASS_TO_HEAPCLASS(ramClass), stringObject); | |
| } | |
| J9VMJAVALANGSTACKWALKERSTACKFRAMEIMPL_SET_CLASSNAME(vmThread, PEEK_OBJECT_IN_SPECIAL_FRAME(vmThread, 0), stringObject); |
This work was carried out by @DanHeidinga as part of #7776. Unfortunately there isn’t a natural fit for the method and signature strings as there is no backing object in that case.
Additional Tracing
We took a broad look at the APIs and how things piece together. From our understanding we construct the walk state for the stack walk in java/lang/StackWalker.walkWrapperImpl and much deeper down the stack we call java/lang/StackWalker.getImpl which actually populates the StackFrame object with the class name, method name, and method signature. This can be confirmed via a method trace:
18:27:42.645 0x1aa00 mt.10 < java/lang/StackWalker.getImpl(J)Ljava/lang/StackWalker$StackFrameImpl; compiled static method
18:27:42.645 0x1aa00 mt.9 < java/lang/StackWalker.lambda$walkImpl$3(JLjava/lang/StackWalker$StackFrame;)Ljava/lang/StackWalker$StackFrame; bytecode static method
18:27:42.645 0x1aa00 mt.3 > java/lang/StackWalker.lambda$walkImpl$2(Ljava/lang/StackWalker$StackFrame;)Z bytecode static method
18:27:42.645 0x1aa00 j9trc_aux.0 - jstacktrace:
18:27:42.645 0x1aa00 j9trc_aux.1 - [1] java.lang.StackWalker.lambda$walkImpl$2 (StackWalker.java:206)
18:27:42.645 0x1aa00 j9trc_aux.1 - [2] java.lang.StackWalker$$Lambda$73.0x0000000000000000.test (Bytecode PC: 4)
18:27:42.645 0x1aa00 j9trc_aux.1 - [3] java.util.stream.Stream$2.tryAdvance (Stream.java:1296)
18:27:42.645 0x1aa00 j9trc_aux.1 - [4] java.util.stream.ReferencePipeline.forEachWithCancel (ReferencePipeline.java:127) (Compiled Code)
18:27:42.645 0x1aa00 j9trc_aux.1 - [5] java.util.stream.AbstractPipeline.copyIntoWithCancel (AbstractPipeline.java:515) (Compiled Code)
18:27:42.645 0x1aa00 j9trc_aux.1 - [6] java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:501) (Compiled Code)
18:27:42.645 0x1aa00 j9trc_aux.1 - [7] java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:487) (Compiled Code)
18:27:42.645 0x1aa00 j9trc_aux.1 - [8] java.util.stream.FindOps$FindOp.evaluateSequential (FindOps.java:150)
18:27:42.645 0x1aa00 j9trc_aux.1 - [9] java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:239)
18:27:42.645 0x1aa00 j9trc_aux.1 - [10] java.util.stream.ReferencePipeline.findFirst (ReferencePipeline.java:543)
18:27:42.645 0x1aa00 j9trc_aux.1 - [11] org.apache.logging.log4j.util.StackLocator.lambda$getCallerClass$3 (StackLocator.java:50)
18:27:42.645 0x1aa00 j9trc_aux.1 - [12] org.apache.logging.log4j.util.StackLocator$$Lambda$72.0x0000000000000000.apply (Bytecode PC: 12)
18:27:42.645 0x1aa00 j9trc_aux.1 - [13] java.lang.StackWalker.walkImpl (StackWalker.java:207)
18:27:42.645 0x1aa00 j9trc_aux.1 - [14] java.lang.StackWalker.walkWrapperImpl (Native Method)
18:27:42.645 0x1aa00 j9trc_aux.1 - [15] java.lang.StackWalker.walk (StackWalker.java:223)
18:27:42.645 0x1aa00 j9trc_aux.1 - [16] org.apache.logging.log4j.util.StackLocator.getCallerClass (StackLocator.java:48)
18:27:42.645 0x1aa00 j9trc_aux.1 - [17] org.apache.logging.log4j.util.StackLocatorUtil.getCallerClass (StackLocatorUtil.java:60)
18:27:42.645 0x1aa00 j9trc_aux.1 - [18] org.apache.logging.log4j.util.StackLocatorUtil.getCallerClass (StackLocatorUtil.java:54)
18:27:42.645 0x1aa00 j9trc_aux.1 - [19] org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext (ClassLoaderContextSelector.java:138)
18:27:42.645 0x1aa00 j9trc_aux.1 - [20] org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext (ClassLoaderContextSelector.java:123)
18:27:42.645 0x1aa00 j9trc_aux.1 - [21] org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext (Log4jContextFactory.java:230)
18:27:42.645 0x1aa00 j9trc_aux.1 - [22] org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext (Log4jContextFactory.java:47)
18:27:42.645 0x1aa00 j9trc_aux.1 - [23] org.apache.logging.log4j.LogManager.getContext (LogManager.java:176)
18:27:42.645 0x1aa00 j9trc_aux.1 - [24] org.apache.logging.log4j.LogManager.getLogger (LogManager.java:666)
18:27:42.645 0x1aa00 j9trc_aux.1 - [25] Test.foo (Test.java:11)
18:27:42.645 0x1aa00 j9trc_aux.1 - [26] Test.main (Test.java:7)
An important detail is that the walkstate was constructed in frame [14] in the above snippet, so the first time we call getImpl the class + method + signature that gets populated is actually frame [16]. We don’t fully understand why it is not [15] but the data shows it clearly. We placed an NLS trace inside of the native implementation of Java_java_lang_StackWalker_getImpl and here is what we see:
JVMDUMP053I org/apache/logging/log4j/util/StackLocator.getCallerClass(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/Class;
JVMDUMP053I org/apache/logging/log4j/util/StackLocatorUtil.getCallerClass(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/Class;
JVMDUMP053I org/apache/logging/log4j/util/StackLocatorUtil.getCallerClass(Ljava/lang/String;)Ljava/lang/Class;
JVMDUMP053I org/apache/logging/log4j/core/selector/ClassLoaderContextSelector.getContext(Ljava/lang/String;Ljava/lang/ClassLoader;Ljava/util/Map$Entry;ZLjava/net/URI;)Lorg/apache/logging/log4j/core/LoggerContext;
JVMDUMP053I org/apache/logging/log4j/core/selector/ClassLoaderContextSelector.getContext(Ljava/lang/String;Ljava/lang/ClassLoader;ZLjava/net/URI;)Lorg/apache/logging/log4j/core/LoggerContext;
JVMDUMP053I org/apache/logging/log4j/core/impl/Log4jContextFactory.getContext(Ljava/lang/String;Ljava/lang/ClassLoader;Ljava/lang/Object;ZLjava/net/URI;Ljava/lang/String;)Lorg/apache/logging/log4j/core/LoggerContext;
JVMDUMP053I org/apache/logging/log4j/core/impl/Log4jContextFactory.getContext(Ljava/lang/String;Ljava/lang/ClassLoader;Ljava/lang/Object;ZLjava/net/URI;Ljava/lang/String;)Lorg/apache/logging/log4j/spi/LoggerContext;
JVMDUMP053I org/apache/logging/log4j/LogManager.getContext(Z)Lorg/apache/logging/log4j/spi/LoggerContext;
JVMDUMP053I org/apache/logging/log4j/LogManager.getLogger(Ljava/lang/String;)Lorg/apache/logging/log4j/Logger;
JVMDUMP053I Test.foo()V
This matches that of the method trace output. It appears there is a total of 9 frames that get traversed before the filter from log4j matches the frame we are looking for. This number is static because the callees never change.
Potential Solutions
Ideally we'd like to eliminate the cost of calling j9gc_createJavaLangString and the various hashing/hash table lookup APIs. This is possible since 9/10 stack frames in this case will always be the same and the strings representing the methods and signatures are interned. A natural solution is to use a cache for these objects.
- Option 1: Introduce a small (~10ish element) thread local least recently used (LRU) cache mapping ROM method pointers to a pair of
java.lang.Stringobjects representing the method name and signature. This cache would be populated and used inJava_java_lang_StackWalker_getImplfor fast retrieval of these values which should eliminate much of the performance cost. - Option 2: Same as above except using a global cache. This will require some form of locking.
- Option 3: Create a hashmap on the Java side in
java/lang/Classwhich is lazily populated with the key being alongrepresenting the ROM method pointer, and the value being the pair of method name and signature. This solution allows us to reuse such a cache in other places within the JVM, i.e.getStackTraceAPI. - Option 4: Rather than using a cache, attempt to recover some of the performance by improving the hashing functions, and the string comparison functions. Perhaps caching the hash of the UTF8 string on the RAM/ROM method could save cycles as well.
- Option 5: Other suggestions?
Activity
fjeremic commentedon Aug 6, 2021
Tagging @tajila @pshipton @DanHeidinga @gacholio @joransiu @amicic @vijaysun-omr for discussion on which direction we should work towards to improve this workload and satisfy the customer request to recover the Java 8 vs. Java 11 performance gap.
amicic commentedon Aug 6, 2021
We have an example of thread local caching that we could follow: objectMonitorLookupCache.
I checked what kind of mapping it uses, and it's not LRU, but hash based (macro J9_OBJECT_MONITOR_LOOKUP_SLOT). There is no collision resolutions, the old entry just gets evicted (hence the whole cache is of fixed size).
This is possibly good enough, even in this case. (Or even better than LRU - if the cache is more than 8 or so entries, while being very precise, it becomes very expensive to maintain).
The key for a frame, from what I understand, should consists of class address, method address and bytecodeOffset, which I hope should be xor-ed to determine the index in the table.
Monitor table, specifically, is 32 entries. Might be reasonably start here, too. Bottom line, due to lack of collision resolution, we need to pick relatively large table (something like 3x bigger from what we expect as the number of hot frames), which, if footprint is a concern, can be allocated on first use.
As Filip mentioned, we can consider global cache too, if we observe relatively flat and wide profile of frames being used. We observed increase (from J8 to J11) of 6000+ entries in string table. We add ~3 strings per frame object, so seems like 2000+ unique frames (sounds high to me, though). But we don't have any string table access stats (so no info from here, if there is small group of hot frames or not).
BTW, object monitor caching uses 2-tier caching (both global and thread local).
GC consideration.... Similarly to monitor table, the cache would be probably a weak root - cleared at the end of GC for frames that are not referenced from elsewhere (if a reference to them can even ever escape?). Could be even a hard root (to avoid rebuilding it after each GC), or a combination (hard at local GCs and weak at global GCs).
DanHeidinga commentedon Aug 7, 2021
Option 4 makes a lot of sense to me as the string intern table is already a global cache. Speeding up the lookup in the cache will provide an improvement to a lot of code that relies on interned strings. In particular, we recognized in #7776 that
This accounts for the
- 14.32% comparatorToEqualFn / stringComparatorFncost related to lookup.The other consideration for caching is to look at the UTF8's for method name and signatures. We attempt to cache UTF8's in multiple places (all class in the SCC, per class loader tables) that, assuming those caches are working and the common method name and signature UTF8s are coming from the SCC or the same classloader, mapping UTF8->String would be a win. If the cache used a ptr comparison for the equals function, and kept the cache local in one of the
userDataslots on theJ9StackWalkState, it would be reasonable efficient and have a higher hit rate then aJ9ROMMethod-based cachefjeremic commentedon Aug 9, 2021
FWIW I did a quick and dirty hack to add a cache in
Java_java_lang_StackWalker_getImpl. It seems we can get 2x performance here by caching:https://github.com/fjeremic/openj9/commit/c32d8c7aa2a258f19899c4d7e803390035555326
fjeremic commentedon Aug 9, 2021
We also compared dumped strings from the intern table on Java 8 vs. Java 11. Some high level stats:
There are quite a bit of what looks like package/module names in there for Java 11. Looks like method names of some pretty weird classes, like
getHostCpuClockSpeedetc. are all from OpenJ9 specific classes. There are also only 6 string of file names:We also compared the string table of an empty benchmark, i.e. without the call to
LogManager.getLogger("JJ.510845011882325196");as seen in the OP. Here are the high level stats for that:This means we have an increase in Java 8 from 1905 -> 2012 and an increase in Java 11 from 3640 -> 6169 strings from just making the call to
LogManager.getLogger. Hence log4j is responsible for adding ~2500 strings to the intern table in Java 11 by executing different code, but a large majority of those strings are not related to the stack walking which is where the application spends most of the time in. Only a handful of strings in the intern table are related to stack walking.pshipton commentedon Aug 9, 2021
Perhaps the strings created for stack walking shouldn't be interned.
fjeremic commentedon Aug 9, 2021
That proves to be even more expensive than the string table lookup:
pshipton commentedon Aug 9, 2021
Why is the Error so big on jdk11?
fjeremic commentedon Aug 9, 2021
Not sure how that is calculated. It looks to be from the confidence interval. The 99.9% confidence interval may be misleading. Here is the full output:
The standard deviation is only 2670 (~2% of the mean score) and the results are quite stable and reproducible.
fjeremic commentedon Aug 9, 2021
@DanHeidinga I'm trying to understand your suggestion here as a potential option.
If I understand correctly the current string intern table implemented in StringTable.cpp is effectively a
map<int,j9object_t>which maps computed integer hashcode values to string objects. Thej9gc_createJavaLangStringAPI computes the hashcode of the input UTF8 data and looks it up in the string stable and returns if there is a cache hit. As part of the table lookup if two strings have equal hashcode then we have to do character-by-character comparison to prove equality. This is why we have so many ticks instringComparatorFn, i.e. because we are generating the same stack backtrace and are hitting cache hits on lookup.Is your suggestion to have a separate cache (ex. StringUTF8Table.cpp) which implements a
map<J9UTF8*,j9object_t>which maps theJ9UTF8data stored in theJ9ROMMethodto an interned string object, such that character-by-character comparison viastringComparatorFnis not needed at all since it would have already been performed as part of interning the original string? This cache could then be reused in other parts of the VM which createjava.lang.Stringobjects based off ofJ9UTF8*.DanHeidinga commentedon Aug 9, 2021
That's pretty close. This suggestion was that rather than building a global cache (ex. StringUTF8Table.cpp) is that we add a local cache in the StackWalker iterator that implements
map<J9UTF8*,j9object_t>so each UTF8 (method name, signature) only has allocate 1 String object.We probably don't want the footprint increase that a global cache would cause but we a short lived cache would mean we're able to do a ptr compare to find the right string and would have reasonable hit rate for a stack trace like:
where "getContext" is used 4 or 5 times, "getCallerClass" 3 times, and various signatures would also match.
DanHeidinga commentedon Aug 9, 2021
My other suggestion was the
Option 4makes a lot of sense to pursue and would provide global benefits. Primarily by looking at thestringComparatorFnwhich is more complex than it needs to be in the common case to support falling back to an AVL tree in the uncommon case.gacholio commentedon Aug 10, 2021
I don't like the idea of using methods to hold the hashes. If anything, we could extend the J9UTF8 structure to hold the hash.
46 remaining items