Repository navigation
Closed
Labels
Description
When trying to collect trace log using the following options to both -Xjit and -Xaot [1], I ran into a crash in TR_Debug::findLogFile. The prev->_options->_logFileName is invalid [2]. I found that what happened is when jitLatePostProcess calls _debug->findLogFile, it doesn't differentiate whether or not it is for AOT or JIT [3]. It calls both both when it's only parsing for AOT options[4].
One fix I'd propose is to add a default argument isAOT to jitLatePostProcess. The default value is false. In jitLatePostProcess, we could call findLogFile based on which option it is [5].
[1]
export TR_Options="verbose,vlog=vlog,{*WrappingSpliterator.lambda*initPartialTraversalState*,*ArrayListSpliterator.tryAdvance*,*ReferencePipeline*.accept*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=logjit)"
export TR_OptionsAOT="aotrtDebugLevel=30,rtLog=rtLog,{*WrappingSpliterator.lambda*initPartialTraversalState*,*ArrayListSpliterator.tryAdvance*,*ReferencePipeline*.accept*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=logaot)"
[2]
(gdb) bt
#0 0x00007ffff795bf9b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fffefa9e8a0 in TR_Debug::findLogFile (this=<optimized out>, cmdLineOptions=<optimized out>,
optSet=0x7fffe2f30ef0, logFileName=0x7fffe2faba10 "logaot")
at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/ras/OptionsDebug.cpp:86
#2 0x00007fffef727fa4 in OMR::Options::jitLatePostProcess (this=0x7fffe2fab5c0, optionSet=optionSet@entry=0x7fffe2f30ef0,
jitConfig=jitConfig@entry=0x7ffff00b7940, isAOT=isAOT@entry=true)
at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:2321
#3 0x00007fffef728e76 in OMR::Options::latePostProcess (options=0x7fffe2f301c0, jitConfig=jitConfig@entry=0x7ffff00b7940,
isAOT=isAOT@entry=true) at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:1956
#4 0x00007fffef728ff0 in OMR::Options::latePostProcessAOT (jitConfig=jitConfig@entry=0x7ffff00b7940)
at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:1917
#5 0x00007fffef374dfd in aboutToBootstrap (javaVM=javaVM@entry=0x7ffff000f710, jitConfig=0x7ffff00b7940)
at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/compiler/control/rossa.cpp:1880
#6 0x00007fffef35b3eb in J9VMDllMain (vm=0x7ffff000f710, stage=<optimized out>, reserved=<optimized out>)
at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/compiler/control/DLLMain.cpp:419
#7 0x00007ffff5f6f4c1 in runJ9VMDllMain (dllLoadInfo=0x7ffff001e228, userDataTemp=0x7ffff6f785b0)
at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/vm/jvminit.c:3771
(gdb) fr 1
#1 0x00007fffefa9e8a0 in TR_Debug::findLogFile (this=<optimized out>, cmdLineOptions=<optimized out>,
optSet=0x7fffe2f30ef0, logFileName=0x7fffe2faba10 "logaot")
at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/ras/OptionsDebug.cpp:86
86 if (fileName && !STRICMP(logFileName, fileName))
(gdb) print prev->_options->_logFileName
$3 = 0x4000007fffefb879 <error: Cannot access memory at address 0x4000007fffefb879>
[4]
findLogFile: call findLogFile for aotCmdLineOptions logFileName logaot
TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f301c0 optSet 0x7fffe2f30ef0, logFileName logaot
TR_Debug::findLogFile: fileName
findLogFile: call findLogFile for jitCmdLineOptions logFileName logaot
TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f30f30 optSet 0x7fffe2f30ef0, logFileName logaot
TR_Debug::findLogFile: fileName
TR_Debug::findLogFile: prev 0x7fffe2f31b80 fileName 0x4000007fffefb879
[5]
OMR::Options::jitLatePostProcess(TR::OptionSet *optionSet, void * jitConfig, bool isAOT)
...
if (isAOT)
{
_logFile = _debug->findLogFile(TR::Options::getAOTCmdLineOptions(), NULL, optionSet, _logFileName);
}
else
{
_logFile = _debug->findLogFile(NULL, TR::Options::getJITCmdLineOptions(), optionSet, _logFileName);
}
Activity
a7ehuo commentedon Sep 30, 2021
@dsouzai @mpirvu Could you review the proposed fix for this issue? Or any suggestion? Thank you!
mpirvu commentedon Oct 4, 2021
I believe that the idea behind the original code was that, if two different option sets specify the same log name, we want to share the same log file (this needs to be opened only once). This is why we use
TR::FILE *TR_Debug::findLogFile(TR::Options *aotCmdLineOptions, TR::Options *jitCmdLineOptions, TR::OptionSet *optSet, char *logFileName)to search all option sets from both JIT and AOT command line options for a match.
The problem stems from the fact that when we search an option set, that set may not have the options object already created. This is created in https://github.com/eclipse/omr/blob/master/compiler/control/OMROptions.cpp#L1943-L1963.
Maybe we should initialize the
_optionsfield fromTR::OptionSetto NULL and then we can properly test against NULL in https://github.com/eclipse/omr/blob/master/compiler/ras/OptionsDebug.cpp#L79a7ehuo commentedon Oct 4, 2021
I specified the log name to be the same as
log=logfor both JIT and AOT with the existing OMR code. The same crash also happens. The sequence of events are as below [1]. When it tries to access to the previous Options inTR_Debug::findLogFile,OptionSet::setOptionshasn't called on the JITOptionSetyet. It is notNULLand it is still a type ofchar* _optionStringinstead ofTR::Options *_options.OptionSet. _optionsandOptionSet. _optionStringare union [2].getOptionsorgetOptionStringwill never beNULLbecause_optionStringwill always set in constructorOptionSet:: OptionSet(char *s). UnlesssetOptions(TR::Options *o)has been called,getOptions()will return a char pointer to_optionStringthat's implicitly casted asTR::Options *.[1]
[2]
https://github.com/eclipse/omr/blob/1ec11640d4974dc68384c032b7118329bd379d7f/compiler/control/OptionsUtil.hpp#L179-L183
a7ehuo commentedon Oct 5, 2021
@mpirvu, if I understand the suggestion correctly, we could remove the
uniondeclaration onOptionSet. _optionsandOptionSet._optionString, and_optionsshould be initialized asNULLinOptionSet::init.Currently there is no check on if it’s
NULLwhenOptionSet::getOptions()is called. After initializing it asNULL, would it be a concern that we need to checkNULLwherevergetOptions()is called, besides ofTR_Debug::findLogFile? I did a quick test, it seems so farTR_Debug::findLogFilewould be the only place concerned.mpirvu commentedon Oct 5, 2021
ok, I understand better now, that
_optionsfield plays a double role and we cannot initialize to 0. Thus, when we process options sets we cannot tell whether an option set has allocated its options object.Going back to the proposed solution, I think we have the same problem if there are two option sets under AOT of JIT. For instance: -Xaot:{method1}(traceFull,log=log1),{method2}(traceFull,log=log2)
When we process the option set for method1, we would get into
TR_Debug::findLogFileand iterate over all option sets to see if the log file is the same. At that point, the option set for method2 does not have its_optionsfield initialized and may result in a crash (something to be verified).If what I said above is true, we may have no choice but to split that union into 2 separate fields, so that we can have a way to tell when the
_optionsfield is initialized.I believe not because in the old implementation you would get an option string and if you try to use it as an options object you would create the crash we are trying to solve now.
a7ehuo commentedon Oct 5, 2021
I specified two log names (
log=log1,log=log2) under the same AOT env option [1] . It doesn't have any issue. It also works fine if I specify two log names under JIT option.When specifying two log names under AOT, the sequence of events [2] shows before the first time
TR_Debug::findLogFileis called,OptionSet::setOptionshas been called to set upOptionSet._optionsproperly foraotCmdLineOptions. When it gets toTR_Debug::findLogFilethe first time, there is no otheroptionSetunderaotCmdLineOptionsexcept itself (0x7fffe2f2ff60), and there is noOptionSetfromjitCmdLineOptionsto compare with. The second round whenTR_Debug::findLogFileis called, it is also fine because there is noOptionSetfromjitCmdLineOptionsto compare with, and the filename from previousOptionSet(0x7fffe2f2ff60) is also valid becauseOptionSet._options(0x7fffe2fa9e10) has been set up properly.I think the issue is really when both AOT and JIT have logs specified and when one is processed first, the
OptionSet._optionsfor the other one has not been set up properly yet. It wouldn't be an issue when specifying multiple logs under either AOT or JIT, becausesetOptionsis always called for the current option that's being processed before it gets toTR_Debug::findLogFile.I think splitting the union into 2 as separate fields might still be the right way to fix the crash.
[1]
[2]
Split _options and _optionString in OptionSet into separate fields