Skip to content

Crash when specifying log name in both -Xjit and -Xaot  #6189

Closed

Description

@a7ehuo

When trying to collect trace log using the following options to both -Xjit and -Xaot [1], I ran into a crash in TR_Debug::findLogFile. The prev->_options->_logFileName is invalid [2]. I found that what happened is when jitLatePostProcess calls _debug->findLogFile, it doesn't differentiate whether or not it is for AOT or JIT [3]. It calls both both when it's only parsing for AOT options[4].

One fix I'd propose is to add a default argument isAOT to jitLatePostProcess. The default value is false. In jitLatePostProcess, we could call findLogFile based on which option it is [5].

[1]

export TR_Options="verbose,vlog=vlog,{*WrappingSpliterator.lambda*initPartialTraversalState*,*ArrayListSpliterator.tryAdvance*,*ReferencePipeline*.accept*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=logjit)"

export TR_OptionsAOT="aotrtDebugLevel=30,rtLog=rtLog,{*WrappingSpliterator.lambda*initPartialTraversalState*,*ArrayListSpliterator.tryAdvance*,*ReferencePipeline*.accept*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=logaot)"

[2]

(gdb) bt
#0  0x00007ffff795bf9b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fffefa9e8a0 in TR_Debug::findLogFile (this=<optimized out>, cmdLineOptions=<optimized out>, 
    optSet=0x7fffe2f30ef0, logFileName=0x7fffe2faba10 "logaot")
    at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/ras/OptionsDebug.cpp:86
#2  0x00007fffef727fa4 in OMR::Options::jitLatePostProcess (this=0x7fffe2fab5c0, optionSet=optionSet@entry=0x7fffe2f30ef0, 
    jitConfig=jitConfig@entry=0x7ffff00b7940, isAOT=isAOT@entry=true)
    at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:2321
#3  0x00007fffef728e76 in OMR::Options::latePostProcess (options=0x7fffe2f301c0, jitConfig=jitConfig@entry=0x7ffff00b7940, 
    isAOT=isAOT@entry=true) at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:1956
#4  0x00007fffef728ff0 in OMR::Options::latePostProcessAOT (jitConfig=jitConfig@entry=0x7ffff00b7940)
    at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/control/OMROptions.cpp:1917
#5  0x00007fffef374dfd in aboutToBootstrap (javaVM=javaVM@entry=0x7ffff000f710, jitConfig=0x7ffff00b7940)
    at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/compiler/control/rossa.cpp:1880
#6  0x00007fffef35b3eb in J9VMDllMain (vm=0x7ffff000f710, stage=<optimized out>, reserved=<optimized out>)
    at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/compiler/control/DLLMain.cpp:419
#7  0x00007ffff5f6f4c1 in runJ9VMDllMain (dllLoadInfo=0x7ffff001e228, userDataTemp=0x7ffff6f785b0)
    at /root/home/ahuo/src/openj9-openjdk-jdk/openj9/runtime/vm/jvminit.c:3771

(gdb) fr 1
#1  0x00007fffefa9e8a0 in TR_Debug::findLogFile (this=<optimized out>, cmdLineOptions=<optimized out>, 
    optSet=0x7fffe2f30ef0, logFileName=0x7fffe2faba10 "logaot")
    at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/ras/OptionsDebug.cpp:86
86	         if (fileName && !STRICMP(logFileName, fileName))

(gdb) print prev->_options->_logFileName
$3 = 0x4000007fffefb879 <error: Cannot access memory at address 0x4000007fffefb879>

[3]
https://github.com/eclipse/omr/blob/1ec11640d4974dc68384c032b7118329bd379d7f/compiler/control/OMROptions.cpp#L2306

[4]

findLogFile: call findLogFile for aotCmdLineOptions logFileName logaot
TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f301c0 optSet 0x7fffe2f30ef0, logFileName logaot
TR_Debug::findLogFile: fileName 
findLogFile: call findLogFile for jitCmdLineOptions logFileName logaot
TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f30f30 optSet 0x7fffe2f30ef0, logFileName logaot
TR_Debug::findLogFile: fileName 
TR_Debug::findLogFile: prev 0x7fffe2f31b80 fileName 0x4000007fffefb879

[5]

OMR::Options::jitLatePostProcess(TR::OptionSet *optionSet, void * jitConfig, bool isAOT)
...
if (isAOT)
   {
   _logFile = _debug->findLogFile(TR::Options::getAOTCmdLineOptions(), NULL, optionSet, _logFileName);
   }
else
   {
   _logFile = _debug->findLogFile(NULL, TR::Options::getJITCmdLineOptions(), optionSet, _logFileName);
   }

Activity

  1. a7ehuo commented on Sep 30, 2021

    @a7ehuo
    ContributorAuthor

    @dsouzai @mpirvu Could you review the proposed fix for this issue? Or any suggestion? Thank you!

  2. mpirvu commented on Oct 4, 2021

    @mpirvu
    Contributor

    I believe that the idea behind the original code was that, if two different option sets specify the same log name, we want to share the same log file (this needs to be opened only once). This is why we use
    TR::FILE *TR_Debug::findLogFile(TR::Options *aotCmdLineOptions, TR::Options *jitCmdLineOptions, TR::OptionSet *optSet, char *logFileName)
    to search all option sets from both JIT and AOT command line options for a match.
    The problem stems from the fact that when we search an option set, that set may not have the options object already created. This is created in https://github.com/eclipse/omr/blob/master/compiler/control/OMROptions.cpp#L1943-L1963.
    Maybe we should initialize the _options field from TR::OptionSet to NULL and then we can properly test against NULL in https://github.com/eclipse/omr/blob/master/compiler/ras/OptionsDebug.cpp#L79

  3. a7ehuo commented on Oct 4, 2021

    @a7ehuo
    ContributorAuthor

    I specified the log name to be the same as log=log for both JIT and AOT with the existing OMR code. The same crash also happens. The sequence of events are as below [1]. When it tries to access to the previous Options in TR_Debug::findLogFile, OptionSet::setOptions hasn't called on the JIT OptionSet yet. It is not NULL and it is still a type of char* _optionString instead of TR::Options *_options.

    OptionSet. _options and OptionSet. _optionString are union [2]. getOptions or getOptionString will never be NULL because _optionString will always set in constructor OptionSet:: OptionSet(char *s). Unless setOptions(TR::Options *o) has been called, getOptions() will return a char pointer to _optionString that's implicitly casted as TR::Options *.

    [1]

    Options::processOptionsAOT
    Options::processOptionSet isAOT addOptionSet newSet 0x7fffe2f2fef0 newSet->getOptions() 0x7ffff00b1f96 startOptString 0x7ffff00b1f96
    Options::processOptionsJIT
    Options::processOptionSet isJIT addOptionSet newSet 0x7fffe2f30b80 newSet->getOptions() 0x7ffff00be299 startOptString 0x7ffff00be299
    …
    Options::latePostProcess: isAOT 1 options 0x7fffe2f2f1c0 optionSet 0x7fffe2f2fef0 optionSet->getOptions 0x7ffff00b1f96 
    Options::latePostProcess: isAOT 1 options 0x7fffe2f2f1c0 optionSet 0x7fffe2f2fef0 optionSet newOptions 0x7fffe2faa5c0 
    OptionSet::setOptions: _options 0x7ffff00b1f96 -> 0x7fffe2faa5c0
    …
    jitLatePostProcess: isAOT this 0x7fffe2faa5c0 getAOTCmdLineOptions 0x7fffe2f2f1c0 getJITCmdLineOptions 0x7fffe2f2ff30 optionSet 0x7fffe2f2fef0 _logFileName 0x7fffe2faaa10 log
    ==> Crash in TR_Debug::findLogFile because OptionSet::setOptions hasn’t called on the JIT options yet. It is still char* _optionString instead of TR::Options *_options
    
    (gdb) fr 1
    #1  0x00007fffefa9dd90 in TR_Debug::findLogFile (this=<optimized out>, cmdLineOptions=<optimized out>, 
        optSet=0x7fffe2f2fef0, logFileName=0x7fffe2faaa10 "log")
        at /root/home/ahuo/src/openj9-openjdk-jdk/omr/compiler/ras/OptionsDebug.cpp:86
    86	         if (fileName && !STRICMP(logFileName, fileName))
    (gdb) print prev
    $1 = (TR::OptionSet *) 0x7fffe2f30b80
    (gdb) print prev->_options
    $2 = (TR::Options *) 0x7ffff00be299
    (gdb) print prev->_options->_logFileName
    $3 = 0x4000007fffefb86e <error: Cannot access memory at address 0x4000007fffefb86e>
    (gdb) print (char*)prev->_options
    $4 = 0x7ffff00be299 "traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=log)"
    
    

    [2]
    https://github.com/eclipse/omr/blob/1ec11640d4974dc68384c032b7118329bd379d7f/compiler/control/OptionsUtil.hpp#L179-L183

  4. a7ehuo commented on Oct 5, 2021

    @a7ehuo
    ContributorAuthor

    Maybe we should initialize the _options field from TR::OptionSet to NULL

    @mpirvu, if I understand the suggestion correctly, we could remove the union declaration on OptionSet. _options and OptionSet._optionString, and _options should be initialized as NULL in OptionSet::init.

    Currently there is no check on if it’s NULL when OptionSet::getOptions() is called. After initializing it as NULL, would it be a concern that we need to check NULL wherever getOptions() is called, besides of TR_Debug::findLogFile? I did a quick test, it seems so far TR_Debug::findLogFile would be the only place concerned.

  5. mpirvu commented on Oct 5, 2021

    @mpirvu
    Contributor

    ok, I understand better now, that _options field plays a double role and we cannot initialize to 0. Thus, when we process options sets we cannot tell whether an option set has allocated its options object.

    Going back to the proposed solution, I think we have the same problem if there are two option sets under AOT of JIT. For instance: -Xaot:{method1}(traceFull,log=log1),{method2}(traceFull,log=log2)
    When we process the option set for method1, we would get into TR_Debug::findLogFile and iterate over all option sets to see if the log file is the same. At that point, the option set for method2 does not have its _options field initialized and may result in a crash (something to be verified).

    If what I said above is true, we may have no choice but to split that union into 2 separate fields, so that we can have a way to tell when the _options field is initialized.

    After initializing it as NULL, would it be a concern that we need to check NULL wherever getOptions() is called,

    I believe not because in the old implementation you would get an option string and if you try to use it as an options object you would create the crash we are trying to solve now.

  6. a7ehuo commented on Oct 5, 2021

    @a7ehuo
    ContributorAuthor

    I think we have the same problem if there are two option sets under AOT of JIT. For instance: -Xaot:{method1}(traceFull,log=log1),{method2}(traceFull,log=log2)

    I specified two log names (log=log1, log=log2) under the same AOT env option [1] . It doesn't have any issue. It also works fine if I specify two log names under JIT option.

    When specifying two log names under AOT, the sequence of events [2] shows before the first time TR_Debug::findLogFile is called, OptionSet::setOptions has been called to set up OptionSet._options properly for aotCmdLineOptions. When it gets to TR_Debug::findLogFile the first time, there is no other optionSet under aotCmdLineOptions except itself (0x7fffe2f2ff60), and there is no OptionSet from jitCmdLineOptions to compare with. The second round when TR_Debug::findLogFile is called, it is also fine because there is no OptionSet from jitCmdLineOptions to compare with, and the filename from previous OptionSet (0x7fffe2f2ff60) is also valid because OptionSet._options(0x7fffe2fa9e10) has been set up properly.

    I think the issue is really when both AOT and JIT have logs specified and when one is processed first, the OptionSet._options for the other one has not been set up properly yet. It wouldn't be an issue when specifying multiple logs under either AOT or JIT, because setOptions is always called for the current option that's being processed before it gets to TR_Debug::findLogFile.

    I think splitting the union into 2 as separate fields might still be the right way to fix the crash.

    [1]

    export TR_Options="verbose,vlog=vlog"
    export TR_OptionsAOT="aotrtDebugLevel=30,rtLog=rtLog,{*ArrayListSpliterator.tryAdvance*,*ReferencePipeline*.accept*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=log1),{*WrappingSpliterator.lambda*initPartialTraversalState*}(traceFull,traceILGen,traceCG,traceRelocatableDataCG,traceRelocatableDataDetailsCG,log=log2)"
    

    [2]

    Options::latePostProcess: isAOT 1 options 0x7fffe2f2f1c0 optionSet 0x7fffe2f2ff60 newOptions 0x7fffe2fa9e10 
    OptionSet::setOptions: _options 0x7ffff00b2574 -> 0x7fffe2fa9e10
    ...
    jitLatePostProcess:
       _jitCmdLineOptions 0x7fffe2f2ffa0 -> _logFileName (nil) _jitCmdLineOptions -> getFirstOptionSet (nil) -> getOptions (nil) -> getLogFileName (nil)
       _aotCmdLineOptions 0x7fffe2f2f1c0 -> _logFileName (nil) _aotCmdLineOptions -> getFirstOptionSet 0x7fffe2f2ff60 -> getOptions 0x7fffe2fa9e10 -> getLogFileName 0x7fffe2faa260
    ...
    findLogFile: call findLogFile for aotCmdLineOptions 0x7fffe2f2f1c0 optSet 0x7fffe2f2ff60 logFileName log2
    TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f2f1c0 optSet 0x7fffe2f2ff60, logFileName log2
    TR_Debug::findLogFile: fileName (nil) // return NULL because optSet is the same as _aotCmdLineOptions -> getFirstOptionSet() 
    
    findLogFile: call findLogFile for jitCmdLineOptions 0x7fffe2f2ffa0 optSet 0x7fffe2f2ff60 logFileName log2 
    TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f2ffa0 optSet 0x7fffe2f2ff60, logFileName log2
    TR_Debug::findLogFile: fileName (nil) // return NULL because jitCmdLineOptions getFirstOptionSet returns NULL 
    
    ... 
    jitLatePostProcess:
       _jitCmdLineOptions 0x7fffe2f2ffa0 -> _logFileName (nil) _jitCmdLineOptions -> getFirstOptionSet (nil) -> getOptions (nil) -> getLogFileName (nil)
       _aotCmdLineOptions 0x7fffe2f2f1c0 -> _logFileName (nil) _aotCmdLineOptions -> getFirstOptionSet 0x7fffe2f2ff60 -> getOptions 0x7fffe2fa9e10 -> getLogFileName 0x7fffe2faa260
    ...
    findLogFile: call findLogFile for aotCmdLineOptions 0x7fffe2f2f1c0 optSet 0x7fffe2f2fc00 logFileName log1
    TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f2f1c0 optSet 0x7fffe2f2fc00, logFileName log1
    TR_Debug::findLogFile: fileName (nil)
    TR_Debug::findLogFile: prev 0x7fffe2f2ff60 prev->getOptions() 0x7fffe2fa9e10 fileName 0x7fffe2faa260 // return NULL because the filename is different
    
    findLogFile: call findLogFile for jitCmdLineOptions 0x7fffe2f2ffa0 optSet 0x7fffe2f2fc00 logFileName log1
    TR_Debug::findLogFile: cmdLineOptions 0x7fffe2f2ffa0 optSet 0x7fffe2f2fc00, logFileName log1
    TR_Debug::findLogFile: fileName (nil)  // return NULL because jitCmdLineOptions getFirstOptionSet returns NULL
    
  7. added a commit that references this issue on Oct 6, 2021
    1aad2a6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions