Repository navigation
Closed
eclipse-omr/omr
#6208Labels
blockercomp:jitproject:MHUsed to track Method Handles related workUsed to track Method Handles related worktest failure
Milestone
Description
Failure link
From an internal build Test_openjdk16_j9_extended.functional_ppc64le_linux_testList_0/50/:
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment Temurin-16.0.1+9 (build 16.0.1+9)
Eclipse OpenJ9 VM Temurin-16.0.1+9 (build master-d5a8afc9d, JRE 16 Linux ppc64le-64-Bit Compressed References 20210710_57 (JIT enabled, AOT enabled)
OpenJ9 - d5a8afc9d
OMR - bdee22f4b
JCL - 3c4bffefe5 based on jdk-16.0.1+9)
Optional info
Failure output (captured from console output)
[2021-07-10T15:22:16.062Z] Running test jsr292_MethodHandleAPI_Test_JitCount1_0 ...
[2021-07-10T15:22:16.062Z] ===============================================
[2021-07-10T15:22:16.062Z] jsr292_MethodHandleAPI_Test_JitCount1_0 Start Time: Sat Jul 10 08:22:15 2021 Epoch Time (ms): 1625930535049
[2021-07-10T15:22:16.062Z] "/home/jenkins/workspace/Test_openjdk16_j9_extended.functional_ppc64le_linux_testList_0/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/home/jenkins/workspace/Test_openjdk16_j9_extended.functional_ppc64le_linux_testList_0/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
[2021-07-10T15:22:16.062Z] JVMSHRC005I No shared class caches available
[2021-07-10T15:22:16.062Z] JVMSHRC005I No shared class caches available
[2021-07-10T15:22:16.062Z] cache cleanup done
[2021-07-10T15:22:16.062Z] variation: NoOptions
[2021-07-10T15:22:16.062Z] JVM_OPTIONS:
[2021-07-10T15:22:53.453Z] FAILED: test_countedLoop_ByteType
[2021-07-10T15:22:53.453Z] java.lang.NullPointerException: Cannot read field "rtype" because "this" is null
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodType.hashCode(MethodType.java:876)
[2021-07-10T15:22:53.454Z] at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodType$ConcurrentWeakInternSet.get(MethodType.java:1370)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodType.makeImpl(MethodType.java:343)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodType.methodType(MethodType.java:263)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodHandles.countedLoopChecks(MethodHandles.java:7363)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.invoke.MethodHandles.countedLoop(MethodHandles.java:7311)
[2021-07-10T15:22:53.454Z] at com.ibm.j9.jsr292.api.MethodHandleAPI_countedLoop.test_countedLoop_ByteType(MethodHandleAPI_countedLoop.java:215)
[2021-07-10T15:22:53.454Z] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2021-07-10T15:22:53.454Z] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78)
[2021-07-10T15:22:53.454Z] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2021-07-10T15:22:53.454Z] at java.base/java.lang.reflect.Method.invoke(Method.java:567)
[2021-07-10T15:22:53.454Z] at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
[2021-07-10T15:22:53.454Z] at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
[2021-07-10T15:22:53.454Z] at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
[2021-07-10T15:22:53.454Z] at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
[2021-07-10T15:22:53.454Z] at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
[2021-07-10T15:22:53.454Z] at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
[2021-07-10T15:22:53.454Z] at org.testng.TestRunner.privateRun(TestRunner.java:648)
[2021-07-10T15:22:53.454Z] at org.testng.TestRunner.run(TestRunner.java:505)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunner.run(SuiteRunner.java:364)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
[2021-07-10T15:22:53.454Z] at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.runSuites(TestNG.java:1049)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.run(TestNG.java:1017)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.privateMain(TestNG.java:1354)
[2021-07-10T15:22:53.454Z] at org.testng.TestNG.main(TestNG.java:1323)
[2021-07-10T15:22:53.475Z] ===============================================
[2021-07-10T15:22:53.475Z] jsr292_Suite_SE90
[2021-07-10T15:22:53.475Z] Total tests run: 436, Failures: 39, Skips: 0
[2021-07-10T15:22:53.475Z] ===============================================
[2021-07-10T15:22:53.475Z]
[2021-07-10T15:22:56.515Z]
[2021-07-10T15:22:56.515Z] jsr292_MethodHandleAPI_Test_JitCount1_0_FAILED
Metadata
Metadata
Assignees
Labels
blockercomp:jitproject:MHUsed to track Method Handles related workUsed to track Method Handles related worktest failure
Activity
pshipton commentedon Jul 15, 2021
@tajila fyi
JasonFengJ9 commentedon Aug 19, 2021
The error is observed w/ a recent JDK16 build at
Test_openjdk16_j9_extended.functional_ppc64le_linux_Personal/10/tapResults/Re-run the Grinder
JasonFengJ9 commentedon Aug 26, 2021
Observed in latest JDK17 https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/4/consoleFull
Re-run the Grinder
[-]JDK16 (OJDK MH) jsr292_MethodHandleAPI_Test_JitCount1_0_FAILED - NullPointerException: Cannot read field "rtype" because "this" is null[/-][+]JDK17 jsr292_MethodHandleAPI_Test_JitCount1_0_FAILED - NullPointerException: Cannot read field "rtype" because "this" is null[/+]fengxue-IS commentedon Sep 7, 2021
10/10 passed on x86-64_linux
14/15 failed on ppcle_linux
I suspect this have to do with power codegen? maybe due to weak memory ordering?
@nbhuiyan can you please take a look at this
0xdaryl commentedon Sep 8, 2021
The backtrace for the remaining Power failure does not suggest a MH handle problem due to the JIT transformations for LambdaForms where many of the other problems are seen.
In light of that, @zl-wang could you have someone triage this problem on Power Linux please? It seems quite reproducible (14/15 runs). Thank you.
zl-wang commentedon Sep 8, 2021
@rmnattas vijay fixed EA for this very same issue previously you investigated. not sure why it happens again. please have a look.
JasonFengJ9 commentedon Sep 9, 2021
Observed at an internal build
Test_openjdk17_j9_extended.functional_ppc64le_linux_Personal/32/tapResults/Similar NPEs were observed at JDK17 head stream as well
Test_openjdk17_j9_extended.functional_ppc64le_linux_Personal/33/tapResults/JasonFengJ9 commentedon Sep 10, 2021
Observed at JDK17
0.28releaseMacOSbuild atjob/Test_openjdk17_j9_extended.functional_x86-64_mac_testList_0/50/Rerun in Grinder
21 remaining items
tajila commentedon Sep 21, 2021
@ymanton
I looked at the test and there isn't a reason why the rtype should ever be null. So I think there is still another problem here. Adding a null check on the rtype before searching the intern table will just push the NPE somewhere else.
tajila commentedon Sep 21, 2021
Based on the stack trace, the null rtype comes from the
start.type().returnType();of theMethodHandle countedLoop(MethodHandle start, MethodHandle end, MethodHandle init, MethodHandle body)call. In this test case it should beint.classas the start MH isMethodHandles.constant(int.class, 5);pshipton commentedon Sep 27, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_x86-64_windows_Nightly_testList_1/35
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/35
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/34
jsr292_MethodHandleAPI_Test_JitCount1_0 (NoOptions)
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/35/
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/36
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/40
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_x86-64_linux_Nightly_testList_0/41
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/42
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/44
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64le_linux_Nightly_testList_1/46
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/45
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/46
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/48
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Release_testList_0/7
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_x86-64_linux_Nightly_testList_0/50
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/51
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_x86-64_windows_Nightly_testList_1/53
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/54
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.functional_ppc64_aix_Nightly_testList_0/58
ymanton commentedon Sep 27, 2021
Managed to get a core file closer to the exception throw by hacking the exception handler to crash if execution is in
MethodType.hashCode(). Working backwards from there toMethodHandles.countedLoopChecks()gets me to this bit of code:There are multiple compiled bodies containing the above, but for a particular warm compile of
MethodHandles.countedLoopChecks()we generate the wrong code for the last line and callmethodType(NULL, outerList):Going to get a trace log next to see why we generated the wrong code for that first parameter.
nbhuiyan commentedon Sep 27, 2021
I have been able to reliably make this crash occur in every run by inserting a
sleeppoint intest_countedLoop_DoubleTypetest case.With that setup, I observed that the test failure was also happening withI then inserted print statements in-Xint, suggesting that this may not be due to something going wrong during compilation.MethodHandles.countedLoop,MethodHandles.countedLoopChecks,MethodType.methodType, andMethodType.makeImplthat I could toggle using env variables. The goal was to track wherecounterTypewas becoming invalid, as that's the reason behind the MethodType object being created with a nullrtypefield.As Tobi mentioned above,
counterTypeis obtained fromstart.type().returnType(), and should have been the value that is provided as the first arg inEach of the failed tests are set to run 2 iterations, and for some of them, the first iteration completes without issues only to fail in the second run. In all failing cases, there is a point along
countedLoopChecksthat I was able to detect the first instance of the value ofcountedLoopChecksto change from int.class tonull: right after this line.So the
counterTypecan somehow become null, and I am not sure why. This eventually then results in the nullMethodType.rtypefield created viaMethodType.makeImpl. WhencounterTypebecomes NULL,start.type().returnType()was stillint.class, and somehow by replacingcounterTypewithstart.type().returnType()inMethodType expected = methodType(counterType, outerList);fixes the problem.Edit: the test setup ignores the -Xint option, so this could still be due to the JIT
JasonFengJ9 commentedon Oct 1, 2021
Just note this is observed in JDK17
0.29release buildjob/Test_openjdk17_j9_extended.functional_ppc64le_linux_Personal/59/tapResults/.jdmpapin commentedon Oct 15, 2021
Based on analysis of log files produced using @nbhuiyan's repro setup, we traced the cause of this to an intersection bug in VP. The code reads the
rtypefield of aMethodType, which gets a global{JavaLangClassObject}constraint, and then stores the reference into an auto namedcounterType. It then compares the value againstint.class, creating a local (path-dependent) known object constraint in the successor wherecounterType == int.class. Later, in a block only reachable withcounterType == int.class, the code compares it again to the result ofList.get, which is constrained with a type bound ofjava/lang/Object. In the successor where they're equal, VP attempted to create a local constraint with the intersection of the known object and thejava/lang/Objecttype, but first it intersected thejava/lang/Objectconstraint with the global{JavaLangClassObject}constraint, resulting in aVPClasscontaining both:class Ljava/lang/Object; {JavaLangClassObject}, which means thatcounterTypewill be (null or) an instance ofjava/lang/Classthat representsjava/lang/Objector a subtype of it. This is the incorrect constraint - it should be just{JavaLangClassObject}. The incorrect constraint was then intersected with the known object, resulting in a known null reference because as a primitive,intis not a subtype ofObject. (For this last intersection, VP could have produced an unreachable path constraint, but in that case any other constraint of the correct type will do as well.) Finally, there was downstream code passingcounterTypeas the return type toMethodType.methodType, and the uses ofcounterTypewere folded to null.VPAcmpObjectPrimJLClass.java is a small test case that reproduces the underlying VP intersection bug
It should print
intboth times, but instead printsintfollowed bynull.0xdaryl commentedon Oct 26, 2021
This will require an openj9-omr PR in 0.29.1 as well.
pshipton commentedon Oct 26, 2021
We can keep this open until the 0.29.1 PR is merged.
jdmpapin commentedon Oct 26, 2021
The fix has been merged into the 0.29.1 release branch in eclipse-openj9/openj9-omr#129