Repository navigation
Closed
Labels
blockercomp:jitproject:MHUsed to track Method Handles related workUsed to track Method Handles related worksegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Milestone
Description
Latest occurrence at https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/8/consoleFull
21:27:44 openjdk version "17-internal" 2021-09-14
21:27:44 OpenJDK Runtime Environment (build 17-internal+0-adhoc.****.buildjdk17x86-64windowsnightly)
21:27:44 Eclipse OpenJ9 VM (build v0.28.0-release-18bebe9f4e3, JRE 17 Windows Server 2012 R2 amd64-64-Bit Compressed References 20210829_8 (JIT enabled, AOT enabled)
21:27:44 OpenJ9 - 18bebe9f4e3
21:27:44 OMR - 1d0a329
21:27:44 JCL - 712145ee3f5 based on jdk-17+35)
21:51:08 test DefaultStaticInvokeTest.testMethodHandleInvoke("TestClass7", "TestIF7.TestClass7"): failure
21:51:08 java.lang.NullPointerException: Cannot invoke "java.lang.invoke.MethodHandle.linkToVirtual(java.lang.Object, java.lang.Object, java.lang.invoke.MemberName)" because "<parameter1>" is null
21:51:08 at java.base/java.lang.invoke.LambdaForm$DMH/0x0000000000000000.invokeVirtual(LambdaForm$DMH)
21:51:08 at java.lang.invoke.LambdaForm$MH/0x0000000095dfc060.invoke(LambdaForm$MH)
21:51:08 at java.lang.invoke.LambdaForm$MH/0x0000000095ddf6f0.invoke_MT(LambdaForm$MH)
21:51:08 at DefaultStaticInvokeTest.testMethodHandleInvoke(DefaultStaticInvokeTest.java:169)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
21:51:08 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:51:08 at java.base/java.lang.reflect.Method.invoke(Method.java:568)
21:51:08 at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
21:51:08 at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
21:51:08 at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
21:51:08 at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
21:51:08 at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
21:51:08 at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
21:51:08 at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
21:51:08 at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
21:51:08 at org.testng.TestRunner$$Lambda$129/0x0000000096210428.accept(Unknown Source)
21:51:08 at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
21:51:08 at org.testng.TestRunner.privateRun(TestRunner.java:764)
21:51:08 at org.testng.TestRunner.run(TestRunner.java:585)
21:51:08 at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
21:51:08 at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
21:51:08 at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
21:51:08 at org.testng.SuiteRunner.run(SuiteRunner.java:286)
21:51:08 at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
21:51:08 at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
21:51:08 at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
21:51:08 at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
21:51:08 at org.testng.TestNG.runSuites(TestNG.java:1069)
21:51:08 at org.testng.TestNG.run(TestNG.java:1037)
21:51:08 at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
21:51:08 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:51:08 at java.base/java.lang.reflect.Method.invoke(Method.java:568)
21:51:08 at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
21:51:08 at java.base/java.lang.Thread.run(Thread.java:884)
21:51:08 ===============================================
21:51:08 java/lang/reflect/DefaultStaticTest/DefaultStaticInvokeTest.java
21:51:08 Total tests run: 276, Passes: 274, Failures: 2, Skips: 0
21:51:08 ===============================================
21:51:08
21:51:08 STDERR:
21:51:08 java.lang.Exception: failures: 2
21:51:08 at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:96)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
21:51:08 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
21:51:08 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
21:51:08 at java.base/java.lang.reflect.Method.invoke(Method.java:568)
21:51:08 at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
21:51:08 at java.base/java.lang.Thread.run(Thread.java:884)
21:51:08
21:51:08 JavaTest Message: Test threw exception: java.lang.Exception
21:51:08 JavaTest Message: shutting down test
21:51:08
21:51:08
21:51:08 TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.Exception: failures: 2
21:51:08 --------------------------------------------------
21:54:09 Test results: passed: 766; failed: 1
21:54:10 Report written to F:\Users\jenkins\workspace\Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly\jvmtest\openjdk\report\html\report.html
21:54:10 Results written to F:\Users\jenkins\workspace\Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly\aqa-tests\TKG\output_16302870106882\jdk_lang_1\work
21:54:10 Error: Some tests failed or other problems occurred.
21:54:10
21:54:10 jdk_lang_1_FAILED
Metadata
Metadata
Assignees
Labels
blockercomp:jitproject:MHUsed to track Method Handles related workUsed to track Method Handles related worksegfaultIssues that describe segfaults / JVM crashesIssues that describe segfaults / JVM crashestest failure
Activity
0xdaryl commentedon Sep 8, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/12/consoleFull
Similar NPEs at https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/12/consoleFull
pshipton commentedon Sep 8, 2021
Tagging as jit, assuming this is jit.
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/17
JasonFengJ9 commentedon Sep 9, 2021
Similar NPE observed at an internal build w/ JDK17 M1
Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Personal/29/consoleFullThe test in question passed w/ JDK17 head stream build though.
Edit failed at grinder below, so the problem is also in head stream.
Launched grinders to confirm the difference between the head stream and M1 release.
JDK17 M1 release -
Test_grinder/job/Grinder/18015/- Edit: passed, launched another oneGrinder/18017/JDK17 head stream -
Test_grinder/job/Grinder/18016/- Edit: passed, also launched another oneGrinder/18020- 19/50 failed.pshipton commentedon Sep 10, 2021
Crash instead of NPE. I'll put it here since it's the DefaultStaticInvokeTest test, but can open another Issues if necessary.
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/19/
jdk_lang_0
-Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:+UseCompressedOops
java/lang/reflect/DefaultStaticTest/DefaultStaticInvokeTest.java
pshipton commentedon Sep 10, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/19
jdk_lang_0
jdk_lang_1
JasonFengJ9 commentedon Sep 13, 2021
Observed w/ latest JDK17
0.28release build atjob/Test_openjdk17_j9_sanity.openjdk_x86-64_mac/56/pshipton commentedon Sep 13, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/21
jdk_lang_1
-Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-JITServerTechPreviewMessage -XX:-UseCompressedOops
https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/21/openjdk_test_output.tar.gz
pshipton commentedon Sep 13, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/22
jdk_lang_0
https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/22/openjdk_test_output.tar.gz
pshipton commentedon Sep 13, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/22
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/22
jdmpapin commentedon Sep 13, 2021
The failure is highly reproducible on Mac. In internal grinders it failed 6/10 and another 2/5 with
-Xjit:verbose,vlog=vlog. I did get verbose logs but each failed iteration produced three verbose logs, and I haven't been able to tell which is the relevant oneI tried running this on my machine with all tests excluded other than
DefaultStaticInvokeTest, and I was unable to reproduce it that way (passed 150/150 total iterations). However, both @nbhuiyan and I have just now been able to reproduce this (on x86 Mac and x86 Linux, resp.) by allowingjdk_lang_0to run all of the tests it normally runsNext I'll try to determine a failure rate for my local repro, and if it's reasonably high I can debug locally
pshipton commentedon Sep 14, 2021
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/23 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/23 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/24 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/24 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/26 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/26 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/26 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/28 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/29/ - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/30 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/32 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/35 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/34 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/34 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/36 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/37 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/36 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/36 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/4 - SIGBUS
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/37 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/39 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/6 - SIGBUS
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/41 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/42 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/7 - SIGBUS
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/41 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/43 - crash
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/45 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/44 - NPE
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/47
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/45
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/11
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Nightly/47
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/46
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/13
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/47
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/14
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Nightly/48
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_mac_Release/7
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_s390x_linux_Release/8
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_x86-64_windows_Nightly/52
jdmpapin commentedon Sep 16, 2021
From analysis of the core generated by the first z Linux crash listed in the previous comment...
LambdaForm$DMH/0x0000000000000000.invokeVirtualis hitting an unexpected segfault (which in some tests appears as an NPE due to our implicit null checks) because the "VFT offset" it finds via theMemberNameis 0x8000000000000000The high bit there is
J9_JNI_MID_INTERFACE:openj9/runtime/oti/j9consts.h
Lines 51 to 52 in 7d96165
which is a flag that indicates that
J9JNIMethodID::vTableIndexactually represents an itable index instead:openj9/runtime/vm/jnicsup.cpp
Lines 2261 to 2270 in 6450b81
So a value of 0x8000000000000000 indicates itable index 0. For a method found by virtual lookup, we can still end up stuck with an itable index rather than a VFT index when the method is a default implementation inherited from an interface
With this understanding, Repro.java consistently reproduces the issue when run like so:
And adding
-Xrsto the command line shows that this NPE is really an unexpected segfault as wellTo fix this, the trees generated for
linkToVirtual()in recognized call transformer will need to detect and handleJ9_JNI_MID_INTERFACE. FYI @nbhuiyan, @0xdaryl, @vijaysun-omrpshipton commentedon Sep 23, 2021
First time we've had aarch64 testing in the open nightly.
https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_sanity.openjdk_aarch64_linux_Nightly/1 - cent8-aarch64-1
Opened #13556 for the SIGBUS
jdmpapin commentedon Sep 23, 2021
I have a fix in the works based on my previous comment, and I should be able to open a PR for it pretty soon
pshipton commentedon Oct 15, 2021
Re-opening as the fix was temporarily reverted, see #13578 (comment)
pshipton commentedon Oct 15, 2021
Merged the fix again via #13716
pshipton commentedon Oct 18, 2021
We can keep this open until the fix is delivered to the 0.29.1 branch.