Skip to content

JDK18 java/nio/channels/FileChannel/LargeGatheringWrite.java - OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384) #14503

Closed

Description

@JasonFengJ9

Failure link

From an internal build job/Test_openjdk18_j9_extended.openjdk_ppc64_aix/3/(paix907):

openjdk version "18" 2022-03-22
IBM Semeru Runtime Open Edition 18.0.0.0-m1 (build 18+35)
Eclipse OpenJ9 VM 18.0.0.0-m1 (build openj9-0.31.0-m1, JRE 18 AIX ppc64-64-Bit Compressed References 20220210_28 (JIT enabled, AOT enabled)
OpenJ9   - 700cde7b8
OMR      - bb5d44b8c
JCL      - cdd4205c649 based on jdk-18+35)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2022-02-11T07:42:13.549Z] variation: Mode150
[2022-02-11T07:42:13.549Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

[2022-02-11T08:08:10.204Z] TEST: java/nio/channels/FileChannel/LargeGatheringWrite.java

[2022-02-11T08:08:10.205Z] STDOUT:
[2022-02-11T08:08:10.205Z] Seed from RandomFactory = -8247103834647789495L
[2022-02-11T08:08:10.205Z] Writing 5368709122 bytes of data...
[2022-02-11T08:08:10.205Z] STDERR:
[2022-02-11T08:08:10.205Z] java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
[2022-02-11T08:08:10.205Z] 	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
[2022-02-11T08:08:10.205Z] 	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
[2022-02-11T08:08:10.205Z] 	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
[2022-02-11T08:08:10.205Z] 	at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:243)
[2022-02-11T08:08:10.205Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:209)
[2022-02-11T08:08:10.205Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:165)
[2022-02-11T08:08:10.205Z] 	at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:317)
[2022-02-11T08:08:10.205Z] 	at java.base/java.nio.channels.FileChannel.write(FileChannel.java:436)
[2022-02-11T08:08:10.205Z] 	at LargeGatheringWrite.main(LargeGatheringWrite.java:91)
[2022-02-11T08:08:10.205Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[2022-02-11T08:08:10.205Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
[2022-02-11T08:08:10.205Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2022-02-11T08:08:10.205Z] 	at java.base/java.lang.Thread.run(Thread.java:889)
[2022-02-11T08:08:10.205Z] 
[2022-02-11T08:08:10.205Z] JavaTest Message: Test threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)

[2022-02-11T08:08:10.206Z] TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)

job/Test_openjdk18_j9_extended.openjdk_s390x_linux/5/(ub18s390xrt-1-9)

[2022-02-11T07:08:46.817Z] variation: Mode650
[2022-02-11T07:08:46.817Z] JVM_OPTIONS:  -XX:-UseCompressedOops 

[2022-02-11T07:14:29.129Z] TEST: java/nio/channels/FileChannel/LargeGatheringWrite.java

[2022-02-11T07:14:29.131Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 137]
[2022-02-11T07:14:29.131Z] --------------------------------------------------
[2022-02-11T07:34:06.023Z] Test results: passed: 445; failed: 1
[2022-02-11T07:34:23.488Z] Report written to /home/jenkins/workspace/Test_openjdk18_j9_extended.openjdk_s390x_linux/jvmtest/openjdk/report/html/report.html
[2022-02-11T07:34:23.488Z] Results written to /home/jenkins/workspace/Test_openjdk18_j9_extended.openjdk_s390x_linux/aqa-tests/TKG/output_16445604291340/jdk_nio_1/work
[2022-02-11T07:34:23.488Z] Error: Some tests failed or other problems occurred.
[2022-02-11T07:34:23.488Z] 
[2022-02-11T07:34:23.488Z] jdk_nio_1_FAILED

job/Test_openjdk18_j9_extended.openjdk_x86-64_windows/5/(win16x64rt-1-2)

[2022-02-11T08:03:13.395Z] variation: Mode150
[2022-02-11T08:03:13.833Z] JVM_OPTIONS:  -XX:+UseCompressedOops 

openjdk version "18" 2022-03-22
IBM Semeru Runtime Open Edition 18.0.0.0-m1 (build 18+35)
Eclipse OpenJ9 VM 18.0.0.0-m1 (build openj9-0.31.0-m1, JRE 18 Windows Server 2016 amd64-64-Bit Compressed References 20220211_28 (JIT enabled, AOT enabled)
OpenJ9   - 700cde7b8
OMR      - bb5d44b8c
JCL      - cdd4205c649 based on jdk-18+35)

[2022-02-11T08:09:00.188Z] TEST: java/nio/channels/FileChannel/LargeGatheringWrite.java

[2022-02-11T08:09:00.190Z] STDERR:
[2022-02-11T08:09:00.190Z] java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
[2022-02-11T08:09:00.190Z] 	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
[2022-02-11T08:09:00.190Z] 	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
[2022-02-11T08:09:00.190Z] 	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
[2022-02-11T08:09:00.190Z] 	at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:243)
[2022-02-11T08:09:00.190Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:209)
[2022-02-11T08:09:00.190Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:165)
[2022-02-11T08:09:00.190Z] 	at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:317)
[2022-02-11T08:09:00.190Z] 	at java.base/java.nio.channels.FileChannel.write(FileChannel.java:436)
[2022-02-11T08:09:00.190Z] 	at LargeGatheringWrite.main(LargeGatheringWrite.java:91)
[2022-02-11T08:09:00.190Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
[2022-02-11T08:09:00.190Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
[2022-02-11T08:09:00.190Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2022-02-11T08:09:00.190Z] 	at java.base/java.lang.Thread.run(Thread.java:889)
[2022-02-11T08:09:00.190Z] 
[2022-02-11T08:09:00.190Z] JavaTest Message: Test threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)

Activity

  1. changed the title [-]JDK18 AIX java/nio/channels/FileChannel/LargeGatheringWrite.java - OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)[/-] [+]JDK18 java/nio/channels/FileChannel/LargeGatheringWrite.java - OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)[/+] on Feb 13, 2022
  2. tajila commented on Feb 17, 2022

    @tajila
    Contributor

    Looking at the test, it writes 3Gbs of data, half of the data comes from a heap buffer the other half from a direct buffer.
    When attempting to write the contents of the heap buffer java.base/sun.nio.ch.FileChannelImpl.write will allocate a temporary direct buffer, this is where we fail with OOM, m guess is that we have reached the memory limit.

  3. tajila commented on Feb 17, 2022

    @tajila
    Contributor

    @JasonFengJ9 Does this fail on all platforms?

  4. pshipton commented on Feb 17, 2022

    @pshipton
    Member

    A couple days ago I started running a few open grinders on this test (just 1 iteration). It failed on AIX but not alinux or zlinux.

  5. self-assigned this
    on Feb 18, 2022
  6. pshipton commented on Feb 24, 2022

    @pshipton
    Member

    @tajila any progress on this one?

  7. JasonFengJ9 commented on Feb 24, 2022

    @JasonFengJ9
    MemberAuthor

    Does this fail on all platforms?

    Oops missed this, it also failed at Windows - job/Test_openjdk18_j9_extended.openjdk_x86-64_windows/8/ (win16x64rt-1-7)

    openjdk version "18" 2022-03-22
    IBM Semeru Runtime Open Edition 18.0.0.0-m2 (build 18+36)
    Eclipse OpenJ9 VM 18.0.0.0-m2 (build openj9-0.31.0-m2, JRE 18 Windows Server 2016 amd64-64-Bit Compressed References 20220224_35 (JIT enabled, AOT enabled)
    OpenJ9   - 6c2d82c95
    OMR      - 3792851d0
    JCL      - 303814df05f based on jdk-18+36)
    
    [2022-02-24T05:21:01.277Z] TEST: java/nio/channels/FileChannel/LargeGatheringWrite.java
    
    [2022-02-24T05:21:01.277Z] STDERR:
    [2022-02-24T05:21:01.277Z] java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
    [2022-02-24T05:21:01.277Z] 	at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:243)
    [2022-02-24T05:21:01.277Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:209)
    [2022-02-24T05:21:01.277Z] 	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:165)
    [2022-02-24T05:21:01.277Z] 	at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:317)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.nio.channels.FileChannel.write(FileChannel.java:436)
    [2022-02-24T05:21:01.277Z] 	at LargeGatheringWrite.main(LargeGatheringWrite.java:91)
    [2022-02-24T05:21:01.277Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    [2022-02-24T05:21:01.277Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
    [2022-02-24T05:21:01.277Z] 	at java.base/java.lang.Thread.run(Thread.java:889)
    [2022-02-24T05:21:01.277Z] 
    [2022-02-24T05:21:01.277Z] JavaTest Message: Test threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
    [2022-02-24T05:21:01.277Z] JavaTest Message: shutting down test
    [2022-02-24T05:21:01.277Z] 
    [2022-02-24T05:21:01.277Z] STATUS:Failed.`main' threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
    
  8. tajila commented on Mar 1, 2022

    @tajila
    Contributor

    The memory limit is determined by j9gc_get_maximum_heap_size. The new requested amount is 48bytes over the the limit

    [2022-02-24T05:21:01.277Z] STATUS:Failed.`main' threw exception: java.lang.OutOfMemoryError: Cannot reserve 536870912 bytes of direct buffer memory (allocated: 3221225520, limit: 3758096384)
    
  9. tajila commented on Mar 1, 2022

    @tajila
    Contributor

    The test is run with Xmx4g, @dmitripivkine how is the max heap sized determined based on Xmx?

  10. 3 remaining items

  11. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    Could OpenJDK have changed the default limit to be higher?

  12. tajila commented on Mar 1, 2022

    @tajila
    Contributor

    On xlinux

    with Xmx4g

    java.lang.Runtime.getRuntime().totalMemory() == 4g
    jdk.internal.misc.VM.maxDirectMemory() == 4g
    

    on windows

    java.lang.Runtime.getRuntime().totalMemory() == 4g
    jdk.internal.misc.VM.maxDirectMemory() == 3.7g
    

    The cause is that the sun.nio.MaxDirectMemorySize property is set to 3.7g on windows but it doesnt appear to be set on xlinux

  13. tajila commented on Mar 1, 2022

    @tajila
    Contributor

    The tests above were done with openj9-jdk18

  14. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    There is code to set it at 7/8 of the heap size https://github.com/eclipse-openj9/openj9/blob/master/runtime/jcl/common/system.c#L476

    On xlinux 18 I see the system property is set (with -Xmx4g) sun.nio.MaxDirectMemorySize = 3758096384

  15. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    With OpenJDK (8 - 18) I don't see any default setting for MaxDirectMemorySize

  16. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    The OpenJ9 code comes from #2629

  17. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    I don't see why OpenJ9 needs to set the limit at 7/8, seems it should just not set MaxDirectMemorySize at all.

  18. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    The cause is that the sun.nio.MaxDirectMemorySize property is set to 3.7g on windows but it doesnt appear to be set on xlinux

    This is weird because from the code it appears it's set the same on every platform, and I can see the property set on xlinux.

    bin/java -Xmx4g -XshowSettings:properties -version 2>&1 | grep MaxDirectMemorySize
    
  19. tajila commented on Mar 1, 2022

    @tajila
    Contributor

    This is weird because from the code it appears it's set the same on every platform, and I can see the property set on xlinux.

    May mistake, I was using the RI JDK18. On j9 xlinux it is set as well

  20. pshipton commented on Mar 1, 2022

    @pshipton
    Member

    We should validate that OpenJDK uses Runtime.getRuntime().totalMemory() as the default maxDirectMemory() for all versions, and if that is confirmed then remove setting sun.nio.MaxDirectMemorySize to be 7/8 of the max heap size, and just use the default. This doesn't need to occur for the Java 18 release, we can defer to 0.32.

  21. pshipton commented on Mar 3, 2022

    @pshipton
    Member

    I'll keep this open until we decide if we are going to backport the fix.

  22. pshipton commented on Mar 3, 2022

    @pshipton
    Member

    Fix is merged in head, 0.31, 0.32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions