You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While running a Java virtual thread benchmark named Skynet, poor performance is seen in omrmem_allocate_memory32 as the benchmark exhausts the native memory below 4G. This either results in a time out or an OutOfMemoryError.
Potential Issues
It takes too much time to search for memory as more memory is used and the memory below 4G is exhausted.
There is a single lock/monitor and large critical sections which act as a bottleneck and make the alloc/free operations single threaded.
Fragmentation is prevalent and creates gaps (small chunks of unused memory), which prevents memory to be efficiently used.
Two Potential Solutions
[Approach 1] Improve the existing omrmem_allocate_memory32 implementation; or
[Approach 2] Adopt an existing memory allocator implementation.
Approach 2 is preferred since there are existing implementations for the memory allocator, which address the above perf issues.
Are the copyrights compatible such that the memory allocator can be embedded in OpenJ9 and OMR?
Time estimate on embedding the memory allocator in OMR. Key tasks:
Makefiles will need to be updated to embed the memory allocator and use its API.
Is it possible to control memory allocation below 4G? There might be an option to achieve this goal or we might need to modify the memory allocator's implementation.
Functional testing: existing OpenJ9 and OMR should be sufficient.
Performance evaluation plan: might take some effort since we will need to define metrics and find benchmarks to study the perf impact.
Here's an update. (tldr; This doesn't appear to be an issue currently. And I think this can be closed.)
I'll answer some of the questions from above out-of-order:
While running a Java virtual thread benchmark named Skynet, poor performance is seen in omrmem_allocate_memory32 as the benchmark exhausts the native memory below 4G. This either results in a time out or an OutOfMemoryError.
I'vn't been able to observe this running hundreds of iterations of JDK21 builds on x86_64 Linux and Power Linux machines.
It takes too much time to search for memory as more memory is used...
Though our sub-4GB allocator isn't as efficient as just a call to malloc that the size_t-width referenced-objects use, any limited sub-n GB allocation is inherently less efficient because sub-n GB allocation requires looping and hinting to the underlying memory mapping system calls (at least in the case of POSIX-compliant system).
Is it possible to control memory allocation below 4G?
I've embedded (without LD_PRELOAD) a custom implementation of jemalloc[1,2,3] wherein allocations are constrained below the 4GB boundary. There isn't an allocator that I have found that allows for this type of constriction out-of-the-box based on the limitations discussed in the previous response. I chose jemalloc because: 1) its goal and design targets minimizing lock-contention; 2) it has positive results in various benchmarking reports, but namely [4]; and 3) the project uses make which integrates well into our build system (as opposed to other contenders like tcmalloc[5] which would add bazel as a dependency). Moreover, the only copyright-condition is retaining their copyright notice.
Performance evaluation plan
Since I'vn't been able to reproduce the Skynet regression, I also reached for the renaissance[6] test-suite. I ran many iterations of Skynet and each of the renaissance benchmarks -- with a particular focus on the memory-bound and contention benchmarks -- both with and without the custom jemalloc and did not observe any efficiency improvements (in terms of run-time) nor reduction in lock-contention (since I couldn't reproduce this in the first place).
Conclusion
In light of the above, I conclude that there shouldn't be any action taken since there is no observable problem with the current implementation of the sub-4GB allocator (at least on our tested workloads). And without the positive performance data to back up the embedding, the cost of supporting, shipping, and maintaining a custom third-party allocator is not ameliorated.
That being said, I will document -- in a comment below -- how I embedded jemalloc into OpenJ9 and OMR for furture reference, and preserve my patches and preliminary custom implementation in case we ever want to revisit this.
Aside
I was thinking we could also explore using jemalloc as the default allocator rather than malloc for all other allocations. But preliminary testing here too was neutral. And modern glibcmalloc implementations already do deliver some of the advancement from other allocators like local caches and arenas. So I don't think there is anything fruitful here currently either.
For future interest/reference, here are the patches to embed a third-party allocator (well jemalloc specifically, but it should be the same for others too).
OMR: ThanHenderson@94f023b
OpenJ9: ThanHenderson/openj9@a3aa879
It is a little funky to get the cmake scripts in order; this solution works, but it would need more care if embedding for production e.g. jemalloc is built twice during the build process, for both OpenJ9 and OMR but the build results are in the same directory, a better solution would be to check if the library exists before invoking the fetch and build commands.
To keep things internal to the project, I used cmake's FetchContent functionality to pull from a hosted Git repo (this can be anything) and installed it locally rather than to one of the system library search paths. This required some configuring to force the use of, and make changes to, the rpaths of certain dependent targets because using target_link_libraries wasn't sufficient on its own and cmake doesn't support runpath. FetchContent relies on a CMakeLists.txt file in allocator's repo. Since jemalloc is make-based, I needed to create a wrapper file that simply invokes the appropriate build pipeline (autogen.sh, configure, make) e.g. shown below:
cmake_minimum_required(VERSION 3.1...3.28)
project(
jemalloc
VERSION 1.0
LANGUAGES CXX)
add_custom_target(nuke
COMMAND make clean
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
add_custom_target(jemalloc ALL
COMMAND ./autogen.sh --disable-initial-exec-tls COMMAND make
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
FetchContent resolves the repo at configure time and automatically invokes the CMakeLists.txt file for the external project at build time. Projects that are not make-based or cmake-based likely involve a similar pipeline.
Regardless of the allocator used, OMR just expects that malloc32 and free32 procedures are exported. Due to how some targets are linked to the shared library, one needs to take care not to export other functionality -- malloc, calloc, realloc, free etc. -- from the embedded allocator so as to not interfere with other allocations. This would likely cause some downstream issues in the GC and such.
So it sounds like swapping the our current allocator for another one is not the answer. However, perhaps there is a way to tune our allocator to be more optimal for vthread workloads? Have you looked into toggling the initial region size of the heaps? the amount that is initially committed? Perhaps, there is a size that is more optimal for vthread heavy workloads.
@tajila I have not, but it seems like there is an opportunity for a limit study here. Other than region size, are there any other parameters that you think I should look into?
There is also PPG_mem_mem32_subAllocHeapMem32.suballocator_commitSize which TBH I havent looked at too closely. But it seems like we commit only a portion of the initial reservation size, then when we run out we commit more.
Overall, I think we should consider increasing the region size (static increase), but also look at a dynamic policy where the region size grows as we allocate more (we do something similar for class memory segments).
Perhaps, there is a size that is more optimal for vthread heavy workloads.
@tajila is there a collection of vthread heavy workloads somewhere? I've only been running the Skynet stress benchmark, and am not observing any difference from tweaking the commitSize and initialSize.
I would configure jmeter to hit the endpoints (you can toggle the amount of load). The relevant endpoint in this case would be http://localhost:8080/parallel?...
I noticed pretty significant deltas between compressedrefs and non-compressedrefs with this example
I am noticing differences for helidon nima when increasing the initial region size: -Xgc:suballocatorInitialSize=<#>
No noticeable difference when tweaking the commit size: -Xgc:suballocatorCommitSize=<#>
I was testing with spawning a new server 20 times and sending a curl -i http://localhost:8080/parallel\?count\=10000 request once on each server. On my machine, with -Xnocompressedrefs the requests averaged to 5.91 seconds. With compressedrefs enabled, and the default values for initialSize (200 MB) and commitSize (50 MB) the requests averaged to 11.87 seconds.
I iterated over initialSizes from 200 to 1000 MB with a step size of 100 MB. The data in the table below shows those average values from the iterations:
initialSize (MB)
Avg (s)
200 (default)
11.87
300
10.55
400
8.77
500
7.62
600
7.17
700
8.14
800
8.90
900
8.63
1000
8.46
On this workload, a static increase does show some benefit. It would be beneficial to test this on a set of workloads though that have observable improvements that may motivate a dynamic solution.
@dmitripivkine@amicic
Do you guys know of any other workloads that could be important to test that could stress the change to the initial reservation size for the suballocator?
Moreover, are there any consequences/reasons not to increase suballocatorInitialSize that we should consider?
@dmitripivkine@amicic Do you guys know of any other workloads that could be important to test that could stress the change to the initial reservation size for the suballocator? Moreover, are there any consequences/reasons not to increase suballocatorInitialSize that we should consider?
This is more VM than GC question. Suballocator area located below 4G bar is used for artifacts must be 32-bit for Compressed refs (j9class, j9vmthread etc.). The only GC aspect is not prevent most performant 0-shift run where entire heap is located below 4G bar as well. So, by taking more memory for Suballocator initially you can compromise 0-shilt runs.
Also it is not clear for my how exactly Suballocator initial size effects your test performance.
Also it is not clear for my how exactly Suballocator initial size effects your test performance.
A larger initial size reduces the number of subsequent sub-4G allocations which can be costly due to how -- a brute force looping strategy -- sub-4G memory is reserved.
A larger initial size reduces the number of subsequent sub-4G allocations which can be costly due to how -- a brute force looping strategy -- sub-4G memory is reserved.
You comment is not very practical I afraid. The question where bottle neck is.
We looked to the code with @tajila and seems the logic is:
initial Suballocator memory reserved (200m by default, controlled by -Xgc:suballocatorInitialSize)
this memory is committed by chunks (50m default, controlled by -Xgc:suballocatorCommitSize)
Activity
babsingh commentedon Nov 23, 2023
@ThanHenderson Opened this issue to document our earlier discussion. All future updates can also be posted here.
fyi @tajila
ThanHenderson commentedon Mar 5, 2024
Here's an update. (tldr; This doesn't appear to be an issue currently. And I think this can be closed.)
I'll answer some of the questions from above out-of-order:
I'vn't been able to observe this running hundreds of iterations of JDK21 builds on
x86_64 LinuxandPower Linuxmachines.Though our sub-4GB allocator isn't as efficient as just a call to
mallocthat thesize_t-width referenced-objects use, any limited sub-n GB allocation is inherently less efficient because sub-n GB allocation requires looping and hinting to the underlying memory mapping system calls (at least in the case of POSIX-compliant system).I've embedded (without
LD_PRELOAD) a custom implementation ofjemalloc[1,2,3] wherein allocations are constrained below the 4GB boundary. There isn't an allocator that I have found that allows for this type of constriction out-of-the-box based on the limitations discussed in the previous response. I chosejemallocbecause: 1) its goal and design targets minimizing lock-contention; 2) it has positive results in various benchmarking reports, but namely [4]; and 3) the project usesmakewhich integrates well into our build system (as opposed to other contenders liketcmalloc[5] which would addbazelas a dependency). Moreover, the only copyright-condition is retaining their copyright notice.Since I'vn't been able to reproduce the Skynet regression, I also reached for the
renaissance[6] test-suite. I ran many iterations of Skynet and each of therenaissancebenchmarks -- with a particular focus on thememory-boundandcontentionbenchmarks -- both with and without the customjemallocand did not observe any efficiency improvements (in terms of run-time) nor reduction in lock-contention (since I couldn't reproduce this in the first place).Conclusion
In light of the above, I conclude that there shouldn't be any action taken since there is no observable problem with the current implementation of the sub-4GB allocator (at least on our tested workloads). And without the positive performance data to back up the embedding, the cost of supporting, shipping, and maintaining a custom third-party allocator is not ameliorated.
That being said, I will document -- in a comment below -- how I embedded
jemallocinto OpenJ9 and OMR for furture reference, and preserve my patches and preliminary custom implementation in case we ever want to revisit this.Aside
I was thinking we could also explore using
jemallocas the default allocator rather thanmallocfor all other allocations. But preliminary testing here too was neutral. And modernglibcmallocimplementations already do deliver some of the advancement from other allocators like local caches and arenas. So I don't think there is anything fruitful here currently either.[1] https://github.com/jemalloc/jemalloc
[2] https://people.freebsd.org/~jasone/jemalloc/bsdcan2006/jemalloc.pdf
[3] http://jemalloc.net/jemalloc.3.html
[4] http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/
[5] https://github.com/google/tcmalloc
[6] https://renaissance.dev/
ThanHenderson commentedon Mar 7, 2024
For future interest/reference, here are the patches to embed a third-party allocator (well
jemallocspecifically, but it should be the same for others too).OMR: ThanHenderson@94f023b
OpenJ9: ThanHenderson/openj9@a3aa879
It is a little funky to get the
cmakescripts in order; this solution works, but it would need more care if embedding for production e.g.jemallocis built twice during the build process, for both OpenJ9 and OMR but the build results are in the same directory, a better solution would be to check if the library exists before invoking the fetch and build commands.To keep things internal to the project, I used
cmake'sFetchContentfunctionality to pull from a hosted Git repo (this can be anything) and installed it locally rather than to one of the system library search paths. This required some configuring to force the use of, and make changes to, therpathsof certain dependent targets because usingtarget_link_librarieswasn't sufficient on its own andcmakedoesn't supportrunpath.FetchContentrelies on aCMakeLists.txtfile in allocator's repo. Sincejemallocismake-based, I needed to create a wrapper file that simply invokes the appropriate build pipeline (autogen.sh,configure,make) e.g. shown below:FetchContentresolves the repo at configure time and automatically invokes theCMakeLists.txtfile for the external project at build time. Projects that are notmake-based orcmake-based likely involve a similar pipeline.Regardless of the allocator used, OMR just expects that
malloc32andfree32procedures are exported. Due to how some targets are linked to the shared library, one needs to take care not to export other functionality --malloc,calloc,realloc,freeetc. -- from the embedded allocator so as to not interfere with other allocations. This would likely cause some downstream issues in the GC and such.Everything should be smooth sailing after that.
tajila commentedon Mar 14, 2024
Thanks for your analysis @ThanHenderson
So it sounds like swapping the our current allocator for another one is not the answer. However, perhaps there is a way to tune our allocator to be more optimal for vthread workloads? Have you looked into toggling the initial region size of the heaps? the amount that is initially committed? Perhaps, there is a size that is more optimal for vthread heavy workloads.
ThanHenderson commentedon Mar 14, 2024
@tajila I have not, but it seems like there is an opportunity for a limit study here. Other than region size, are there any other parameters that you think I should look into?
tajila commentedon Mar 14, 2024
There is also
PPG_mem_mem32_subAllocHeapMem32.suballocator_commitSizewhich TBH I havent looked at too closely. But it seems like we commit only a portion of the initial reservation size, then when we run out we commit more.Overall, I think we should consider increasing the region size (static increase), but also look at a dynamic policy where the region size grows as we allocate more (we do something similar for class memory segments).
ThanHenderson commentedon Apr 18, 2024
@tajila is there a collection of vthread heavy workloads somewhere? I've only been running the Skynet stress benchmark, and am not observing any difference from tweaking the
commitSizeandinitialSize.tajila commentedon Apr 18, 2024
You can use helidon nima https://github.com/tomas-langer/helidon-nima-example.
I would configure jmeter to hit the endpoints (you can toggle the amount of load). The relevant endpoint in this case would be
http://localhost:8080/parallel?...I noticed pretty significant deltas between compressedrefs and non-compressedrefs with this example
ThanHenderson commentedon Apr 19, 2024
I am noticing differences for
helidon nimawhen increasing the initial region size:-Xgc:suballocatorInitialSize=<#>No noticeable difference when tweaking the commit size:
-Xgc:suballocatorCommitSize=<#>I was testing with spawning a new server
20times and sending acurl -i http://localhost:8080/parallel\?count\=10000request once on each server. On my machine, with-Xnocompressedrefsthe requests averaged to5.91seconds. With compressedrefs enabled, and the default values forinitialSize(200 MB) andcommitSize(50 MB) the requests averaged to11.87seconds.I iterated over
initialSizes from200 to 1000 MBwith a step size of100 MB. The data in the table below shows those average values from the iterations:On this workload, a static increase does show some benefit. It would be beneficial to test this on a set of workloads though that have observable improvements that may motivate a dynamic solution.
ThanHenderson commentedon Jun 24, 2024
@dmitripivkine @amicic
Do you guys know of any other workloads that could be important to test that could stress the change to the initial reservation size for the suballocator?
Moreover, are there any consequences/reasons not to increase
suballocatorInitialSizethat we should consider?dmitripivkine commentedon Jun 26, 2024
This is more VM than GC question. Suballocator area located below 4G bar is used for artifacts must be 32-bit for Compressed refs (j9class, j9vmthread etc.). The only GC aspect is not prevent most performant 0-shift run where entire heap is located below 4G bar as well. So, by taking more memory for Suballocator initially you can compromise 0-shilt runs.
Also it is not clear for my how exactly Suballocator initial size effects your test performance.
ThanHenderson commentedon Jun 26, 2024
A larger initial size reduces the number of subsequent sub-4G allocations which can be costly due to how -- a brute force looping strategy -- sub-4G memory is reserved.
dmitripivkine commentedon Jun 26, 2024
You comment is not very practical I afraid. The question where bottle neck is.
We looked to the code with @tajila and seems the logic is:
-Xgc:suballocatorInitialSize)-Xgc:suballocatorCommitSize)Would you please try to change this hardcoded value of 8m to larger one (50m or more) and try to measure performance again?
19 remaining items
Add Xgc options for suballocator heap size and quick allocation
Add Xgc options for suballocator heap size and quick allocation
Add Xgc options for suballocator heap size and quick allocation
Add Xgc options for suballocator heap size and quick allocation
Add Xgc options for suballocator heap size and quick allocation