Repository navigation
Description
After upgrading openj9 from 21.0.3 to 21.0.7, the application will inexplicably experience OOM.
During concurrent thread operations, the size of Concurrent HashMap suddenly increases from 1 to the maximum value for shaping. The code Sets.newConcurrentHashSet() is an implementation of guava, and the underlying layer uses Concurrent HashMap
The HashMap used at the bottom layer of Sets.newHashSet (allocStacks) is OOM because the size reaches the maximum shaping value, resulting in the size of newHashSet also being the maximum shaping value.
why the size of Concurrent HashMap suddenly increases from 1 to its maximum value during concurrent thread operations
This only occurs when using shared class cache, and if the shared class cache is removed, it will not occur.
ConnectionAllocMonitor:
private Set<ConnectionAllocStack> allocStacks = Sets.newConcurrentHashSet();
private long lastOutTime = 0;
public void addConnectionAllocStack(ConnectionAllocStack allocStack){
allocStacks.add(allocStack);
logger.info("====" + this + ": " + allocStacks.size());
if(allocStacks.size()>=poolMaxSize&){
Set<ConnectionAllocStack> tempallocStacks = Sets.newHashSet(allocStacks);
logger.info(">>>>>>>>> connection pool is full ,output statck info:");
tempallocStacks.stream().forEach(stack->{
});
logger.info(">>>>>>>>> end connection output statck");
lastOutTime= System.currentTimeMillis();
}
}
log:
2025-05-30 09:56:14 453 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][Thread-9] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@234b0671: 1
2025-05-30 09:56:14 462 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][Thread-9] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@556d58e7: 1
2025-05-30 09:56:14 652 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][Thread-9] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@556d58e7: 1
2025-05-30 09:56:21 946 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][app-executor-lv:1-idx:1] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@556d58e7: 2147483647
2025-05-30 09:56:21 946 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][app-executor-lv:1-idx:3] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@556d58e7: 2147483647
2025-05-30 09:56:21 947 INFO [com.tmp.db.jdbi.ConnectionAllocMonitor][app-executor-lv:1-idx:6] - ====com.tmp.db.jdbi.ConnectionAllocMonitor@556d58e7: 2147483647
Activity
[-]openj9 21.0.7 ooom[/-][+]openj9 21.0.7 oom[/+]pshipton commentedon May 30, 2025
Does using the shared class cache with the option
-Xnoaotresolve the problem?pshipton commentedon May 30, 2025
Also as a separate test, pls delete the existing shared class cache, and then run with
-XX:-ShareOrphansalong with a shared class cache to see if that resolves the problem.lp2jx0309 commentedon Jun 3, 2025
Can solve the problem
lp2jx0309 commentedon Jun 3, 2025
Not using shared class cache will not result in OOM, but once added, it will result in OOM. What is the reason for OOM?
pshipton commentedon Jun 3, 2025
If using
-Xnoaotsolves the problem, then there is an issue with the AOT code in the share cache.Is it possible to get a reproducible test case?
@hzongaro fyi
lp2jx0309 commentedon Jun 3, 2025
i cannot reproduce the test cases.My usage scenario is that the class cache is created in Docker image construction and used directly in the business container. If the class cache is regenerated in the business container and reused, it will not cause OOM.
lp2jx0309 commentedon Jun 4, 2025
Through exclusion method, it was found that it was caused by this submission
83b0f89
@pshipton
pshipton commentedon Jun 4, 2025
This is #20937
What platform are you running on?
@keithc-ca fyi
DHbigfart commentedon Jun 12, 2025
What's the progress on this issue? I also encountered this problem
pshipton commentedon Jun 12, 2025
@DHbigfart pls confirm which platform and version you are running on.
lp2jx0309 commentedon Jun 13, 2025
linux x86
DHbigfart commentedon Jun 13, 2025
pshipton commentedon Jun 13, 2025
To proceed we need either a system core file produced at the time of the OOM, or a reproducible test case.
Even a javacore file would be helpful.
hzongaro commentedon Aug 13, 2025
Moving this out for now until we have a core file or some other dump file to work.
hzongaro commentedon Dec 1, 2025
I'll move this to the "Future" release until we have something that will allow investigation to proceed.