You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Marist has requested we replace the existing Linux 390 systems with ones created on their new self-serve cloud environment by October 15th. @AdamBrousseau and I have been added to the new environment as owners, but we can add others as requested.
This issue is for tracking the replacement progress and the verification of the machine via build & test pipelines before making them live in nightly and PR builds.
rhel8 is currently available, but rhel9 and ub22 are not.
Some of the ub20 can be converted to rhel8, I don't think that can be done for the rhel7 systems as they are used for compile atm and not within a docker container. ub20-390-7 & ub20-390-8 convert to rhel8??
When ub22 is available we can upgrade 2 or all the ub20 guests at that time. Sound okay with that distribution?
I disabled and annotated the old systems on jenkins so just the new ones will carry the nightly build/test load. @pshipton if there are problems they can be re-enabled.
rh8-390-1 & rh8-390-2 are not yet in service but num of old systems == num of new systems
Redhat 8 testing works for all JDK8 pipelines but for JDK11 there is a failure in sanity.openjdk, specifically jdk11_tier1_pack200_0 and jdk11_tier1_pack200_1
16:41:23 STDERR:
16:41:23 java.io.IOException: Files are differ starting at position: 36
16:41:23 at PackTestZip64.compareTwoFiles(PackTestZip64.java:117)
16:41:23 at PackTestZip64.testPacking(PackTestZip64.java:89)
16:41:23 at PackTestZip64.main(PackTestZip64.java:48)
16:41:23 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
16:41:23 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
16:41:23 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:41:23 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
16:41:23 at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
16:41:23 at java.base/java.lang.Thread.run(Thread.java:839)
Both these tests run fine on existing Redhat 7 and Ubuntu 20 test systems. This JDK11 test grinder is on the last nightly source levels targeted at rh8-390-2. This system passed all JDK8 testing. https://openj9-jenkins.osuosl.org/job/Grinder/1410/console
The failure appears to be #14948 caused by z15 hardware zlib compression. I thought it was RH9 only, but I guess it affects RH8 now as well. We don't have any solution for this failure I'm aware of except to disable the hardware compression. I believe setting the environment variable DFLTCC=0 will do that.
Looking at the Linux mainframe blog [1], from RH 8.4 or newer. NXU would be enabled by default [1], so for now, as Peter suggested, we may have to disable through environment variable to get that test pass. I still have not get chance to do investigations on Pack file failures, I will try to make some progress on that front this week.
Same tests fail with DFLTCC=0 set via jenkins and .bashrc, so that doesn't appear to be the solution in this case.
I am going to leave these systems unlabeled so test jobs don't land on them and ruin nightly/omr.acceptance or pr builds. I'll watch #14948 and retest as a solution becomes available.
@jdekonin I found another way you can try: Restart Linux® with the kernel parameter setting dfltcc=off.
Maybe the environment variable doesn't get passed along to the test.
Setting via grub didn't work either, which lead to another problem of being unrecoverable and needing to be rebuilt. My thoughts were to wait on #14948, the systems would work for common jenkins tasks though.
Activity
jdekonin commentedon Oct 4, 2022
testare these pipelinessanity.functional,extended.functional,sanity.system,extended.system,sanity.openjdkrh8-390-1 - test only; problem with core name, corrected and restarted jobs.
rh8-390-2 - test only
pshipton commentedon Oct 4, 2022
Can we update to any newer OSes while we are doing this? ub22, rhel8, rhel9
jdekonin commentedon Oct 4, 2022
rhel8 is currently available, but rhel9 and ub22 are not.
Some of the ub20 can be converted to rhel8, I don't think that can be done for the rhel7 systems as they are used for compile atm and not within a docker container. ub20-390-7 & ub20-390-8 convert to rhel8??
When ub22 is available we can upgrade 2 or all the ub20 guests at that time. Sound okay with that distribution?
pshipton commentedon Oct 4, 2022
I approve the plan.
jdekonin commentedon Oct 12, 2022
I disabled and annotated the old systems on jenkins so just the new ones will carry the nightly build/test load. @pshipton if there are problems they can be re-enabled.
rh8-390-1 & rh8-390-2 are not yet in service but
num of old systems==num of new systemsjdekonin commentedon Oct 12, 2022
Tested for OMR builds, both pass so I disabled the 2 ub16 nodes on OMR jenkins as well.
rh7-390-7 - https://ci.eclipse.org/omr/job/Build-linux_390-64/3559/console
rh7-390-8 - https://ci.eclipse.org/omr/job/Build-linux_390-64/3560/console
Update: PR to remove old OMR systems eclipse-cbi/jiro#228
jdekonin commentedon Oct 14, 2022
These systems have been remove from service/jenkins.
jdekonin commentedon Oct 14, 2022
Renamed systems
rh7-390-5 -> rh7-390-1
rh7-390-6 -> rh7-390-2
rh7-390-7 -> rh7-390-3
rh7-390-8 -> rh7-390-4
ub20-390-4 -> ub20-390-1
ub20-390-5 -> ub20-390-2
ub20-390-6 -> ub20-390-3
ub20-390-7 -> ub20-390-4
ub20-390-3 -> ub20-390-5
ub20-390-8 -> ub20-390-6
jdekonin commentedon Oct 14, 2022
Just the 2 Redhat8 installs to test left on this issue.
jdekonin commentedon Oct 17, 2022
Redhat 8 testing works for all JDK8 pipelines but for JDK11 there is a failure in sanity.openjdk, specifically
jdk11_tier1_pack200_0andjdk11_tier1_pack200_1eg.
https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Personal/28/consoleFull
Both these tests run fine on existing Redhat 7 and Ubuntu 20 test systems. This JDK11 test grinder is on the last nightly source levels targeted at
rh8-390-2. This system passed all JDK8 testing.https://openj9-jenkins.osuosl.org/job/Grinder/1410/console
jdekonin commentedon Oct 18, 2022
@llxia @pshipton need some direction here on possible resolutions on the previous comment.
pshipton commentedon Oct 18, 2022
The failure appears to be #14948 caused by z15 hardware zlib compression. I thought it was RH9 only, but I guess it affects RH8 now as well. We don't have any solution for this failure I'm aware of except to disable the hardware compression. I believe setting the environment variable
DFLTCC=0will do that.@r30shah @joransiu fyi
r30shah commentedon Oct 18, 2022
Looking at the Linux mainframe blog [1], from RH 8.4 or newer. NXU would be enabled by default [1], so for now, as Peter suggested, we may have to disable through environment variable to get that test pass. I still have not get chance to do investigations on Pack file failures, I will try to make some progress on that front this week.
[1]. https://linux.mainframe.blog/zlib-acceleration/
jdekonin commentedon Oct 18, 2022
Both are running 8.6, so I applied env var to the jenkins node configuration as suggested and retesting now.
https://openj9-jenkins.osuosl.org/job/Grinder/1415/
https://openj9-jenkins.osuosl.org/job/Grinder/1418/
jdekonin commentedon Oct 19, 2022
Same tests fail with
DFLTCC=0set via jenkins and .bashrc, so that doesn't appear to be the solution in this case.I am going to leave these systems unlabeled so test jobs don't land on them and ruin nightly/omr.acceptance or pr builds. I'll watch #14948 and retest as a solution becomes available.
pshipton commentedon Oct 19, 2022
@jdekonin I found another way you can try: Restart Linux® with the kernel parameter setting dfltcc=off.
Maybe the environment variable doesn't get passed along to the test.
AdamBrousseau commentedon Nov 30, 2022
Did we resolve the RH8 issue? Machines are still disabled.
jdekonin commentedon Nov 30, 2022
Setting via grub didn't work either, which lead to another problem of being unrecoverable and needing to be rebuilt. My thoughts were to wait on #14948, the systems would work for common jenkins tasks though.
AdamBrousseau commentedon Nov 30, 2022
Put back all labels except for test. Annotated RH8 the machines with that issue.