Skip to content

Linux 390 jenkins system replacement #16037

Closed

Description

@jdekonin

Marist has requested we replace the existing Linux 390 systems with ones created on their new self-serve cloud environment by October 15th. @AdamBrousseau and I have been added to the new environment as owners, but we can add others as requested.

This issue is for tracking the replacement progress and the verification of the machine via build & test pipelines before making them live in nightly and PR builds.

These are the older systems requiring replacement

  • rh7-390-1 -> rh7-390-4
  • ub18-390-1 -> ub18-390-2
  • ub20-390-1 -> ub20-390-2

New systems (we have 2 more than before)

  • rh7-390-5 -> rh7-390-8
  • ub20-390-3 -> ub20-390-8

Activity

  1. self-assigned this
    on Oct 4, 2022
  2. jdekonin commented on Oct 4, 2022

    @jdekonin
    Author

    test are these pipelines sanity.functional,extended.functional,sanity.system,extended.system,sanity.openjdk



  3. pshipton commented on Oct 4, 2022

    @pshipton
    Member

    Can we update to any newer OSes while we are doing this? ub22, rhel8, rhel9

  4. jdekonin commented on Oct 4, 2022

    @jdekonin
    Author

    rhel8 is currently available, but rhel9 and ub22 are not.

    Some of the ub20 can be converted to rhel8, I don't think that can be done for the rhel7 systems as they are used for compile atm and not within a docker container. ub20-390-7 & ub20-390-8 convert to rhel8??

    When ub22 is available we can upgrade 2 or all the ub20 guests at that time. Sound okay with that distribution?

  5. pshipton commented on Oct 4, 2022

    @pshipton
    Member

    I approve the plan.

    • ub20-390-7 & ub20-390-8 convert to rhel8
    • When ub22 is available we can upgrade 2 of the ub20 guests
  6. jdekonin commented on Oct 12, 2022

    @jdekonin
    Author

    I disabled and annotated the old systems on jenkins so just the new ones will carry the nightly build/test load. @pshipton if there are problems they can be re-enabled.

    rh8-390-1 & rh8-390-2 are not yet in service but num of old systems == num of new systems

  7. jdekonin commented on Oct 12, 2022

    @jdekonin
    Author

    Tested for OMR builds, both pass so I disabled the 2 ub16 nodes on OMR jenkins as well.
    rh7-390-7 - https://ci.eclipse.org/omr/job/Build-linux_390-64/3559/console
    rh7-390-8 - https://ci.eclipse.org/omr/job/Build-linux_390-64/3560/console

    Update: PR to remove old OMR systems eclipse-cbi/jiro#228

  8. jdekonin commented on Oct 14, 2022

    @jdekonin
    Author

    These systems have been remove from service/jenkins.

    • rh7-390-1 -> rh7-390-4
    • ub18-390-1 -> ub18-390-2
    • ub20-390-1 -> ub20-390-2
  9. jdekonin commented on Oct 14, 2022

    @jdekonin
    Author

    Renamed systems
    rh7-390-5 -> rh7-390-1
    rh7-390-6 -> rh7-390-2
    rh7-390-7 -> rh7-390-3
    rh7-390-8 -> rh7-390-4
    ub20-390-4 -> ub20-390-1
    ub20-390-5 -> ub20-390-2
    ub20-390-6 -> ub20-390-3
    ub20-390-7 -> ub20-390-4
    ub20-390-3 -> ub20-390-5
    ub20-390-8 -> ub20-390-6

  10. jdekonin commented on Oct 14, 2022

    @jdekonin
    Author

    Just the 2 Redhat8 installs to test left on this issue.

  11. jdekonin commented on Oct 17, 2022

    @jdekonin
    Author

    Redhat 8 testing works for all JDK8 pipelines but for JDK11 there is a failure in sanity.openjdk, specifically jdk11_tier1_pack200_0 and jdk11_tier1_pack200_1

    16:41:23  STDERR:
    16:41:23  java.io.IOException: Files are differ starting at position: 36
    16:41:23  	at PackTestZip64.compareTwoFiles(PackTestZip64.java:117)
    16:41:23  	at PackTestZip64.testPacking(PackTestZip64.java:89)
    16:41:23  	at PackTestZip64.main(PackTestZip64.java:48)
    16:41:23  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    16:41:23  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    16:41:23  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    16:41:23  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    16:41:23  	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
    16:41:23  	at java.base/java.lang.Thread.run(Thread.java:839)
    

    eg.
    https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.openjdk_s390x_linux_Personal/28/consoleFull

    Both these tests run fine on existing Redhat 7 and Ubuntu 20 test systems. This JDK11 test grinder is on the last nightly source levels targeted at rh8-390-2. This system passed all JDK8 testing.
    https://openj9-jenkins.osuosl.org/job/Grinder/1410/console

  12. jdekonin commented on Oct 18, 2022

    @jdekonin
    Author

    @llxia @pshipton need some direction here on possible resolutions on the previous comment.

  13. pshipton commented on Oct 18, 2022

    @pshipton
    Member

    The failure appears to be #14948 caused by z15 hardware zlib compression. I thought it was RH9 only, but I guess it affects RH8 now as well. We don't have any solution for this failure I'm aware of except to disable the hardware compression. I believe setting the environment variable DFLTCC=0 will do that.

    @r30shah @joransiu fyi

  14. r30shah commented on Oct 18, 2022

    @r30shah
    Contributor

    Looking at the Linux mainframe blog [1], from RH 8.4 or newer. NXU would be enabled by default [1], so for now, as Peter suggested, we may have to disable through environment variable to get that test pass. I still have not get chance to do investigations on Pack file failures, I will try to make some progress on that front this week.

    [1]. https://linux.mainframe.blog/zlib-acceleration/

  15. jdekonin commented on Oct 18, 2022

    @jdekonin
    Author

    Both are running 8.6, so I applied env var to the jenkins node configuration as suggested and retesting now.
    https://openj9-jenkins.osuosl.org/job/Grinder/1415/
    https://openj9-jenkins.osuosl.org/job/Grinder/1418/

  16. jdekonin commented on Oct 19, 2022

    @jdekonin
    Author

    Same tests fail with DFLTCC=0 set via jenkins and .bashrc, so that doesn't appear to be the solution in this case.

    I am going to leave these systems unlabeled so test jobs don't land on them and ruin nightly/omr.acceptance or pr builds. I'll watch #14948 and retest as a solution becomes available.

  17. pshipton commented on Oct 19, 2022

    @pshipton
    Member

    @jdekonin I found another way you can try: Restart Linux® with the kernel parameter setting dfltcc=off.
    Maybe the environment variable doesn't get passed along to the test.

  18. AdamBrousseau commented on Nov 30, 2022

    @AdamBrousseau
    Contributor

    Did we resolve the RH8 issue? Machines are still disabled.

  19. jdekonin commented on Nov 30, 2022

    @jdekonin
    Author

    Setting via grub didn't work either, which lead to another problem of being unrecoverable and needing to be rebuilt. My thoughts were to wait on #14948, the systems would work for common jenkins tasks though.

  20. AdamBrousseau commented on Nov 30, 2022

    @AdamBrousseau
    Contributor

    Put back all labels except for test. Annotated RH8 the machines with that issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions