[#8133] Add s390x as a parameter for Jenkins #956

mattrpav · 2023-01-12T15:20:45Z

No description provided.

mattrpav · 2023-01-12T15:28:13Z

@srajmane s390x build convo resumes here

mattrpav · 2023-01-12T16:47:56Z

Ugh.. this approach switches nodes b/w stages. Got to make another pass.. Apache INFRA backlogged ~ 30 min to get a build node.

mattrpav · 2023-01-12T19:24:56Z

s390x compiled fine w/ multiple JDKs

Unit tests running now:
ref: https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/PR-956/3/pipeline/174

mattrpav · 2023-01-13T16:07:53Z

Notes:

Some tests failing
Total execution time exceeded max limit (20 hrs)

ref: https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/view/change-requests/job/PR-956/3/

mattrpav · 2023-01-14T20:13:26Z

I'm going to go ahead and merge this. There is no day-to-day impact, this simply allows us to manually select a s390x build.

@srajmane there are still some test failures. Not sure if it is due to a s390x-specific bug or if it is due to the s390x build server being super busy.

srajmane · 2023-01-16T14:30:11Z

I'm going to go ahead and merge this. There is no day-to-day impact, this simply allows us to manually select a s390x build.

@srajmane there are still some test failures. Not sure if it is due to a s390x-specific bug or if it is due to the s390x build server being super busy.

@mattrpav Thank you for the update.
I tried failed test cases locally. Looks like the tests were flaky which passed after retries.
Could you please re-trigger the build for s390x?

mattrpav · 2023-01-16T17:16:47Z

@srajmane we do have a new setting that re-runs failed unit tests 3 times. My suspicion is that Apache's s390x's are super busy (and they don't have a lot of them) so we probably should not expect regular 'green' runs.

I kicked off a new s390x pipeline run here:
https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231

mattrpav · 2023-01-17T16:08:24Z

A few tests failed with 'port already in use' issue



Test Name | Duration | Age
-- | -- | --
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-demo.xml] | 1.7 sec | 1
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-dynamic-network-broker1.xml] | 1 sec | 1
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-jdbc-performance.xml] | 1.1 sec | 1
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-throughput.xml] | 1 sec | 1
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-security.xml] | 1 sec | 1
org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-specjms.xml] | 1 sec | 1

[Test Name](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/#)	[Duration](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/#)	[Age](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/#)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-demo.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_demo_xml_/)	1.7 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-dynamic-network-broker1.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_dynamic_network_broker1_xml_/)	1 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-jdbc-performance.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_jdbc_performance_xml_/)	1.1 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-throughput.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_throughput_xml_/)	1 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-security.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_security_xml_/)	1 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)
 [org.apache.activemq.config.BrokerXmlConfigStartTest.testStartBrokerUsingXmlConfig1[activemq-specjms.xml]](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/testReport/org.apache.activemq.config/BrokerXmlConfigStartTest/testStartBrokerUsingXmlConfig1_activemq_specjms_xml_/)	1 sec	[1](https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/231/)

srajmane · 2023-01-18T13:13:22Z

@mattrpav Yes, l will check this issue with nodes.

namrata-ibm · 2023-02-27T12:52:15Z

Hi @mattrpav , we have observed that builds of another Apache package have same error mentioned above and the issue seems to be specific to Jenkins node.
Our team member @cwsolee (Cindy Lee) did try to reach out to Apache Infra team, @gmcdonald (Gavin), they’re trying to add more resource to the build environment because we think these errors are likely caused by lack of resources.
Will keep you posted on this. Thank you.

namrata-ibm · 2023-04-03T14:24:20Z

Hi @mattrpav the s390x nodes seem to be better now. Could you please help in triggering a job on s390x?
Also wanted to know about regular s390x triggers. As of now it is manually triggered correct? Can we have a schedule to test activemq on a regular basis on s390x ?

mattrpav · 2023-04-03T14:27:31Z

@namrata-ibm I triggers the following ci-job for s390x

https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/264/pipeline

namrata-ibm · 2023-05-09T12:28:49Z

Hi @mattrpav Regarding jdk20 unavailable for s390x, we tried following up but ETA is not known yet, wanted to know your thoughts on enabling s390x builds.
Tested that build and test with jdk 11/17 locally works on s390x, should we continue builds without jdk 20 on s390x?

namrata-ibm · 2023-07-10T10:38:54Z

@mattrpav The latest jdk 20 is now installed on s390x nodes, could you please trigger s390x build once?
Thank you for your help.

mattrpav · 2023-07-10T14:19:24Z

@namrata-ibm

CI job:
https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/303/pipeline

namrata-ibm · 2023-07-10T14:29:45Z

Thank you @mattrpav , Looks like jdk 20 is still not configured or not at correct path, I am following up via Jira issue
edit - updated the issue link

namrata-ibm · 2023-07-18T14:39:21Z

@mattrpav as discussed in Jira issue , could you please trigger a build for s390x specifically on jenkins-s390x-6 node?
Thank you for your help

namrata-ibm · 2023-07-24T10:29:37Z

@mattrpav gentle reminder, could you please trigger a build for s390x on jenkins-s390x-6 node?

mattrpav · 2023-07-31T00:16:34Z

@namrata-ibm We do not currently have the ability to specify node in our Jenkins parameters. My sense is we shouldn't ever target a node.

Do all the s390x nodes not have the current JDKs?

I kicked this one off:
https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/304/pipeline

gmcdonald · 2023-07-31T08:31:34Z

They all have the current JDKs. It is the symlink of 'latest*' that is the problem. It is hard to match 'latest' on all OS platforms to the same JDK, as the same JDK is not available on all platforms. So I asked for a test on a workaround that I did on one node. As you can not alter your Jenkinsfile to point to just one node, then I will add the workaround to all s390x nodes. Will comment here shortly when done.

gmcdonald · 2023-07-31T10:31:22Z

Added to all s390x nodes, please kick a build

mattrpav · 2023-07-31T14:23:16Z

https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/307/pipeline

namrata-ibm · 2023-08-02T08:49:56Z

Thank you @mattrpav , in above job all "Build JDK xx", "Verify" and "Tests" stages have passed.
"Deploy" stage is showing [INFO] ActiveMQ :: Client Jakarta ......................... FAILURE which doesn't seem to be related to s390x as it is also seen on amd job here:
https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/309/consoleFull

Is it ok to start a regular s390x build for every merge commit like amd jobs?

mattrpav · 2023-08-02T15:26:51Z

@namrata-ibm we need to figure out why the deploy step is failing and fix that.

Also, I'd prefer not to add s390x for every run until after the Jakarta migration work (v5.19.x) is done.

namrata-ibm · 2023-12-04T11:45:42Z

Hi @mattrpav , since v6.0.0 is out, is it ok to trigger s390x build for every run?

rposts · 2024-05-08T12:27:15Z

hi @mattrpav wanted to follow-up on the state of s390x builds - do you think we are in a position to reenable it? Thx!

mattrpav · 2024-05-08T13:23:26Z

@rposts no, unfortunately I don't think we are in a good state to do that on every PR.

I do not think there are enough s390x's build servers to run builds on every PR.
Test execution times are long.

The ActiveMQ unit tests need to be modernized and optimized. I estimate we could cut test run times in half.

I kicked a s390x main branch test run just now:
https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/443/pipeline/

EDIT: .. and the build failed. No JDK 22 on s390x yet.

rposts · 2024-05-08T14:01:26Z

Thanks @mattrpav for quick response.

Currently Eclipse Temurin is releasing s390x JDK 22. My colleague @namrata-ibm can raise an issue with Apache Infra team to add JDK 22 to s390x nodes.

I do understand about your point about resources/execution time. Do you have any suggestions on how we can make this work given these constraints and until unit tests are optimized? Can we start with weekly/bi-weekly build triggers on s390x ?

Thanks for your help!

mattrpav · 2024-05-08T14:18:15Z

@rposts great, adding JDK 22 to s390x will get us past this current hurdle.

I'm good w/ adding an updated build trigger, but I currently do not have time to work on it.

rposts · 2024-05-23T12:31:33Z

@mattrpav wrt adding JDK22 to s390x nodes, It seems like underlying process for requesting changes to infrastructure has changed recently. As noted in this issue only ActiveMQ PMC member can open such requests. Could you please help us navigate this when you have free cycle?

Thanks so much for your continued assistance.

mattrpav · 2024-05-23T15:09:12Z

JIRA created https://issues.apache.org/jira/browse/INFRA-25817

namrata-ibm · 2024-08-09T15:00:47Z

@mattrpav JDK22 is now available on s390x nodes, could you please trigger a build?
Thank you very much

mattrpav · 2024-08-09T15:22:22Z

@namrata-ibm I attempted to kick a job, but all s390x nodes are offline at the moment.

gmcdonald · 2024-08-09T15:25:12Z

Offline, how, where ?

Note that all Shared nodes live on the main Operations Center and get leased out to ci-builds. ci-cassandra, etc, etc

https://jenkins-ccos.apache.org/view/Shared%20-%20s390x%20nodes/

All are available and currently 2 in use.
It can take a few minutes for a build to grab one.

mattrpav · 2024-08-09T15:31:01Z

Hi @gmcdonald-- good call. Yes, the Jenkins job page flashed a message that s390x nodes were offline, then picked up a node a few minutes later.

The job did fail on 'mvn' not being found. Perhaps s390x needs different 'mvn' tool name from other platforms in the Jenkinsfile?

link to job: https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/476

gmcdonald · 2024-08-09T15:37:43Z

Can I get a link to the JenkinsFile ?
Note that Maven is not usually installed by default as a system tool.

See : https://cwiki.apache.org/confluence/display/INFRA/Maven+Installation+Matrix for usual locations we install various Maven versions across the board.

mattrpav · 2024-08-09T15:56:47Z

Hi @gmcdonald - Thanks, see Jenkinsfile below.

ref: https://github.com/apache/activemq/blob/main/Jenkinsfile

One thing I noted was that the names of the tools can change between platforms (see the Windows JDKs). If the Jenkins tool alias is the same across all, it would make for simpler to define multi-platform jobs.

gmcdonald · 2024-08-09T16:04:28Z

ergh ok, thanks, looks like maven_3_latest is broken, give me a bit for fix and deploy and new one

gmcdonald · 2024-08-10T11:24:13Z

See: https://github.com/apache/infrastructure-p6/pull/1909/files

gmcdonald · 2024-08-10T12:38:09Z

ok Maven 3 issue should be resolved now, please test. @mattrpav

mattrpav · 2024-08-10T13:52:53Z

Build started: https://ci-builds.apache.org/blue/organizations/jenkins/ActiveMQ%2FActiveMQ/detail/main/478

namrata-ibm · 2024-08-12T06:16:37Z

Thank you @gmcdonald for helping with the infra.

@mattrpav The above job triggered on s390x - failed with error:

JournalArchiveTest.testRecoveryOnArchiveFailure:122 » UnsupportedOperation The Security Manager is deprecated and will be removed in a future release

This fails on s390x as tests were run with jdk22 whereas on amd, jdk 17 is used and the tests pass with a warning about The Security Manager:

[INFO] Running org.apache.activemq.store.kahadb.JournalArchiveTest
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.apache.activemq.store.kahadb.JournalArchiveTest (file:/home/jenkins/jenkins-agent/workspace/ActiveMQ_ActiveMQ_main/activemq-kahadb-store/target/test-classes/)
WARNING: Please consider reporting this to the maintainers of org.apache.activemq.store.kahadb.JournalArchiveTest
WARNING: System::setSecurityManager will be removed in a future release
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.645 s -- in org.apache.activemq.store.kahadb.JournalArchiveTest

Could you please trigger a job with jdk 17?

namrata-ibm · 2024-08-13T04:55:21Z

The latest build for s390x is passing, Thank you @mattrpav!
https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/479/

gmcdonald · 2024-08-13T08:46:13Z

awesome

namrata-ibm · 2024-08-20T11:49:53Z

@mattrpav can we enable regular s390x builds ? The last s390x job took similar time as Intel job to pass.
Thank you.

namrata-ibm · 2024-11-12T06:15:13Z

@mattrpav can we please consider running s390x builds on every commit similar to amd64?

rposts · 2025-02-24T14:09:14Z

@mattrpav - can we revisit this enablement? I'll be happy to assist with any s390x blockers we may see along the way.

Thanks!

rishigits · 2025-08-21T12:08:19Z

@mattrpav - circling back on this re-enablement - could we revisit? Thanks for your attention in the matter.

mattrpav · 2025-08-21T13:38:14Z

I don't think it makes sense to add s390x for every commit. I don't believe Apache Infra has enough s390x capacity for that. Perhaps IBM could donate more s390x? Or perhaps look into a GitHub action (does GitHub offer s390x?)

I think a weekly scheduled job would make sense -- perhaps a Sunday early morning. The Jenkinsfile would need to be patched-- if you could take a pass at making that change, that would be helpful.

Fresh s390x build started: https://ci-builds.apache.org/job/ActiveMQ/job/ActiveMQ/job/main/599/

rposts · 2025-08-21T14:31:19Z

Thanks @mattrpav. I think weekly scheduled job will work for now. Let me see if we can do that.

Also, I'd like to update you with our work on enabling GHA support on s390x. We now have hosted GHA actions runners available for IBM Z and gradually onboarding opensource communities to utilize it as we work on expanding the capacity.

I'll be happy to engage with ActiveMQ to see if this solution helps with the enablement. It will be useful to get an idea of daily commits from you on ActiveMQ to see if it can be contained in the existing runners. An onboarding document is available.

Please let me know your thoughts. Thanks!

mattrpav · 2025-08-21T15:50:03Z

@rposts ActiveMQ is at ~1 commit per day (note: that many times these get batched up where we'll do a bunch of updates in one day leading up to a release)

namrata-ibm · 2025-10-14T04:35:22Z

@mattrpav could you please have a look at #1496 - this will trigger s390x job weekly once as suggested here

mattrpav mentioned this pull request Jan 12, 2023

Add Jenkinsfile for s390x #907

Closed

[#8133] Add s390x as a parameter for Jenkins

c7dfcd5

mattrpav merged commit c582864 into apache:main Jan 14, 2023

mattrpav deleted the AMQ-8133 branch January 14, 2023 20:14

SanyogDeshmukh mentioned this pull request Oct 6, 2025

Weekly s390x Job #1496

Open

[#8133] Add s390x as a parameter for Jenkins #956

[#8133] Add s390x as a parameter for Jenkins #956

Uh oh!

Conversation

mattrpav commented Jan 12, 2023

Uh oh!

mattrpav commented Jan 12, 2023

Uh oh!

mattrpav commented Jan 12, 2023

Uh oh!

mattrpav commented Jan 12, 2023

Uh oh!

mattrpav commented Jan 13, 2023

Uh oh!

mattrpav commented Jan 14, 2023

Uh oh!

srajmane commented Jan 16, 2023

Uh oh!

mattrpav commented Jan 16, 2023

Uh oh!

mattrpav commented Jan 17, 2023

Uh oh!

srajmane commented Jan 18, 2023

Uh oh!

namrata-ibm commented Feb 27, 2023

Uh oh!

namrata-ibm commented Apr 3, 2023

Uh oh!

mattrpav commented Apr 3, 2023

Uh oh!

namrata-ibm commented May 9, 2023

Uh oh!

namrata-ibm commented Jul 10, 2023

Uh oh!

mattrpav commented Jul 10, 2023

Uh oh!

namrata-ibm commented Jul 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

namrata-ibm commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

namrata-ibm commented Jul 24, 2023

Uh oh!

mattrpav commented Jul 31, 2023

Uh oh!

gmcdonald commented Jul 31, 2023

Uh oh!

gmcdonald commented Jul 31, 2023

Uh oh!

mattrpav commented Jul 31, 2023

Uh oh!

namrata-ibm commented Aug 2, 2023

Uh oh!

mattrpav commented Aug 2, 2023

Uh oh!

namrata-ibm commented Dec 4, 2023

Uh oh!

rposts commented May 8, 2024

Uh oh!

mattrpav commented May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rposts commented May 8, 2024

Uh oh!

mattrpav commented May 8, 2024

Uh oh!

rposts commented May 23, 2024

Uh oh!

mattrpav commented May 23, 2024

Uh oh!

namrata-ibm commented Aug 9, 2024

Uh oh!

mattrpav commented Aug 9, 2024

Uh oh!

gmcdonald commented Aug 9, 2024

Uh oh!

mattrpav commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

namrata-ibm commented Jul 10, 2023 •

edited

Loading

namrata-ibm commented Jul 18, 2023 •

edited

Loading

mattrpav commented May 8, 2024 •

edited

Loading

mattrpav commented Aug 9, 2024 •

edited

Loading

mattrpav commented Aug 9, 2024 •

edited

Loading