[9.0] Fix JDK matrix pipeline after configurable it split (backport #17461)

[mergify]

Release notes

[rn:skip]

What does this PR do?

PR #17219 introduced configurable split quantities for IT tests, which resulted in broken JDK matrix pipelines (e.g. as seen via the elastic internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

 File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263 def integration_tests(self, part: int, parts: int) -> JobRetValues: ^^^ SyntaxError: invalid syntax There was a problem rendering the pipeline steps. Exiting now. 

)

This PR fixes the above problem.

Why is it important/What is the impact to the user?

Restores the functionality of the JDK (Windows/Linux) matrix pipelines.

How to test this PR locally

Linux

BUILDKITE_PIPELINE_NAME="Logstash Linux JDK matrix pipeline" MATRIX_OSES="ubuntu-22.04" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

produces:

Output from generate-steps for Linux

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json steps: - group: ubuntu-22.04/adoptiumjdk_21 key: ubuntu-22_04_adoptiumjdk_21 steps: - label: Initialize annotation key: ubuntu-22.04-adoptiumjdk_21-initialize-annotation command: |-  buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 "### Group: ubuntu-22.04 / adoptiumjdk_21  | **Status** | **Test** |  | --- | ----|  "  - label: Java Unit Test key: ubuntu-22_04_adoptiumjdk_21-java-unit-test depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport ENABLE_SONARQUBE=\"false\"\nci/unit_tests.sh java\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Java Unit Test |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Java Unit Test |\n\"\nfi\n " - label: Ruby Unit Test key: ubuntu-22_04_adoptiumjdk_21-ruby-unit-test depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/unit_tests.sh ruby\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Ruby Unit Test |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Ruby Unit Test |\n\"\nfi\n " - label: Integration Tests - 1/3 key: ubuntu-22_04_adoptiumjdk_21-integration-tests-1-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 0 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 1/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 1/3 |\n\"\nfi\n " - label: Integration Tests - 2/3 key: ubuntu-22_04_adoptiumjdk_21-integration-tests-2-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 1 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 2/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 2/3 |\n\"\nfi\n " - label: Integration Tests - 3/3 key: ubuntu-22_04_adoptiumjdk_21-integration-tests-3-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 2 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 3/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 3/3 |\n\"\nfi\n " - label: IT Persistent Queues - 1/3 key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-1-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 0 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 1/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 1/3 |\n\"\nfi\n " - label: IT Persistent Queues - 2/3 key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-2-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 1 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 2/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 2/3 |\n\"\nfi\n " - label: IT Persistent Queues - 3/3 key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-3-of-3 depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 2 3\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 3/3 |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 3/3 |\n\"\nfi\n " - label: x-pack unit tests key: ubuntu-22_04_adoptiumjdk_21-x-pack-unit-test depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/unit_tests.sh\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack unit tests |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack unit tests |\n\"\nfi\n " - label: x-pack integration key: ubuntu-22_04_adoptiumjdk_21-x-pack-integration depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd retry: automatic: - limit: 3 command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/integration_tests.sh\n \nif [[ $$? -ne 0 ]]; then\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack integration |\n\"\n exit 1\nelse\n buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack integration |\n\"\nfi\n "

Windows

BUILDKITE_PIPELINE_NAME="Logstash Windows JDK matrix pipeline" MATRIX_OSES="windows-2025" MATRIX_JDKS="adoptiumjdk_21" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

results in the following output:

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json steps: - group: windows-2025/adoptiumjdk_21 key: windows-2025_adoptiumjdk_21 steps: - label: Initialize annotation key: windows-2025-adoptiumjdk_21-initialize-annotation command: |-  buildkite-agent annotate --style=info --context=windows-2025_adoptiumjdk_21 "### Group: windows-2025 / adoptiumjdk_21  | **Status** | **Test** |  | --- | ----|  "  - label: Java Unit Test key: windows-2025_adoptiumjdk_21-java-unit-test depends_on: windows-2025-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-windows-2025 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd artifact_paths: - build_reports.zip retry: automatic: - limit: 3 command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Java Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 java\" -Annotate\n " - label: Ruby Unit Test key: windows-2025_adoptiumjdk_21-ruby-unit-test depends_on: windows-2025-adoptiumjdk_21-initialize-annotation agents: provider: gcp imageProject: elastic-images-prod image: family/platform-ingest-logstash-multi-jdk-windows-2025 machineType: n2-standard-4 diskSizeGb: 200 diskType: pd-ssd artifact_paths: - build_reports.zip retry: automatic: - limit: 3 command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Ruby Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 ruby\" -Annotate\n "

Related issues

PR #17219

---

This is an automatic backport of pull request #17461 done by [Mergify](https://mergify.com).

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 181c52a

cc @dliappis

[dliappis]

Clean backport LGTM