[ML] Make Appex QA/PyTorch tests version-aware on release branches

[edsavage]

Summary

Fixes the spurious PyTorch/QA build failure seen on backport PRs (e.g. build 2879 on the [9.5] backport of #3081), where the synchronous Appex QA trigger went red with:

Searching for artifacts: "build/distributions/ml-cpp-9.6.0-SNAPSHOT-linux-x86_64.zip" within step: build_test_linux-x86_64-RelWithDebInfo
buildkite-agent: fatal: failed to download artifacts: No artifacts found for downloading

Root cause

The downstream appex-qa-stateful-custom-ml-cpp-build-testing pipeline (generated by elastic/qaf-tests, pipeline_generators/local/ml-cpp-custom.py) downloads the ml-cpp artifact from the parent build (--build $BUILDKITE_TRIGGERED_FROM_BUILD_ID) but names it from STACK_VERSION, which defaults to ACTIVE_VERSIONS[0] (the current main SNAPSHOT), and builds ES from ES_BRANCH, which defaults to main. On a release-branch/backport build the parent produced ml-cpp-<release>-SNAPSHOT-... (e.g. 9.5.0), so the 9.6.0-SNAPSHOT lookup missed.

ci:run-pytorch-tests is inherited by backport PRs, so every backport hit this. It didn't block merges (the gating status is posted by the launch step; the synchronous downstream result isn't a required check), but it produced a red, misleading job on every backport.

Fix (ml-cpp side only — no qaf-tests change)

qaf-tests already supports STACK_VERSION/ES_BRANCH overrides and lists our release lines as active (main, 9.5, 9.4, 9.3, 8.19). We just need to pass the right values:

• New sourced helper .buildkite/pipelines/derive_qa_stack_env.sh:
  • STACK_VERSION ← gradle.properties elasticsearchVersion (exactly the artifact version).
  • ES_BRANCH ← main if this build's version equals ml-cpp main's version (fetched from github.com/ghraw/elastic/ml-cpp/main/gradle.properties, the same host the downstream already uses), else <major>.<minor>. This is self-updating — no hard-coded "current dev minor" to bump at feature freeze. Falls back to a branch-name heuristic if that fetch fails.
  • Respects values already set (e.g. PR-comment vars GITHUB_PR_COMMENT_VAR_BRANCH/_VERSION), and writes only to stderr (the generators pipe stdout to buildkite-agent pipeline upload).
• run_pytorch_tests.yml.sh now sources the helper and forwards ES_BRANCH/STACK_VERSION into the trigger (mirroring run_qa_tests.yml.sh, which already forwarded them but had nothing to derive them).
• run_qa_tests.yml.sh sources the same helper, so QA on backports gets the same treatment.

Behaviour

• main builds: derived values equal today's defaults (main / 9.6.0) → no change.
• release-branch/backport builds: now 9.5 / 9.5.0 etc. → the parent's artifact matches and the custom ES distribution is built from the right branch.

Dry-run of the generators confirms the emitted trigger YAML:

# main:          ES_BRANCH: "main"  STACK_VERSION: "9.6.0"
# 9.5 backport:  ES_BRANCH: "9.5"   STACK_VERSION: "9.5.0"

Note

The generator scripts live in the checked-out branch, so this must be backported to 9.5/9.4/8.19 for their backport builds to benefit (same as the ci:skip-es-tests fix). Backports will follow.

Test plan

• Dry-run generators emit correct ES_BRANCH/STACK_VERSION for main and a simulated release build (done locally).
• After backporting to release branches, a real backport PR carrying ci:run-pytorch-tests triggers a downstream QA build that finds the ml-cpp-<release>-SNAPSHOT artifact and runs (no "No artifacts found").
• Confirm main-targeted PRs with ci:run-pytorch-tests are unaffected.

Made with Cursor

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[edsavage]

buildkite build

[edsavage]

buildkite build

[Copilot]

Pull request overview

This PR fixes spurious downstream Appex QA/PyTorch failures on release-branch/backport builds by ensuring the triggered qaf-tests pipeline receives the correct STACK_VERSION (matching the produced ml-cpp artifact version) and ES_BRANCH (matching the corresponding Elasticsearch branch), instead of relying on qaf-tests defaults that assume main.

Changes:

• Add a new sourced helper script to derive STACK_VERSION from gradle.properties and compute ES_BRANCH by comparing against ml-cpp main’s version (with a branch-name heuristic fallback).
• Source the helper from the QA and PyTorch pipeline generator scripts.
• Forward derived ES_BRANCH/STACK_VERSION into the downstream trigger env when non-empty.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
.buildkite/pipelines/run_qa_tests.yml.sh	Sources the new helper so QA triggers on backports use the correct stack version/branch.
.buildkite/pipelines/run_pytorch_tests.yml.sh	Sources the new helper and forwards ES_BRANCH/STACK_VERSION into the downstream PyTorch trigger env.
.buildkite/pipelines/derive_qa_stack_env.sh	New helper that derives STACK_VERSION/ES_BRANCH in a stdout-safe way for piped pipeline uploads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[elastic-vault-github-plugin-prod]

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
✅	9.4
✅	9.5

Manual backport

To create the backport manually run:

backport --pr 3115

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details