Context
PR #303 (ci: nightly evaluation harness, dashboard, alerting, and automated bumps) landed the CI/CD plan end-to-end. Its Phase 4 (automated bumps + reproducible deps) work wired up dependency-bump automation, and along with it bump-validate.yml, which runs the full GPU evaluation matrix on the single self-hosted MI350 runner for every PR that touches the stack-defining files:
# .github/workflows/bump-validate.yml
on:
pull_request:
paths:
- "docker/**"
- "requirements.txt"
- "requirements-dev.txt"
- "config/ci/ci-constraints.txt"
Dependabot (.github/dependabot.yml) opens weekly pip / docker (ROCm digest) / github-actions bump PRs, and each of those bump PRs touches exactly these paths — so every automated bump PR raise kicks off a heavy GPU eval on the one shared runner (eval-reusable.yml, timeout-minutes: 150). That is expensive and serializes the runner behind bump/CI PRs.
Problem
- The GPU eval runs per PR raise for any stack-touching change (every Dependabot bump, plus any human PR editing
docker/** / requirements*.txt / ci-constraints.txt).
- We only have a single self-hosted GPU runner, so per-PR bump validation contends with (and delays) other GPU work.
- The nightly evaluation (
nightly-eval.yml) already installs the wheel, builds the pinned ROCm container from the repo, and runs the identical matrix + baseline comparison against main. A merged stack bump is therefore already exercised by the very next nightly.
Proposal
Move Phase-4 bump validation to nightly instead of running on each PR raise:
- Drop the per-PR
pull_request trigger for the bump GPU eval (retire bump-validate.yml).
- Rely on
nightly-eval.yml (which shares eval-reusable.yml) to catch the correctness/baseline impact of merged stack bumps post-merge.
- Keep
workflow_dispatch on nightly-eval.yml available for on-demand validation of a specific bump when a maintainer wants pre-merge confirmation.
- Update the doc/comment references (
nightly-eval.yml, eval-reusable.yml, docs/ci-nightly-eval.md, config/ci/nightly_eval_matrix.yaml) so bump validation is documented as nightly rather than per-PR.
This keeps the fail-closed regression coverage for bumps while freeing the shared runner from a full matrix run on every bump/stack PR.
Acceptance criteria
- Stack-touching PRs (incl. Dependabot bumps) no longer trigger a GPU eval on raise.
- Bump regressions are still caught by the nightly run (and on-demand dispatch).
- References/docs to per-PR bump validation are updated.
Refs #303.
Context
PR #303 (
ci: nightly evaluation harness, dashboard, alerting, and automated bumps) landed the CI/CD plan end-to-end. Its Phase 4 (automated bumps + reproducible deps) work wired up dependency-bump automation, and along with itbump-validate.yml, which runs the full GPU evaluation matrix on the single self-hosted MI350 runner for every PR that touches the stack-defining files:Dependabot (
.github/dependabot.yml) opens weeklypip/docker(ROCm digest) /github-actionsbump PRs, and each of those bump PRs touches exactly these paths — so every automated bump PR raise kicks off a heavy GPU eval on the one shared runner (eval-reusable.yml,timeout-minutes: 150). That is expensive and serializes the runner behind bump/CI PRs.Problem
docker/**/requirements*.txt/ci-constraints.txt).nightly-eval.yml) already installs the wheel, builds the pinned ROCm container from the repo, and runs the identical matrix + baseline comparison againstmain. A merged stack bump is therefore already exercised by the very next nightly.Proposal
Move Phase-4 bump validation to nightly instead of running on each PR raise:
pull_requesttrigger for the bump GPU eval (retirebump-validate.yml).nightly-eval.yml(which shareseval-reusable.yml) to catch the correctness/baseline impact of merged stack bumps post-merge.workflow_dispatchonnightly-eval.ymlavailable for on-demand validation of a specific bump when a maintainer wants pre-merge confirmation.nightly-eval.yml,eval-reusable.yml,docs/ci-nightly-eval.md,config/ci/nightly_eval_matrix.yaml) so bump validation is documented as nightly rather than per-PR.This keeps the fail-closed regression coverage for bumps while freeing the shared runner from a full matrix run on every bump/stack PR.
Acceptance criteria
Refs #303.