Skip to content

ci: move Phase-4 bump validation to nightly instead of running on every PR raise #344

Description

@amd-vivekag

Context

PR #303 (ci: nightly evaluation harness, dashboard, alerting, and automated bumps) landed the CI/CD plan end-to-end. Its Phase 4 (automated bumps + reproducible deps) work wired up dependency-bump automation, and along with it bump-validate.yml, which runs the full GPU evaluation matrix on the single self-hosted MI350 runner for every PR that touches the stack-defining files:

# .github/workflows/bump-validate.yml
on:
  pull_request:
    paths:
      - "docker/**"
      - "requirements.txt"
      - "requirements-dev.txt"
      - "config/ci/ci-constraints.txt"

Dependabot (.github/dependabot.yml) opens weekly pip / docker (ROCm digest) / github-actions bump PRs, and each of those bump PRs touches exactly these paths — so every automated bump PR raise kicks off a heavy GPU eval on the one shared runner (eval-reusable.yml, timeout-minutes: 150). That is expensive and serializes the runner behind bump/CI PRs.

Problem

  • The GPU eval runs per PR raise for any stack-touching change (every Dependabot bump, plus any human PR editing docker/** / requirements*.txt / ci-constraints.txt).
  • We only have a single self-hosted GPU runner, so per-PR bump validation contends with (and delays) other GPU work.
  • The nightly evaluation (nightly-eval.yml) already installs the wheel, builds the pinned ROCm container from the repo, and runs the identical matrix + baseline comparison against main. A merged stack bump is therefore already exercised by the very next nightly.

Proposal

Move Phase-4 bump validation to nightly instead of running on each PR raise:

  • Drop the per-PR pull_request trigger for the bump GPU eval (retire bump-validate.yml).
  • Rely on nightly-eval.yml (which shares eval-reusable.yml) to catch the correctness/baseline impact of merged stack bumps post-merge.
  • Keep workflow_dispatch on nightly-eval.yml available for on-demand validation of a specific bump when a maintainer wants pre-merge confirmation.
  • Update the doc/comment references (nightly-eval.yml, eval-reusable.yml, docs/ci-nightly-eval.md, config/ci/nightly_eval_matrix.yaml) so bump validation is documented as nightly rather than per-PR.

This keeps the fail-closed regression coverage for bumps while freeing the shared runner from a full matrix run on every bump/stack PR.

Acceptance criteria

  • Stack-touching PRs (incl. Dependabot bumps) no longer trigger a GPU eval on raise.
  • Bump regressions are still caught by the nightly run (and on-demand dispatch).
  • References/docs to per-PR bump validation are updated.

Refs #303.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions