Next iteration: move the stack to Python 3.14 and the latest PyTorch

[amd-vivekag]

Follow-up from #380. The CI base is now the newest published combination available on the classic ROCm layout:

rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0

Newer Python and PyTorch exist, but only behind two independent gates. This issue tracks clearing both.

Gate 1 — the images only ship newer Python/PyTorch on the wheel line

Surveyed from Docker Hub (477 tags):

ROCm line	Layout	Ubuntu	Python	PyTorch
7.14	wheel (TheRock)	24.04, 26.04	3.11–3.14	2.10, 2.11, 2.12
7.2.4 (current)	classic	22.04, 24.04	3.10, 3.12	… 2.10.0

There is no py3.13/3.14 or torch 2.11+ on any classic-layout image, so this is blocked on #381 (layout-agnostic ROCm discovery), which is what makes the 7.14 line usable. Once #381 lands, ROCm 7.14 + py3.14 + torch 2.12 becomes reachable in one step.

Note the trap: ROCm 7.2 (older than our 7.2.4) does publish py3.13. Taking it would mean giving up three ROCm patch releases for one Python minor — the wrong trade for a ROCm debugging tool. Don't "solve" this by moving backwards on ROCm.

Gate 2 — our own packaging caps Python at 3.12

Independent of any image:

requires-python = ">=3.10"
Programming Language :: Python :: 3.10 / 3.11 / 3.12   # stops at 3.12

and cpu-tests.yml runs a 3.10/3.11/3.12 matrix. Running the GPU gate on 3.13/3.14 would be testing a Python we neither declare nor test anywhere else. This half is not blocked on #381 and can be done independently, which is probably the right order — it de-risks the image bump.

Scope

Python 3.13/3.14 support (do first, independent of #381)

• Add the 3.13 (and 3.14 when appropriate) classifiers to pyproject.toml.
• Extend the cpu-tests.yml matrix to cover the new versions. Note the KB#13 precedent: the matrix is gated by a non-matrix aggregator job for branch protection, so adding legs must not break the required check.
• Check dependency support at the new versions. All extras are lower-bounded (>=) with no upper caps, so this is likely a no-op, but numpy/pandas/matplotlib are the ones to verify — and per KB#14, pin to the newest release whose requires_python still admits our floor.
• Decide whether the floor moves too, or whether we keep supporting 3.10.

Stack move (after #381)

• Flip the base to ROCm 7.14 + the newest Ubuntu/Python/PyTorch that line offers.
• Expect the env-knob audit to need updating: audit_env_knobs.py --strict fails on ROCm env vars absent from ENV_KNOB_REGISTRY, and 7.2.4 → 7.14 is a large jump. This did not fire on the 7.2 → 7.2.4 patch bump, but it very likely will here.
• Re-bless baselines via refresh-baselines.yml — torch 2.10 → 2.12 will move numerics/perf.
• Confirm Move CI + default dev image to the newest classic-layout stack (ROCm 7.2.4 / Ubuntu 24.04 / py3.12 / PyTorch 2.10) #380's classic-layout guard is removed or relaxed as part of ROCm discovery hardcodes /opt/rocm, so the probe and CI go blind on TheRock/wheel-based images #381, not as a way to make this bump pass.

Sequencing

#383 (Python support) → #381 (layout-agnostic discovery) → flip to 7.14 + py3.14 + torch 2.12

Doing the Python work first means the eventual flip changes ROCm and layout only, keeping one variable at a time — the same reasoning that led #380 to move Ubuntu/Python ahead of the ROCm jump.

[amd-vivekag]

Flip prerequisites are done; the flip itself is blocked on one open question

While finishing #381 I took the 7.14 flip most of the way and then backed it out
deliberately. Everything below was measured against real images, not
reasoned about, so the next attempt should not need to re-derive it.

The three prerequisites this issue predicted are now landed on
feat/rocm-layout-agnostic-stack (PR for #381/#382), so a future flip is a
Dockerfile/.env.ci change plus the one open decision.

The target image, confirmed

rocm/pytorch:rocm7.14_ubuntu26.04_py3.14_pytorch_release_2.12.0
@ sha256:8785d67393d9f452cbae2362003f2e211289f9c12163a8c168496724664a0fda

Surveyed Docker Hub: 7.14 is still the newest ROCm line, and this is the newest
tag on every axis at once. The only other 7.14 shapes are Ubuntu 24.04 with
py3.11–3.13 and torch 2.10/2.11. Matches this issue's table.

What is no longer a risk

Discovery works. This issue and the handover both flagged that flip
validation depends on _rocm_sdk_core being importable from
/opt/venv/bin/python, "very likely true but unproven". It is proven: the image
sets no ROCM_PATH, no ROCM_HOME and no /opt/rocm on PATH, and the
resolver lands on layout=wheel, source=import:_rocm_sdk_core, version
7.14.0. The full schema-1.16 capture works end-to-end through the real
aorta env probe CLI installed in the image: therock populated
(the_rock_commit, github_run_id, 10 submodule pins) and hipBLASLt/rocBLAS
carrying lib_hash, kernel_db_revision and a full 40-char upstream_commit.
Note that is better evidence than classic gives — the classic header only
exposes a 10-char tweak.

The env-knob audit gap is one knob, not a batch. This issue said
audit_env_knobs.py --strict "very likely will" need registry additions. It
does: exactly TENSILE_ADAPTIVE_GEMM_NTAB_ALGO. Already registered on the
branch, so --strict now exits 0 on both layouts. Coverage actually improves on
7.14 — covered 61 → 76, not_present 15 → 1.

Python 3.14 is fine. The CPU suite runs clean under py3.14 inside the
flipped image: 3305 passed, and the only 17 failures are bpftrace not being
installed in that image. This also flushed out two host-dependent tests, now
fixed on the branch.

The reference-build checks are not lost. They key off hipBLASLt 1.4.70002 /
rocBLAS 5.0.70002, and the current 7.2.4 base ships 1.2.70204, so the audit
already reports reference checks: skipped (different library build) before any
flip. Nothing regresses here.

One real breakage found and already fixed on the branch.
sanitizers-nightly.yml derived the ROCm LLVM bindir in shell as
${ROCM_PATH:-${ROCM_HOME:-/opt/rocm}}/lib/llvm/bin, which validates nothing —
on a wheel image both vars are unset and /opt/rocm does not exist, so it
expanded to a nonexistent directory and left clang-offload-bundler
unreachable. Now derived from the resolver (RocmRoots.llvm_bin_dir), and
prepare_gemm_isa.py falls back to it directly. Byte-identical on classic.

The open blocker

prepare_gemm_isa.py needs one exactly-named Tensile bundle per transpose
layout. On 7.14 that name does not exist — the bundles are split per CU count
and solution-ID range, so _library_for() raises SystemExit and the sanitizer
GEMM fixtures (daily-consan-gemm, daily-waitcheck-gemm,
consan-code-objects) cannot be built. Measured for Ailk_Bjlk / gfx950:

Bundle	.co on disk	Unbundled gfx950 object
7.2.4 ..._Cijk_Dijk_gfx950.co (what the script expects)	3.5 MB	183 MiB
7.14 ..._Cijk_Dijk_CU256_ID75a0_gfx950.co	4.0 MB	156 MiB
7.14 ..._Cijk_Dijk_ID75a0_gfx950.co	3.0 MB	168 MiB
7.14 ..._Cijk_Dijk_ID75a3-75a2_gfx950.co	0.6 MB	14 MiB

Choosing between them changes what ConSan/waitcheck actually analyse, so it
should not be guessed. CU256 is the CU-specialised variant for the 256-CU
MI350X the gate runs on, which argues for it on representativeness grounds;
ID75a0 is closest to the old artifact by size. Deciding needs a look at which
variant hipBLASLt actually loads for these shapes on gfx950 — a research task,
not a hand-off.

Related: the script's docstring claimed that bundle "unbundles to ~16 MB / ~490
kernels". Measured on the current base it is 191,935,808 bytes (~183 MiB), so
that figure was stale by more than 10x; corrected on the branch to a
version-attributed measurement rather than a new fixed number, since it is
clearly a per-release property.

Suggested remaining acceptance criteria

• Decide which 7.14 Tensile bundle variant the sanitizer GEMM fixtures
  should use, and teach prepare_gemm_isa.py to select it deterministically
  on both layouts.
• Flip docker/Dockerfile.ci-gpu (+ docker/Dockerfile.rocm-latest,
  docker/.env.ci) to the digest above.
• Re-bless baselines via refresh-baselines.yml — torch 2.10 → 2.12 will
  move numerics/perf, and the changed GEMM code objects will move sanitizer
  verdict baselines.
• Watch the first nightly on the new stack.

[amd-vivekag]

The Tensile bundle question is resolved, and it was not a judgement call

The first remaining acceptance criterion above — decide which 7.14 bundle variant
the sanitizer GEMM fixtures should use — is answered in #402 (stacked on #387).
It did not need a preference; the split turned out to be self-describing, so
recording the mechanism here so nobody re-derives it.

The filename tokens are the predicates

I decoded
lib/hipblaslt/library/gfx950/TensileLibrary_lazy_gfx950.dat.zlib out of
rocm/pytorch:rocm7.14_ubuntu26.04_py3.14_pytorch_release_2.12.0
@ sha256:8785d673…. It is a Tensile Hardware library — five rows resolved
first-match — and for the heavy f32 family the mapping is:

row	predicate	library
0	PciChipId 0x75a8	—
1	PciChipId ∈ {0x75a2, 0x75a3}	ID75a3-75a2
2	CUCount 256 ∧ PciChipId 0x75a0	CU256_ID75a0
3	CUCount 128 ∧ PciChipId 0x75a0	—
4	PciChipId 0x75a0	ID75a0

So _CU<n> is CUCount == n, and _ID<hex>[-<hex>] is a set of PCI chip
ids — not the opaque "solution-ID range" my earlier comment assumed. 0x75a0
is 30112, the MI350 series. Identical across all four transpose layouts.

Correcting my own earlier framing: I wrote that deciding "needs a look at
which variant hipBLASLt actually loads." That question has no answer, because
hipBLASLt loads all of them — TensileLibrary_*_gfx950 is a glob, every
variant is registered into the lazy master, and ChipIdRegistry /
isFallbackForHW resolve per dispatch. There is only ever a per-device answer.

The gate's hardware picks row 2

rocprofv3 agent info from a real run on the gate reports Name gfx950,
Device_Id 30112 (0x75a0), Cu_Count 256 — row 2 exactly, so
CU256_ID75a0. My earlier note reasoned toward CU256 on representativeness
grounds and that lands in the same place, but it is now derived from the shipped
index rather than argued for.

Note the reasoning that didn't survive: ID75a0 was the candidate "closest to
the old artifact by size". Size is not evidence of anything here — the predicates
are.

What #402 does with it

Rather than hardcoding the name, it parses the tokens and picks the narrowest
variant admitting a target device (defaulting to the gate's part, overridable via
--chip-id / --cu-count), failing closed when nothing serves it. The decoded
table is committed as a test fixture and a drift check requires the selector to
reproduce it, so this stays honest across the next ROCm bump.

Measured on both real images: byte-identical extracted objects on the pinned
7.2.4 base (so nothing moves before the flip), and a full end-to-end extraction
on 7.14 producing real AMD HSA code objects — the three GEMM fixtures
(daily-consan-gemm, daily-waitcheck-gemm, consan-code-objects) are
buildable on the target stack for the first time.

Remaining on this issue

With #387 (discovery, env-knob, py3.13/3.14) and #402 (bundle selection), what is
left is the mechanical part plus one observation window:

• Flip docker/Dockerfile.ci-gpu, docker/Dockerfile.rocm-latest and
  docker/.env.ci to sha256:8785d673….
• Re-bless baselines via refresh-baselines.yml — torch 2.10 → 2.12 moves
  numerics/perf, and the changed GEMM code objects move sanitizer verdicts.
• Watch the first nightly on the new stack.

One thing the flip should expect: the extracted objects change size
(~183 MiB on 7.2.4 → 156–168 MiB for CU256_ID75a0 on 7.14), so every recorded
fixture digest moves. That is the baseline re-bless, not a regression.