fix(presets): enforce that the served repo is a variant of the requested base

[vaibhav8a]

Closes #4197.

The gap

A preset declaring base: X was verified against two things a substitution preserves:

• the service advertises the requested name (verify.py:99-100)
• the report echoes the requested base (verify.py:142-146)

Neither looks at the repo the agent actually served. So vllm serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name Qwen/Qwen3.8-27B answers a request for Qwen3.8 with a different model generation, verifies clean, and is reported successful — exactly as reported.

The rule, and why this shape

base is documented as "the base model for which the agent may select a compatible variant", so verification now checks compatibility: the served repo must be the base, or the base plus a suffix at a separator boundary (-, _, .).

Two deliberate choices, both of which cut the other way from the obvious implementation:

The owner is ignored. Comparing full org/name would have been the simpler rule and it would be wrong — a quantisation is routinely published by someone other than the model's author. This repository's own fixtures already assume that, pairing a Qwen/Qwen3.5-27B base with a community/Qwen3.5-27B-GPTQ-Int4 repo. An owner check would reject the ordinary case and cost base the freedom it exists to grant.

The boundary is required. Qwen3.5-27B accepts Qwen3.5-27B-AWQ but not Qwen3.5-27Bx, so a bare startswith cannot let an unrelated longer name through.

Comparison is case-insensitive, since repo references are.

served	base	verdict
community/Qwen3.5-27B-GPTQ-Int4	Qwen/Qwen3.5-27B	accepted — third-party quantisation
Qwen/Qwen3.5-27B-AWQ	Qwen/Qwen3.5-27B	accepted
qwen/qwen3.5-27b-gptq-int4	Qwen/Qwen3.5-27B	accepted — case
Qwen/Qwen3.8-27B-GPTQ-Int4	Qwen/Qwen3.5-27B	rejected — the reported bug
meta-llama/Llama-3-8B	Qwen/Qwen3.5-27B	rejected
Qwen/Qwen3.5-27Bx	Qwen/Qwen3.5-27B	rejected

Where I would welcome a steer: this is a naming-convention rule, so it is a heuristic about how HuggingFace repos are named rather than a fact about the models. It is deliberately conservative — it will reject a legitimate variant that renames rather than suffixes (say a -v2 republished under a different stem). If you would rather that case be allowed, the rule is one function (_is_variant_of) and easy to loosen. Erring toward rejection seemed right for a check whose absence let a silent substitution pass as verified.

The error names both values, matching the style of the dataset and workload checks nearby — "not a variant" alone is not actionable.

Tests

Eight, parametrised into two groups:

• four substitutions that must be rejected, including the reported one and the -27Bx near-miss. All four fail on master.
• four genuine variants that must still pass, including the third-party quantisation and a lowercased reference. These pass both before and after — they exist to catch an over-correction that would break base rather than fix it.

Verified: pytest src/tests/_internal/cli/ — 484 passed, 18 skipped. ruff check and ruff format --check clean on the touched packages using the pinned ruff==0.12.7 from pyproject.toml.