Enable use_sdpa_with_kv_cache in the qwen3_5 example config - #22390
Conversation
The flag being off costs roughly 2x decode on XNNPACK (8.20 -> 16.64 tok/s on Qwen3.5-2B 8da4w, Mac arm64) while producing the same tokens; the lfm2 config, the same hybrid attention/conv architecture, already sets it True. Discussed in pytorch#22045.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22390
Note: Links to docs will display an error until the docs builds have been completed. ❌ 3 New Failures, 2 Unrelated Failures, 1 Unclassified FailureAs of commit 43a8387 with merge base f5102b8 ( NEW FAILURES - The following jobs have failed:
UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
BROKEN TRUNK - The following job failed but was present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
Fixes #22045.
use_sdpa_with_kv_cache: Falsein this config costs roughly 2x decode. @JakeStevens confirmed in the issue there is no known reason to keep it off, and the lfm2 config — the same hybrid attention/conv architecture — already sets itTrue.Measured on Qwen3.5-2B, XNNPACK, 8da4w + 8-bit embedding, the same checkpoint exported twice with only this line different (Mac arm64, token-by-token prefill then greedy decode, same prompt, 32 new tokens):
use_sdpa_with_kv_cache: Falseuse_sdpa_with_kv_cache: TrueThat it is the same model rather than a faster different one: three of four prompts decode token-identically (the fourth diverges inside
<think>, a near-tie either way), first-step logits corr 0.995 with the same top-1, and at 338 tokens of context corr 0.990 with the same top-1 and top-2. File size is unchanged (1483 MB both).cc @mergennachin @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng