[Fix] Reduce routed-expert trace memory with uint16 storage - #2046
Open
matrix72c wants to merge 2 commits into
Open
[Fix] Reduce routed-expert trace memory with uint16 storage#2046matrix72c wants to merge 2 commits into
matrix72c wants to merge 2 commits into
Conversation
matrix72c
force-pushed
the
fix/compact-routed-experts-uint16
branch
from
September 7, 2026 03:52
e817562 to
7d6a8d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reduce learner memory used by rollout routed-expert traces.
torch.uint16whenn_routed_experts <= 65536.torch.longstorage path for models with more experts.torch.longon the target device before router replay.Problem
LMDeploy, the SessionServer, and the Ray object store already carry routed-expert IDs as
numpy.uint16.TrainingWorker._add_rollout_routed_experts()converted the real routes totorch.long, while padding tensors also defaulted totorch.long.This expanded every expert ID from 2 bytes to 8 bytes. A learner that prepares many long rollout samples therefore keeps a four-times-larger route payload in each training worker. Offloading changes whether that payload consumes host or device memory, but does not reduce its resident size.
Implementation
Compact learner storage
torch.uint16storage for models with at most 65536 routed experts.torch.catcannot promote the result back totorch.long.numpy.uint16path without an additional value scan.0..65535instead of silently wrapping.torch.longbehavior whenn_routed_experts > 65536.Explicit storage/compute boundary
The MoE decoder still selects only the current layer's
[local_seq_len, topk]route slice. It then converts that slice totorch.longonhidden_states.devicebefore entering the router. The existing contiguous copy for cross-device offload is preserved.Keeping this conversion outside the offload condition also makes the non-offload path safe: a full
uint16route tensor may be moved with the sequence context, but unsigned indices never reachgatherdirectly.Impact
torch.longpath.This is the narrow-dtype mitigation discussed in #2025. It does not change batch materialization or sequence-parallel transfer ordering, so those broader improvements remain separate work.
Tests
pytest tests/rl/test_routed_experts_dtype.py tests/rl/test_prepare_train_data.py -q: 20 passed.0,255,256, and65535.gatherindices.git diff --check: passed.Related to #2025.