Skip to content

[Fix] Reduce routed-expert trace memory with uint16 storage - #2046

Open
matrix72c wants to merge 2 commits into
InternLM:mainfrom
matrix72c:fix/compact-routed-experts-uint16
Open

[Fix] Reduce routed-expert trace memory with uint16 storage#2046
matrix72c wants to merge 2 commits into
InternLM:mainfrom
matrix72c:fix/compact-routed-experts-uint16

Conversation

@matrix72c

@matrix72c matrix72c commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Reduce learner memory used by rollout routed-expert traces.

  • Keep routed-expert IDs resident as torch.uint16 when n_routed_experts <= 65536.
  • Fall back to the existing torch.long storage path for models with more experts.
  • Convert only the current layer's route slice to torch.long on the target device before router replay.

Problem

LMDeploy, the SessionServer, and the Ray object store already carry routed-expert IDs as numpy.uint16. TrainingWorker._add_rollout_routed_experts() converted the real routes to torch.long, while padding tensors also defaulted to torch.long.

This expanded every expert ID from 2 bytes to 8 bytes. A learner that prepares many long rollout samples therefore keeps a four-times-larger route payload in each training worker. Offloading changes whether that payload consumes host or device memory, but does not reduce its resident size.

Implementation

Compact learner storage

  • Select torch.uint16 storage for models with at most 65536 routed experts.
  • Use the selected dtype for real routes, per-sample padding, and full padding batches so that torch.cat cannot promote the result back to torch.long.
  • Preserve the normal numpy.uint16 path without an additional value scan.
  • Validate wider integer inputs before narrowing and reject values outside 0..65535 instead of silently wrapping.
  • Preserve the original torch.long behavior when n_routed_experts > 65536.

Explicit storage/compute boundary

The MoE decoder still selects only the current layer's [local_seq_len, topk] route slice. It then converts that slice to torch.long on hidden_states.device before entering the router. The existing contiguous copy for cross-device offload is preserved.

Keeping this conversion outside the offload condition also makes the non-offload path safe: a full uint16 route tensor may be moved with the sequence context, but unsigned indices never reach gather directly.

Impact

  • Route storage is reduced by 75% for models with up to 65536 routed experts.
  • The only narrowing-to-index conversion is on the layer-local slice consumed by the router.
  • No configuration or public API is added.
  • LMDeploy, SessionServer, Ray object-store, batching, and trajectory-logging behavior are unchanged.
  • Models with more than 65536 routed experts retain the previous torch.long path.

This is the narrow-dtype mitigation discussed in #2025. It does not change batch materialization or sequence-parallel transfer ordering, so those broader improvements remain separate work.

Tests

  • pytest tests/rl/test_routed_experts_dtype.py tests/rl/test_prepare_train_data.py -q: 20 passed.
  • Covered dtype selection at 256, 257, 65536, and 65537 experts.
  • Covered real Ray-ref input, padding-only input, and mixed concatenation.
  • Covered exact preservation of IDs 0, 255, 256, and 65535.
  • Covered rejection of negative and overflowing wider integer inputs.
  • Covered offload and non-offload decoder paths, including non-contiguous layer slices and use as gather indices.
  • Ruff and git diff --check: passed.

Related to #2025.

@matrix72c
matrix72c force-pushed the fix/compact-routed-experts-uint16 branch from e817562 to 7d6a8d7 Compare September 7, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant