[metamergebot]

Summary

DeepEP v2 rejects 1x32 MXFP8 experts and applies routing weights before model-specific expert finalization. Support group-32 activation scales through dispatch and DeepGEMM, with unweighted route outputs and an opt-in FP32 SiLU policy for registered model architectures.

Motivation

Activation scale groups must be tracked independently of weight scale groups. Models that transform each expert output also need those outputs before router weighting and reduction.

Modifications

Extend the existing SiLU/post-quant kernels to group 32, carry scale metadata through both dispatch layouts, and expose explicit route layouts for finalization. Preserve upstream's BF16 dispatch, FlashInfer adapter, and existing contiguous MXFP8 behavior. The SM120 contiguous producer records its group-128 activation scales independently of MXFP8 weight groups. The scatter expert-range arguments are already upstream and are not duplicated.

Import the quantizer from its defining module in DeepGEMM and the CUDA FP8/INT8 wrappers, so import order cannot replace a callable with the same-named submodule.

Accuracy Tests

GB300 kernel checks compare packed UE8M0 scales and masked/contiguous FP8 codes, including empty experts and partial slabs. Route-output checks cover deferred weighting, nonlocal slots, and empty inputs. Existing BF16/FP8 masked-versus-contiguous DeepGEMM checks also pass.

The SM120 regression exercises its production quantizer and scatter path with platform selection forced on GB300. Before the fix, the resulting scales triggered MXFP8 activation scale mismatch: block_size=32, K=512, scale_width=1, expected 4. With explicit group-128 metadata, activation and weight recipes remain independent.

Test plan

Ran with a Python 3.12 test environment selected through .venv:

PYTHONPATH=python SGLANG_IS_IN_CI=1 uv run --no-project --python .venv/bin/python python -m pytest -q \
  test/registered/unit/layers/moe/test_hpc_ops_runner_guard.py \
  test/registered/unit/layers/moe/test_moe_runner_extensions.py \
  test/registered/unit/layers/moe/test_deepep_v2_buffer_lifecycle.py \
  test/registered/unit/layers/moe/test_deepep_v2_wire_dtype.py \
  test/registered/unit/layers/moe/test_deepep_v2_masked_slab.py \
  test/registered/kernels/ops/quantization/test_per_token_group_quant.py::test_ue8m0_group_sizes \
  test/registered/kernels/ops/quantization/test_per_token_group_quant.py::test_fp32_silu_post_quant \
  test/registered/kernels/ops/moe/test_minimax_quant_scatter.py \
  --disable-warnings --maxfail=2

Result: 114 passed, 12 subtests passed. The SM120 regression was also run alone before the production fix and failed with the scale-width assertion above.

Also passed:

PYTHONPATH=python uv run --no-project --python .venv/bin/python python -m pre_commit run --files \
  python/sglang/srt/layers/moe/moe_runner/deep_gemm_sm120.py \
  test/registered/kernels/ops/moe/test_minimax_quant_scatter.py
PYTHONPATH=python uv run --no-project --python .venv/bin/python python -m pre_commit run --from-ref main --to-ref HEAD
git diff --cached --check

CI on 40b0ffa328 passed lint, all JIT checks, all 11 base-A jobs, all 20 base-B jobs (including B200), all 17 base-C jobs, and pr-test-finish. Base run 35308289435 completed successfully; no base-C exception was needed.

Tests use PyTorch 2.11/CUDA 13 on GB300. Native SM120 GEMM execution, native DeepEP communication, and distributed model accuracy were not tested locally.

Speed Tests and Profiling

No serving benchmark was run.

Checklist

• Formatting and focused tests pass.
• Added scale-layout and route-finalization coverage to existing test files.
• Documented the route-layout contract in the public types.
• Distributed accuracy and serving benchmarks.

Original commits

• 185637e43120fe1b0b4cd55e037df0de44a1873d

---

CI States

Latest PR Test (Base): ✅ Run #35308289435
Latest PR Test (Extra): ❌ Run #35308289263
Latest PR Test (AMD ROCm 10): ❌ Run #35308289469

[metamergebot]

/rerun-failed-ci

[merrymercy]

/rerun-test test/registered/ep/test_deepep_small_extra.py test/registered/ep/test_routed_experts_dp_readback.py::TestRoutedExpertsReadbackDeepEPv2.test_dp2_readback

[github-actions]

Results for /rerun-test test/registered/ep/test_deepep_small_extra.py test/registered/ep/test_routed_experts_dp_readback.py::TestRoutedExpertsReadbackDeepEPv2.test_dp2_readback:

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/ep/test_deepep_small_extra.py
cd test/ && python3 registered/ep/test_routed_experts_dp_readback.py TestRoutedExpertsReadbackDeepEPv2.test_dp2_readback