[DeepSeek V3] Defer moe finalize and fused it with main stream add by kpham-sgl · Pull Request #27720 · sgl-project/sglang

kpham-sgl · 2026-06-09T21:46:07Z

Motivation

Optimizing Kimi K2.5 NVFP4
Note: this PR does not change behaviour during prefill phase yet (still 2 individual kernel launches). More work needed to speedup there

Modifications

Swap the execution order. Main stream now execute routed experts, alt stream execute shared experts
On main stream, defer finalized (do_finalize = False) and fuse finalize with add kernel

Accuracy Tests

Speed Tests and Profiling

Resulted in -2us per layer time. 1-2% TPOT speedup for 80k, ISL 220, with Kimi K2.5 NVFP4 on TP=8

Before

After

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review and Merge Process

Ping Merge Oncalls to start the process. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #27448273418
Latest PR Test (Extra): ❌ Run #27448273296

gemini-code-assist · 2026-06-09T21:46:11Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

gemini-code-assist · 2026-06-10T22:36:08Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

kpham-sgl · 2026-06-10T22:36:31Z

/tag-and-rerun-ci

kpham-sgl · 2026-06-10T22:54:08Z

/rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py
test/registered/models_e2e/test_deepseek_v32_fp4_mtp_dp.py

github-actions · 2026-06-10T22:54:33Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v3_fp4.py
cd test/ && python3 registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py
cd test/ && python3 registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py

Qiaolin-Yu · 2026-06-10T22:59:27Z

    # Sglang Cache Dir
    SGLANG_CACHE_DIR = EnvStr(os.path.expanduser("~/.cache/sglang"))
    SGLANG_FLASHINFER_AUTOTUNE_CACHE = EnvBool(True)
+    SGLANG_ENABLE_MOE_DEFERRED_FINALIZE = EnvBool(False)


Could we enable it by default, if it's always better

Should always be better perf wise, small concern about correctness. See FIXME note

Add a tracking issue for flashinfer

kpham-sgl · 2026-06-10T23:57:50Z

I will merge this first and try out the new kMoEFinalizeARResidualRMSNorm in flashinfer (which should fuse all of moe finalize + add + AR + RMSNorm + fp4 quant)

The host launcher referenced tokenspeed:: for MAX_TOPK, BF16 and dispatchFinalize, but these are defined in namespace sglang. The stale tokenspeed:: prefix (leftover from the forked source) broke the JIT compile. Use sglang:: to match the definitions. Co-authored-by: Cursor <cursoragent@cursor.com>

kpham-sgl · 2026-06-11T01:27:57Z

/rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py
test/registered/models_e2e/test_deepseek_v32_fp4_mtp_dp.py

github-actions · 2026-06-11T01:28:20Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v3_fp4.py test/registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py test/registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v3_fp4.py
cd test/ && python3 registered/spec/eagle/test_deepseek_v3_fp4_mtp_small.py
cd test/ && python3 registered/models_e2e/test_deepseek_v32_fp4_mtp_tp.py

kpham-sgl · 2026-06-11T05:32:30Z

/rerun-failed-ci

…gl-project#27720) Co-authored-by: Qiaolin Yu <liin1211@outlook.com>

kpham-sgl added 2 commits June 9, 2026 03:24

init

9690f35

add an environ to be explicit

9a4a98f

github-actions Bot added deepseek jit-kernel labels Jun 9, 2026

kpham-sgl added 6 commits June 10, 2026 03:25

view expert_weights buffer as fp32

238d8a5

clean up

f7d6cb1

clean up server args

c4ea6ad

upd

0836168

more upd

0acb1dc

nit

4067ce8

kpham-sgl changed the title ~~Kp/defer moe finalize~~ [DeepSeek V3] Defer moe finalize and fused it with main stream add Jun 10, 2026

kpham-sgl marked this pull request as ready for review June 10, 2026 22:36

kpham-sgl requested review from BBuf, DarkSharpness, Edwardf0t1, Fridge003, HaiShaw, HydraQYH, Ying1123, celve, ch-wan, fzyzcjy, ispobock, merrymercy and yuan-luo as code owners June 10, 2026 22:36

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Jun 10, 2026

Qiaolin-Yu reviewed Jun 10, 2026

View reviewed changes

Qiaolin-Yu assigned b8zhong Jun 10, 2026

b8zhong approved these changes Jun 10, 2026

View reviewed changes

b8zhong reviewed Jun 10, 2026

View reviewed changes

Comment thread python/sglang/jit_kernel/csrc/moe/moe_finalize_fuse_shared.cu Outdated

nit

9ab4f59

kpham-sgl added 2 commits June 11, 2026 11:44

Merge branch 'main' into kp/defer-moe-finalize

14a3db9

Merge branch 'main' into kp/defer-moe-finalize

24cdbf3

amd-bot mentioned this pull request Jun 12, 2026

[CI Monitor] Daily Report - 2026-06-12 bingxche/sglang-ci-bot#102

Open

nit

103b6c6

kpham-sgl requested a review from Qiaolin-Yu June 12, 2026 03:06

Qiaolin-Yu approved these changes Jun 12, 2026

View reviewed changes

Qiaolin-Yu added the bypass-fastfail label Jun 12, 2026

Qiaolin-Yu and others added 2 commits June 12, 2026 11:44

Merge branch 'main' into kp/defer-moe-finalize

79908fd

Merge branch 'main' into kp/defer-moe-finalize

7ea1053

Fridge003 reviewed Jun 13, 2026

View reviewed changes

Comment thread python/sglang/srt/models/deepseek_v2.py

Fridge003 approved these changes Jun 13, 2026

View reviewed changes

Fridge003 merged commit d1a39b0 into main Jun 13, 2026
113 of 129 checks passed

Fridge003 deleted the kp/defer-moe-finalize branch June 13, 2026 02:00

This was referenced Jun 13, 2026

[CI Monitor] Daily Report - 2026-06-13 bingxche/sglang-ci-bot#103

Open

[CI Monitor] Daily Report - 2026-06-14 bingxche/sglang-ci-bot#104

Open

kpham-sgl mentioned this pull request Jun 24, 2026

[DeepSeek V3] Run routed experts on main stream in dual-stream MoE #29142

Merged

1 of 2 tasks

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[DeepSeek V3] Defer moe finalize and fused it with main stream add (s…

b10cfb8

…gl-project#27720) Co-authored-by: Qiaolin Yu <liin1211@outlook.com>

Conversation

kpham-sgl commented Jun 9, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

CI States

Uh oh!

gemini-code-assist Bot commented Jun 9, 2026

Uh oh!

gemini-code-assist Bot commented Jun 10, 2026

Uh oh!

kpham-sgl commented Jun 10, 2026

Uh oh!

kpham-sgl commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Qiaolin-Yu Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

kpham-sgl Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

kpham-sgl Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kpham-sgl commented Jun 10, 2026

Uh oh!

kpham-sgl commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kpham-sgl commented Jun 11, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kpham-sgl commented Jun 9, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

github-actions Bot commented Jun 11, 2026 •

edited

Loading