[DeepSeek-V4] Fuse UE8M0 scale rounding into FP8 group quantization by JoeLee314 · Pull Request #26766 · sgl-project/sglang

JoeLee314 · 2026-05-30T06:30:11Z

Motivation

This PR optimizes the DeepSeek-V4 wo_a FP8 path by removing the separate deep_gemm.ceil_to_ue8m0 call after activation quantization.

The original path launches quantization first, then rounds the generated FP8 scales to UE8M0/power-of-two format:

sglang_per_token_group_quant_fp8 -> deep_gemm.ceil_to_ue8m0 -> deep_gemm.fp8_einsum

The optimized path emits the rounded power-of-two scale directly from the per-token group quantization kernel:

sglang_per_token_group_quant_fp8(scale_ue8m0=True) -> deep_gemm.fp8_einsum

Performance Test

DeepSeek-V4-Flash, TP=4, 32K input / 1K output, 128 requests, max concurrency 4.

Output token throughput increased by 7.8%.

Accuracy Tests

GSM8K score: 0.928

Update: precision fix on Blackwell (B300/B200)

The initial fused kernel in this PR had a precision issue on Blackwell: in the
SCALE_UE8M0 && !IS_COLUMN_MAJOR branch, activations were quantized with the
exact FP32 scale while only the output scale was rounded to UE8M0 (power-of-2).
Since quantization and dequantization then used mismatched scales, EAGLE
speculative decoding accept_length dropped from 2.76 to 2.69 on B300/B200.

Thanks to @yhyang201 for catching this and providing the fix. The special branch
is removed and all UE8M0 paths now go through calculate_fp8_scales<SCALE_UE8M0=true>,
which computes the rounded (power-of-2) scale first and quantizes with it — matching
the official DeepSeek-V4 kernel.py act_quant(scale_fmt="ue8m0"). The row-major
UE8M0 test now asserts the scales are exact powers of 2 and the fp8 codes match the
ceil_to_ue8m0 reference.

CI States

Latest PR Test (Base): 🚫 Run #27789934317
Latest PR Test (Extra): ✅ Run #27791183010

gemini-code-assist · 2026-05-30T06:30:15Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

yhyang201 · 2026-06-15T12:49:59Z

Hi, could you ping me on slack? My name is Yuhao Yang.

Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com> Change-Id: I206d621f90d11b1e2ae42db8ae92b0bd782d5ced

yhyang201 · 2026-06-15T13:59:56Z

Reason for the second commit: #28307

The row-major + SCALE_UE8M0 path quantized with an exact FP32 scale and only rounded the output scale_inv to power-of-2. This caused a scale mismatch: FP8 values quantized with one scale, GEMM dequants with another, amplifying error ~14x (31% average relative error). On Blackwell (SM100) where DeepGEMM uses native UE8M0 hardware scaling, this mismatch fully exposed, dropping EAGLE accept_length from 2.76 (H200) to 2.69 (B300). Hopper (SM90) was less affected because it uses FP32 scale post-correction. Fix: use calculate_fp8_scales<SCALE_UE8M0=true> for all UE8M0 paths, which computes the rounded scale first then quantizes with it. This matches the official DeepSeek-V4 kernel.py act_quant(scale_fmt="ue8m0"). Results (DSV4-Flash EAGLE spec decode, GSM8K 200 prompts): H200 (SM90): 2.76 → 2.76 (unchanged) B300 (SM100): 2.69 → 2.77 (+0.08) Change-Id: Ic011b2dd16a6eb5d41955b2a705e623aaf03b6b4

Fridge003 · 2026-06-15T22:56:27Z

@yuhao318 Please add acc results on harder benchmarks like AIME25

Fridge003 · 2026-06-15T22:56:48Z

/rerun-failed-ci

yhyang201 · 2026-06-16T02:37:34Z

AIME25 Accuracy Report

Model: DeepSeek-V4-Flash
Hardware: 8x NVIDIA B300 SXM6 AC (SM100)
Config: TP=4, EAGLE (num_steps=3, topk=1, draft_tokens=4), flashinfer_mxfp4, chunked_prefill=4096, disable_flashinfer_autotune
Branch: PR #26766 (pr-26582-python, includes fused scale_ue8m0=True in quant kernel)
Eval tool: sgl-eval v0.0.1, AIME25, 30 problems x 16 repeats, temp=1.0, top_p=1.0, max_tokens=400000

Metric	Value
pass@1 (avg-of-16)	93.75% (+/- 2.95%, SEM 0.74%)
majority@16	96.67%
pass@16	100.00%
score	93.75%
no_answer rate	0.21%
output throughput	5742.6 tok/s
total completion tokens	15,439,750
total latency	2688.6s (44.8 min)

JoeLee314 · 2026-06-16T12:03:23Z

/rerun-failed-ci

Fridge003 · 2026-06-17T21:13:31Z

/rerun-failed-ci

Fridge003 · 2026-06-18T00:44:09Z

/rerun-failed-ci

Fridge003 · 2026-06-18T21:18:46Z

/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

github-actions · 2026-06-18T21:19:07Z

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py:

🚀 4-gpu-b200 (3 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py
cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

Fridge003 · 2026-06-18T21:21:23Z

/reurn-test test/registered/jit/test_per_token_group_quant_8bit_v2.py

Fridge003 · 2026-06-18T21:22:40Z

/rerun-test test/registered/4-gpu-models/test_deepseek_v3_cutedsl_4gpu.py test/registered/models_e2e/test_deepseek_v3_fp4.py

github-actions · 2026-06-18T21:23:06Z

Results for /rerun-test test/registered/4-gpu-models/test_deepseek_v3_cutedsl_4gpu.py test/registered/models_e2e/test_deepseek_v3_fp4.py:

🚀 4-gpu-gb300 (1 test): ✅ View workflow run

cd test/ && python3 registered/4-gpu-models/test_deepseek_v3_cutedsl_4gpu.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v3_fp4.py

main advanced 15+ commits since the last merge (sgl-project#28598 base), making the PR CONFLICTING and blocking pr-test. Resolve the one content conflict in python/sglang/srt/layers/quantization/fp8_kernel.py by taking main's version (byte-identical): main added the row-major UE8M0 scale path (sgl-project#26766 DSV4 UE8M0 fusion) as a superset of our column-major+tma-aligned path, and hardcodes //128 which is correct for UE8M0 (group fixed at 128). Everything else auto-merged from main.

…gl-project#26766) Co-authored-by: liqichao <liqichao@baidu.com> Co-authored-by: yhyang201 <yhyang201@gmail.com>

github-actions Bot added the deepseek label May 30, 2026

JoeLee314 mentioned this pull request May 30, 2026

[sgl-kernel] Support row-major UE8M0 scale in per_token_group_quant_fp8 #26765

Closed

JoeLee314 marked this pull request as ready for review May 30, 2026 07:16

JoeLee314 requested review from AniZpZ, BBuf, Edwardf0t1, FlamingoPg, HaiShaw, b8zhong and ch-wan as code owners May 30, 2026 07:16

JoeLee314 mentioned this pull request May 30, 2026

[DeepSeek-V4] Fuse UE8M0 scale rounding into FP8 group quantization #26582

Closed

5 tasks done

b8zhong mentioned this pull request Jun 10, 2026

[DSV4][Perf] Fuse FP8 quantization with UE8M0 scale layout #27818

Closed

Fridge003 self-assigned this Jun 10, 2026

Fridge003 added the high priority label Jun 10, 2026

mattteochen mentioned this pull request Jun 11, 2026

[DSV4] perf: Make FP8 quant output tensor contiguous #27926

Merged

5 tasks

JoeLee314 force-pushed the pr-26582-python branch from 2427af2 to 8c755e9 Compare June 15, 2026 02:42

JoeLee314 requested review from DarkSharpness, HydraQYH, celve and yuan-luo as code owners June 15, 2026 02:42

github-actions Bot added quant LLM Quantization jit-kernel labels Jun 15, 2026

yhyang201 mentioned this pull request Jun 15, 2026

[DONOTMERGE]Fix FP8 wo_a GEMM precision on Blackwell: quantize with UE8M0 scale #28307

Closed

Fuse UE8M0 scale rounding into FP8 group quantization

e008659

Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com> Change-Id: I206d621f90d11b1e2ae42db8ae92b0bd782d5ced

JoeLee314 force-pushed the pr-26582-python branch from 8c755e9 to e008659 Compare June 15, 2026 13:54

JoeLee314 force-pushed the pr-26582-python branch from bc52c46 to fdc448b Compare June 15, 2026 14:17

Fridge003 approved these changes Jun 15, 2026

View reviewed changes

Merge branch 'main' into pr-26582-python

b6e9cc9

Fridge003 added the run-ci-extra CI: also run the extra suite (requires run-ci) label Jun 15, 2026

yhyang201 mentioned this pull request Jun 16, 2026

Fix FP8 wo_a precision on Blackwell: quantize with UE8M0 scale #28418

Closed

JoeLee314 and others added 2 commits June 16, 2026 20:43

Merge branch 'main' into pr-26582-python

6fae04b

Merge branch 'main' into pr-26582-python

678d6b0

Merge branch 'main' into pr-26582-python

c4136e2

Fridge003 added 3 commits June 17, 2026 23:41

Merge branch 'main' into pr-26582-python

3435a95

Merge branch 'main' into pr-26582-python

388cbf6

upd acc len for some tests

050eb40

Fridge003 merged commit bea282c into sgl-project:main Jun 18, 2026
84 of 128 checks passed

Fridge003 added the release-highlight Candidate PR for release note highlight label Jun 18, 2026

Fridge003 mentioned this pull request Jun 19, 2026

[Bug] DeepSeek-V4 has slight accuracy issue #28414

Closed

5 tasks done

amd-bot mentioned this pull request Jun 21, 2026

[CI Monitor] Daily Report - 2026-06-21 bingxche/sglang-ci-bot#111

Open

shyeh25 mentioned this pull request Jul 3, 2026

[Bug] DeepSeekV4 Pro B300 TP4 conc4 MTP perf regression #30009

Closed

3 of 5 tasks

Conversation

JoeLee314 commented May 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Performance Test

Accuracy Tests

Update: precision fix on Blackwell (B300/B200)

CI States

Uh oh!

gemini-code-assist Bot commented May 30, 2026

Uh oh!

yhyang201 commented Jun 15, 2026

Uh oh!

yhyang201 commented Jun 15, 2026

Uh oh!

Fridge003 commented Jun 15, 2026

Uh oh!

Fridge003 commented Jun 15, 2026

Uh oh!

yhyang201 commented Jun 16, 2026

AIME25 Accuracy Report

Uh oh!

JoeLee314 commented Jun 16, 2026

Uh oh!

Fridge003 commented Jun 17, 2026

Uh oh!

Fridge003 commented Jun 18, 2026

Uh oh!

Fridge003 commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fridge003 commented Jun 18, 2026

Uh oh!

Fridge003 commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JoeLee314 commented May 30, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading