Repository navigation
[DSV4] perf: Make FP8 quant output tensor contiguous - #27926
Conversation
…bit_v2 Add support for row-major FP8 quantization with UE8M0 scaling in the per_token_group_quant_8bit_v2 CUDA kernel. When SCALE_UE8M0 is enabled with row-major layout, the kernel now directly rounds scales to power-of-two format, eliminating the need for a separate deep_gemm.ceil_to_ue8m0 call. Split from sgl-project#26582 for wheel release purposes.
…python) Use the new row-major UE8M0 scale support in sgl-kernel to fuse the separate deep_gemm.ceil_to_ue8m0 call into sglang_per_token_group_quant_fp8 with scale_ue8m0=True for the wo_a path. Depends on the kernel change in the sgl-kernel PR. Split from sgl-project#26582 for wheel release purposes.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for row-major float32 scale layouts with power-of-2 values (scale_ue8m0) and outer-major scale layouts (scale_outer_major) in the per-token group quantization FP8 kernel, updating both the CUDA kernel and Python APIs, and integrating them into the DeepSeek-V4 model. Feedback on the changes suggests wrapping the deep_gemm import in a try-except block and adding a guard in the new tests to prevent ImportError and test failures on non-Hopper platforms where deep_gemm is not installed.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
On B300 with eval commands from #29036: == gpqa ==
198 examples (single-shot) | 1796.4s | 1146 tok/s | 2.1M tokens
* score = 89.39%
stop_rate = 100.00%
truncated_rate = 0.00%
error_rate = 0.00% |
|
Updated Pareto 1k1k 8xB300 DSV4-pro low latency config no MTP: series conc TPS/user (sp) out TPS/GPU (sp) total TPS/GPU (sp) TTFT ms (sp)
----------------------------------------------------------------------------------------------------------------------
base 4 85.32 ( 1.00x) 41.72 ( 1.00x) 83.98 ( 1.00x) 87.86 ( 1.00x)
base 8 77.83 ( 1.00x) 75.20 ( 1.00x) 150.85 ( 1.00x) 89.23 ( 1.00x)
base 16 65.91 ( 1.00x) 128.43 ( 1.00x) 254.98 ( 1.00x) 92.10 ( 1.00x)
base 32 52.37 ( 1.00x) 202.78 ( 1.00x) 406.65 ( 1.00x) 97.33 ( 1.00x)
base 64 39.37 ( 1.00x) 307.24 ( 1.00x) 615.31 ( 1.00x) 103.96 ( 1.00x)
opt 4 89.95 ( 1.05x) 43.95 ( 1.05x) 88.48 ( 1.05x) 85.59 ( 1.03x)
opt 8 82.34 ( 1.06x) 79.50 ( 1.06x) 159.47 ( 1.06x) 86.41 ( 1.03x)
opt 16 69.19 ( 1.05x) 134.75 ( 1.05x) 267.51 ( 1.05x) 89.23 ( 1.03x)
opt 32 54.50 ( 1.04x) 211.11 ( 1.04x) 423.36 ( 1.04x) 94.28 ( 1.03x)
opt 64 40.45 ( 1.03x) 313.63 ( 1.02x) 628.10 ( 1.02x) 101.19 ( 1.03x) |
|
/rerun-failed-ci |
|
/rerun-failed-ci |
1 similar comment
|
/rerun-failed-ci |
|
/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_h200.py test/registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py test/registered/disaggregation/test_disaggregation_dsv4.py |
|
/rerun-test test/registered/gb300/test_deepseek_v4_pro_fp4.py |
|
Results for 🚀 🚀 |
|
Results for ⛔ |
|
/rerun-test test/registered/jit/deepseek_v4/test_fp8_wo_a.py |
|
Results for 🚀 |
…ensor contiguous (sgl-project#27926) (sgl-project#30449) Co-authored-by: Kaixi <matteochen3@gmail.com> Co-authored-by: liqichao <liqichao@baidu.com> Co-authored-by: chenbong <bhchen@stu.xmu.edu.cn>
Co-authored-by: liqichao <liqichao@baidu.com> Co-authored-by: chenbong <bhchen@stu.xmu.edu.cn>
Motivation
This PR is an alternative way to achieve #27680.
Deep gemm FP8 einsum silently falls back to an unoptimized path if the input tensor is not contiguous. We can make the
per_token_group_quant_8bit_v2to emit the scale layout needed by deep gemm.This also implicitly fixes #29038. The previous DSV4
wo_aFP8 DeepGEMM path flattenedo: [T, G, D]into[T * G, D]and then viewed the scales back as[T, G, -1]. On the SM100 FP8 einsum path, scales are consumed as per-group slabs, so flattening acrossT * Gcan associate a token's FP8 codes with another group's scale. This PR quantizesoas[T, G, D]and emits the scale tensor in DeepGEMM's expected outer/group-major layout, preserving thescale[t, g]too[t, g, :]association.Modifications
Accuracy Tests
Speed Tests and Profiling
Node: 8XB300 + DSV4-Pro
Pareto for 1k1k, no MTP:
Checklist
Review and Merge Process
/tag-and-rerun-ci,/tag-run-ci-label,/rerun-failed-ciCI States
Latest PR Test (Base): ❌ Run #28783296006
Latest PR Test (Extra): ❌ Run #28783295767