[JoeLee314]

[DeepSeek-V4] Fuse UE8M0 scale rounding into FP8 group quantization

Motivation

This PR optimizes the DeepSeek-V4 wo_a FP8 path by removing the separate deep_gemm.ceil_to_ue8m0 call after activation quantization.

The original path launches quantization first, then rounds the generated FP8 scales to UE8M0/power-of-two format:

sglang_per_token_group_quant_fp8 -> deep_gemm.ceil_to_ue8m0 -> deep_gemm.fp8_einsum

The optimized path emits the rounded power-of-two scale directly from the per-token group quantization kernel:

sglang_per_token_group_quant_fp8(scale_ue8m0=True) -> deep_gemm.fp8_einsum

Performance Test

DeepSeek-V4-Flash, TP=4, 32K input / 1K output, 128 requests, max concurrency 4.

Output token throughput increased by 7.8%.

The configuration of server is from cookbook but without MTP: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4

# launch server
  sglang serve \                                                                                                                                                                         
    --trust-remote-code \                                                                                                                                                                
    --model-path /xxx/DeepSeek-V4-Flash \                                                                                                                              
    --tp 4 \                                                                                                                                                                             
    --moe-runner-backend flashinfer_mxfp4 \                                                                                                                                              
    --chunked-prefill-size 4096 \
    --disable-flashinfer-autotune \
    --swa-full-tokens-ratio 0.1 \
    --host 0.0.0.0 \
    --port 30000

  # bench_serving：
  python3 -m sglang.bench_serving \
    --backend sglang \
    --host 127.0.0.1 \
    --port 30000 \
    --model /xxx/DeepSeek-V4-Flash \
    --dataset-name random-ids \
    --num-prompts 128 \
    --random-input-len 32000 \
    --random-output-len 1000 \
    --max-concurrency 4 \
    --request-rate inf \
    --tokenize-prompt

Accuracy Tests

GSM8K score: 0.928

Modifications

sgl-kernel/csrc/gemm/per_token_group_quant_8bit_v2.cu

• Support row-major scale_ue8m0=True output.
• Keep exact raw scale for quantization.
• Store rounded power-of-two scale values for DeepGEMM consumption.
• Preserve the existing column-major packed UE8M0 behavior.

python/sglang/srt/models/deepseek_v4.py

• Use sglang_per_token_group_quant_fp8(..., scale_ue8m0=True) in the DeepSeek-V4 wo_a FP8 path.
• Remove the separate deep_gemm.ceil_to_ue8m0 call from this path.

Checklist

• Fused UE8M0 scale rounding into FP8 group quantization.
• Preserved exact quantization behavior for row-major scale output.
• Verified kernel-level scale equivalence.
• Verified GSM8Kend-to-end accuracy.
• Verified bench_serving performance improvement on DeepSeek-V4-Flash.

---

CI States

Latest PR Test (Base): ❌ Run #26643741620
Latest PR Test (Extra): ❌ Run #26643741155

[gemini-code-assist]

Code Review

This pull request adds support for row-major FP8 quantization with UE8M0 scaling in the per_token_group_quant_8bit_v2 CUDA kernel and integrates it into DeepSeek-V4, replacing the two-step quantization and ceiling process with an integrated kernel-level approach. Feedback on the changes suggests adding an explicit assertion in fp8_kernel.py to ensure that scale_tma_aligned is true when column_major_scales is enabled under the scale_ue8m0 path, preventing silent fallbacks to row-major layouts.

[b8zhong]

/tag-and-rerun-ci

[JoeLee314]

For the purpose of wheel release, this PR was split to #26765 and #26766