[trevor-m]

Motivation

This PR enables NVFP4 quantization for MoE in DeepSeek-V4. NVFP4 should provide better performance compared to MXFP4. Use --moe-runner-backend flashinfer_trtllm_routed.

Needs #25702 to help with perf

Modifications

• Read new NVFP4 quant info from model config/hf_quant_config.json. The NVFP4 checkpoint uses FP8 quantization but with "moe_quant_algo": "NVFP4".
• Handle apply_routed_scaling_factor_on_output for hash topk
• Support swiglu_limit for flashinfer_trtllm_routed.

Accuracy Tests

sglang serve \
  --trust-remote-code \
  --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
  --tp 8 \
  --moe-runner-backend flashinfer_trtllm_routed \
  --speculative-algo EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --chunked-prefill-size 8192 \
  --disable-flashinfer-autotune \
  --swa-full-tokens-ratio 0.1 \
  --mem-fraction-static 0.90 \
  --host 0.0.0.0 \
  --port 30000
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.965
Invalid: 0.000
Latency: 53.335 s
Output throughput: 2149.225 token/s

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #27657320129
Latest PR Test (Extra): ✅ Run #27925999159

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[trevor-m]

GPQA Results

nemo-run_1/0 ------------------------------------------ gpqa -----------------------------------------                                                                                                                                                                                                                                                       
nemo-run_1/0 evaluation_mode   | num_entries | avg_tokens | gen_seconds | symbolic_correct | no_answer                                                                                                                                                                                                                                                       
nemo-run_1/0 pass@1[avg-of-16] | 198         | 23093      | 81683       | 86.43% ± 1.65%   | 1.89%                                                                                                                                                                                                                                                           
nemo-run_1/0 majority@16       | 198         | 23093      | 81683       | 90.91%           | 0.00%                                                                                                                                                                                                                                                           
nemo-run_1/0 pass@16           | 198         | 23093      | 81683       | 95.96%           | 0.00%

Server command

SGLANG_DEFAULT_THINKING=1 sglang serve   --trust-remote-code   --model-path nvidia/DeepSeek-V4-Pro-NVFP4   --tp 8   --moe-runner-backend flashinfer_trtllm_routed   --speculative-algo EAGLE   --speculative-num-steps 3   --speculative-eagle-topk 1   --speculative-num-draft-tokens 4   --chunked-prefill-size 8192   --disable-flashinfer-autotune   --swa-full-tokens-ratio 0.1   --mem-fraction-static 0.90   --host 0.0.0.0   --port 30000 --reasoning-parser deepseek-v4

Eval command using https://github.com/sgl-project/sglang/blob/deepseek_v4/scripts/bench_gpqa_aime25.py

PORT=30000 python3 bench_gpqa_aime25.py  setup-ns
PORT=30000 python3 bench_gpqa_aime25.py run-gpqa --num-repeats 16 --temperature 1.0 --max-tokens 400000 --max-concurrency 512

[b8zhong]

/tag-and-rerun-ci

[b8zhong]

@trevor-m Can you please fix the conflicts

[trevor-m]

Hi @SkqLiao thank you for testing it out! You will also need this PR #25702 to help the perf. Even so, for DSV4 Flash we only expect to see 1-5% end-to-end decode throughput improvement with the largest gains on prefill with 5-44% lower TTFT.

outdated

[nvpohanh]

All NVIDIA pipelines passed. @Fridge003 could you merge this? thanks

[Fridge003]

@trevor-m Can we add a nightly test for dsv4 nvfp4 checkpoint, thanks
Could be either 8B200 or 4GB300 nightly test

[trevor-m]

@Fridge003 Thanks for help getting this PR merged! I opened #28966 to add the nightly tests