Repository navigation
Refactor FP4 quantization and remove deprecated JIT kernels - #30448
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
Bot
added
quant
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
Bot
added
the
documentation
Left over from the FP4 quantization refactor: suggest_nvfp4_global_scale had no remaining callers after the fused NVFP4 MoE quant kernel was removed.
# Conflicts: # python/sglang/srt/layers/moe/moe_runner/runner.py
|
|
||
| FP4_GEMM_RUNNER_BACKEND_CHOICES = [ | ||
| "auto", | ||
| "cutlass", |
There was a problem hiding this comment.
Need to check whether any existing docs or scripts still use this parameter and were not updated.
There was a problem hiding this comment.
Just checked, and all the references are deleted. Except for docs/, which will be deleted entirely
| FlashInferCutlassMoeQuantInfo, | ||
| ) | ||
|
|
||
| assert ( |
There was a problem hiding this comment.
This switches the compressed-tensors NVFP4 MoE path from the old direct CUTLASS implementation to FlashInfer CUTLASS, but apply_router_weight_on_input=True now asserts during forward. If this mode is no longer supported, can we reject it earlier with a clear config error or keep a fallback for this case?
There was a problem hiding this comment.
Ok, just add an assertion. The only model that use apply router weight is Llama4, and it doesn't use compressed-tensors NVFP4, but actually modelopt FP4.
BBuf
left a comment
BBuf
left a comment
There was a problem hiding this comment.
We need more benchmark results in H100/H200/B200 and so on.
We also need some end2end model accracy results, please compare the accuracy with the main branch. |
|
As expected, the performance of these kernels are no good anymore. On SM100: For MoE: |
|
/rerun-failed-ci |
|
For anyone debugging (in case this causes CI failures): since this PR is large, you can immediately message me on Slack if there CI failures, and I'll address them promptly. |
…ect#30448) Co-authored-by: root <root@sgl-b300-inference.datacrunch.io>
For MoE, must use Flashinfer CUTLASS on SM120 or trtllm-gen on SM100 (better perf in all cases) (high throughput use Flashinfer Cute-DSL)
For dense GEMM on SM100, use Flashinfer Cute-DSL, on SM120, use Flashinfer CUTLASS
For quantize, use backend = cute-dsl of fp4_quantize in Flashinfer
Fix #28663
CI States
Latest PR Test (Base): ❌ Run #29263311113
Latest PR Test (Extra): ❌ Run #29263310306