[b8zhong]

Motivation

Since it seem FI 0.6.14 is blocked by PyPi limit for an indeterminate amount of time.

Modifications

#29865
Optimization process to collect dynamic autotuning results into static heuristic (and save autotune time + extra compilation time):

1. First collect all the accurate runtime shapes by calling Flashinfer mm_bf16 for all BF16 linears which will accurately collect (M,N,K) shape data.
2. Autotune tactics with cold L2 and CUDA graph (M from 1 to 8192) with flashinfer.testing.bench_gpu_time_with_cupti and CUDA graph, build a naive heuristic.
3. Then run comprehensive autotune with all tactics (offline), then ask Claude to refine this heuristic based on the empirical data.

For most shapes, <1% difference. It's more conservative, as staying to the default is safer than not.

Default enablement coming later, since it'd benefit from further optimizations.

Accuracy Tests

sgl-eval run aime25 \
    --base-url http://localhost:30000/v1 \
    --n-repeats 8 \
    --max-tokens 65536 \
    --temperature 1.0 \
    --top-p 0.95 \
    --num-threads 64
Run directory: /root/.sgl_eval/sgl_eval_aime25_20260704-120243
aime25 rep 1/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=90.00%]
aime25 rep 2/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=96.67%]
aime25 rep 3/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=90.00%]
aime25 rep 4/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=90.00%]
aime25 rep 5/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=93.33%]
aime25 rep 6/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=90.00%]
aime25 rep 7/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=93.33%]
aime25 rep 8/8: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [34:53<00:00, 69.77s/it, acc=86.67%]
aime25 overall: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 240/240 [34:53<00:00,  8.72s/it, acc=91.25%]
== aime25 ==
30 examples x 8 repeats  |  2093.0s  |  2323 tok/s  |  4.9M tokens

* pass@1[avg-of-8]  =  91.25% +/- 3.05% (SEM 1.08%)
  pass@8            =  100.00%
  majority@8        =  93.33%
  no_answer         =  0.00%
  stop_rate         =  100.00%
  truncated_rate    =  0.00%
  error_rate        =  0.00%

python test/registered/jit/test_cutedsl_tgv_gemm.py
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /sgl-workspace/sglang/test
configfile: pytest.ini
plugins: anyio-4.14.1, typeguard-4.5.2
collected 384 items                                                                                                                                                                                             
..
=============================================================================================== warnings summary ================================================================================================
../../usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py:1309
../../usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py:1309
  /usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py:1309: PytestAssertRewriteWarning: Module already imported so cannot be rewritten; anyio
    self._mark_plugins_for_rewrite(hook, disable_autoload)

../../usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py:1434
  /usr/local/lib/python3.12/dist-packages/_pytest/config/__init__.py:1434: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================== 384 passed, 3 warnings in 4.07s ========================================================================================

Speed Tests and Profiling

sglang serve \
    --trust-remote-code \
    --model-path nvidia/GLM-5.2-NVFP4 \
    --tp 4 \
    --quantization modelopt_fp4 --bf16-gemm-backend cutedsl

New kernels being called:

---

CI States

Latest PR Test (Base): ❌ Run #28827300600
Latest PR Test (Extra): ❌ Run #28827300418

[gemini-code-assist]

Code Review

This pull request introduces a low-latency Blackwell GEMM kernel (cutedsl_tgv_gemm.py) written using CuTe DSL (cute_ext primitives) for BF16 inputs, FP32 accumulation, and BF16 output, targeting SM100/SM103 architectures. It integrates this new TGV backend into the unquantized GEMM path, adds server arguments to select the BF16 GEMM backend, and updates the scheduler and DeepSeek-V2 model to support it. The review feedback suggests improving the robustness of _detect_leading_dim by searching for the stride-1 dimension from right to left, explicitly validating the upper bound of 256 for cta_n, and correcting a minor formatting typo in the help string for --bf16-gemm-backend.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[b8zhong]

/rerun-test test/registered/jit/test_cutedsl_bf16_gemm.py

[github-actions]

Results for /rerun-test test/registered/jit/test_cutedsl_bf16_gemm.py:

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/jit/test_cutedsl_bf16_gemm.py