[AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP by michaelzhang-ai · Pull Request #29982 · sgl-project/sglang

michaelzhang-ai · 2026-07-02T22:44:33Z

Motivation

#29775 flipped SGLANG_OPT_FLASHMLA_SPARSE_PREFILL to default-on for all DeepSeek-V4 runs. Starting with the first main nightly that included it, the AMD MI355X 1P1D disaggregation nightly (Nightly Test (AMD MI355X 2N 1P1D Disagg)) regressed for DeepSeek-V4-Flash while DeepSeek-V4-Pro stayed green.

Failing run: #28565894749 (scheduled, main, 2026-07-02). Every dsv4flash job (fp8/fp4, base/mtp/dp8ep8) failed; every dsv4pro job passed.

Evidence it is the sparse prefill path

The 07-01 nightly (#28494350421) was green and already contained the transformers 5.12.1 bump ([Deps] Bump transformers to 5.12.1 #29393), ruling that out. The regression window is the commits merged between the 07-01 and 07-02 runs; [DeepSeek V4] Enable FlashMLA sparse prefill by default #29775 (merged 2026-07-01) is the DSV4-specific change that flips this default.
On the decode worker the disagg warmup returns garbage: output_ids: [0, 0, 201, 0, 410, 0, 12389, 0] (near-all-zero). The subsequent PD /generate then fails to serve (decode times out -> 500 -> router 502 -> PD path not serving; aborting), and the GSM8K gate reports Accuracy: <not found in bench.log>.
DeepSeek-V4-Pro on the same MI355X nodes is unaffected, i.e. the dense prefill path (the pre-[DeepSeek V4] Enable FlashMLA sparse prefill by default #29775 default) is correct on ROCm.

Modification

Make the default platform-conditional using the existing _default_hip callable-default pattern already used in environ.py (e.g. SGLANG_DP_USE_REDUCE_SCATTER = EnvBool(_default_hip)):

CUDA: default on — preserves the [DeepSeek V4] Enable FlashMLA sparse prefill by default #29775 optimization.
ROCm/HIP: default off — restores the dense prefill path that was green on the MI355X nightly.

Explicitly setting SGLANG_OPT_FLASHMLA_SPARSE_PREFILL still overrides the default, so the sparse path remains available on ROCm for validation/opt-in.

SGLANG_OPT_FLASHMLA_SPARSE_PREFILL = EnvBool(lambda: not _default_hip())

This is a conservative mitigation to unbreak the nightly; it does not touch NVIDIA behavior. cc @YAMY1234 — happy to flip ROCm back to default-on once the FlashMLA sparse prefill kernel is validated for the Flash variant on MI355X.

Accuracy Test

No accuracy impact on CUDA (default unchanged).
On ROCm the default reverts to the dense prefill path that produced correct GSM8K accuracy in the prior green nightlies (e.g. #28494350421).

Checklist

Minimal, platform-scoped change; explicit env override preserved.
Re-run the MI355X disagg nightly to confirm DeepSeek-V4-Flash is restored.

CI States

Latest PR Test (Base): 🚫 Run #28626803369
Latest PR Test (Extra): ❌ Run #28626803260

#29775 flipped SGLANG_OPT_FLASHMLA_SPARSE_PREFILL to default-on for all DeepSeek-V4 runs. On AMD MI355X this regressed the DeepSeek-V4-Flash disaggregation nightly: the sparse prefill path returns garbage output (warmup emits near-all-zero token ids), the PD decode path then fails to serve and the GSM8K accuracy gate reports no result. DeepSeek-V4-Pro on the same nodes is unaffected, so the dense prefill path (the pre-#29775 default) remains correct on ROCm. Make the default platform-conditional using the existing _default_hip callable-default pattern: keep default-on for CUDA (preserving the #29775 optimization) and default-off for ROCm/HIP, restoring the behavior that was green on the MI355X nightly. Explicitly setting the env var still overrides this, so the sparse path stays available on ROCm for validation.

gemini-code-assist · 2026-07-02T22:44:49Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Fridge003 · 2026-07-02T22:54:41Z

+    # breaks the disaggregation nightly. Keep the previous (dense prefill) default
+    # on ROCm until the sparse kernel is validated for that variant. Explicitly
+    # setting SGLANG_OPT_FLASHMLA_SPARSE_PREFILL still overrides this.
+    SGLANG_OPT_FLASHMLA_SPARSE_PREFILL = EnvBool(lambda: not _default_hip())


Please set this in python/sglang/srt/arg_groups/deepseek_v4_hook.py

Done — moved the ROCm default-off into apply_deepseek_v4_defaults in deepseek_v4_hook.py and reverted environ.py back to EnvBool(True). It disables SGLANG_OPT_FLASHMLA_SPARSE_PREFILL by default only on ROCm/HIP and only when not explicitly set, mirroring the existing CP-based disable. Thanks!

Address review: instead of making the environ default platform-conditional, keep SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=True in environ.py and disable it by default on ROCm/HIP inside apply_deepseek_v4_defaults (alongside the existing CP-based disable). An explicitly set env var still overrides the default.

michaelzhang-ai · 2026-07-02T22:58:48Z

Updated per review: reverted environ.py and moved the ROCm/HIP default-off into apply_deepseek_v4_defaults in deepseek_v4_hook.py (only when the env var isn't explicitly set), mirroring the existing CP-based disable.

Validation run on the latest commit (6f8a474): https://github.com/sgl-project/sglang/actions/runs/28626848640

Scope: all 8 DeepSeek-V4-Flash configs (fp8/fp4 × base/mtp/dp8ep8/dp8ep8-mtp) + dsv4pro-fp8-1k1k-1p1d control.
The earlier run (on the pre-review commit) was cancelled.

Will post per-job results here once it finishes.

michaelzhang-ai · 2026-07-02T23:02:56Z

Correction: the manual workflow_dispatch validation runs (including the one on main, #28626999997) all got cancelled while pending — the nightly-test-mi355x concurrency group only keeps one pending run, so each newer run (and the scheduled nightly) superseded the previous pending one before it could grab the single MI355X node. None of them executed any jobs.

Validation is instead being covered by the scheduled nightly on main, which runs at the merge commit's descendant (contains this fix): https://github.com/sgl-project/sglang/actions/runs/28638558731

It runs the full DSV4 matrix (all Flash + Pro configs). I'll follow up here with the DeepSeek-V4-Flash results once it completes.

michaelzhang-ai · 2026-07-03T15:47:42Z

Important correction on validation: this nightly runs sglang from the prebuilt lmsysorg/sglang-rocm docker image, not from checked-out main. The scheduled run #28638558731 resolved image ...-mi35x-20260702 (built before this PR merged at 2026-07-02 23:00 UTC), so it did not exercise this fix — DeepSeek-V4-Flash still failed there as expected.

Re-validating against the newer image ...-mi35x-20260703 (built 2026-07-03 15:18 UTC, which should contain 8519be8), pinned via the image dispatch input: https://github.com/sgl-project/sglang/actions/runs/28670714372

Note: that same 07-02 image also shows a separate regression where dsv4pro-fp8 fails (it passed on 07-01/07-02 image); dsv4pro-fp4 still passes. That's unrelated to this PR. Will report the Flash results from the 07-03-image run here once it finishes.

michaelzhang-ai · 2026-07-03T21:58:58Z

Validation update — this change does not fix the DeepSeek-V4-Flash disagg nightly. Flagging so we can decide on a follow-up/revert.

I verified against image lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260703 (built 2026-07-03 15:07 UTC) and extracted the baked source to confirm this PR is present in that image (deepseek_v4_hook.py contains the ROCm default-off logic). Dispatched run: https://github.com/sgl-project/sglang/actions/runs/28670714372

Result: all 8 dsv4flash configs still fail with the identical symptom as before — the disagg warmup returns structured garbage, e.g. output_ids: [0, 0, 201, 0, 1263, 0, 1, 0] (odd positions always 0, index 2 always 201), then the PD path fails to serve and the GSM8K gate reports no accuracy. (dsv4pro-fp4 passes; dsv4pro-fp8 fails for a separate reason introduced in a newer image.)

Why the fix is ineffective: the warmup request is tiny (4 prompt tokens), so even with SGLANG_OPT_FLASHMLA_SPARSE_PREFILL defaulted off it should take the dense prefill path (q.shape[0] well under _LARGE_INDEXER_QUERY_THRESHOLD), yet it still produces garbage. That points to the root cause being outside the FlashMLA sparse-prefill path (likely a DSV4-Flash-specific decode/output issue on ROCm), so gating that flag does not help.

Suggested next steps:

Treat this as an open DSV4-Flash correctness bug on MI355X that needs debugging on the hardware — the structured [0, x, 0, y, ...] output is a strong fingerprint.
This PR is now just a conservative ROCm default (reverts to pre-[DeepSeek V4] Enable FlashMLA sparse prefill by default #29775 behavior) and does not achieve its stated goal. Happy to revert it if you'd prefer ROCm stay on the sparse path, or keep it as-is — your call.

Apologies for the premature "fix" framing; I couldn't validate on hardware before merge due to the MI355X queue.

HaiShaw · 2026-07-03T22:08:17Z

@1am9trash @RolaoDenthu @kkHuang-amd - FYI.

HaiShaw

Please revise. cc @bingxche

HaiShaw · 2026-07-06T08:02:19Z

+    # (MI355X), which breaks the disaggregation nightly. Keep the previous
+    # (dense prefill) behavior on ROCm until the sparse kernel is validated
+    # there; an explicit env var still overrides this.
+    if is_hip() and not envs.SGLANG_OPT_FLASHMLA_SPARSE_PREFILL.is_set():


should we remove the not, or remove the second condition?

Remove the second condition — it should just be if is_hip():.

The not ...is_set() guard was the bug in this PR: the SGLANG_OPT_FLASHMLA_SPARSE_PREFILL default gets materialized into os.environ, so is_set() is already True by the time the hook runs → the guard skipped the disable, sparse prefill stayed on, and DeepSeek-V4-Flash kept producing garbage. That's why this PR didn't actually fix the nightly. Removing the not instead would be worse (disable only when explicitly set).

This is already handled by #30237 (merged), which changed it to if is_hip(): — validated on image ...-mi35x-20260706: dsv4flash-fp8-1k1k-1p1d now passes the GSM8K gate, and an A/B run with sparse re-enabled still reproduces the exact [0, 0, 201, 0, ...] garbage, confirming FlashMLA sparse prefill was the root cause after all.

Small follow-up #30289 just cleans up the now-stale "set it explicitly to override" wording in the warning/comment (since the disable is now unconditional on ROCm/HIP). Full flash-matrix re-validation in flight: https://github.com/sgl-project/sglang/actions/runs/28817206537

sgl-project#29982)

michaelzhang-ai requested review from Fridge003 and HaiShaw July 2, 2026 22:49

michaelzhang-ai added the run-ci CI: run the baseline test suite on this PR label Jul 2, 2026

michaelzhang-ai requested a review from YAMY1234 July 2, 2026 22:51

michaelzhang-ai changed the title ~~[DeepSeek V4] Default FlashMLA sparse prefill off on ROCm/HIP~~ [AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP Jul 2, 2026

Fridge003 reviewed Jul 2, 2026

View reviewed changes

github-actions Bot added the deepseek label Jul 2, 2026

Fridge003 approved these changes Jul 2, 2026

View reviewed changes

Fridge003 merged commit 8519be8 into main Jul 2, 2026
58 of 86 checks passed

Fridge003 deleted the michaelzhang-ai/dsv4-flashmla-sparse-prefill-rocm-default branch July 2, 2026 23:00

amd-bot mentioned this pull request Jul 3, 2026

[CI Monitor] Daily Report - 2026-07-03 bingxche/sglang-ci-bot#123

Open

HaiShaw assigned 1am9trash and RolaoDenthu Jul 3, 2026

This was referenced Jul 4, 2026

[CI Monitor] Daily Report - 2026-07-04 bingxche/sglang-ci-bot#125

Open

[CI Monitor] Daily Report - 2026-07-05 bingxche/sglang-ci-bot#126

Open

[CI Monitor] Daily Report - 2026-07-06 bingxche/sglang-ci-bot#127

Open

HaiShaw reviewed Jul 6, 2026

View reviewed changes

This was referenced Jul 6, 2026

[DeepSeek V4] Fix stale override wording in ROCm FlashMLA sparse-prefill disable #30289

Closed

[AMD][DeepSeek V4] Set SGLANG_OPT_FLASHMLA_SPARSE_PREFILL to false on hip code path #30237

Merged

amd-bot mentioned this pull request Jul 7, 2026

[CI Monitor] Daily Report - 2026-07-07 bingxche/sglang-ci-bot#128

Open

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP (

1e82601

sgl-project#29982)

Navigation Menu

Repository navigation

[AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP - #29982

[AMD][DeepSeek V4] Fix default FlashMLA sparse prefill off on ROCm/HIP#29982
Fridge003 merged 2 commits into
mainfrom
michaelzhang-ai/dsv4-flashmla-sparse-prefill-rocm-default

michaelzhang-ai commented Jul 2, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 2, 2026

Uh oh!

Fridge003 Jul 2, 2026

Uh oh!

michaelzhang-ai Jul 2, 2026

Uh oh!

michaelzhang-ai commented Jul 2, 2026

Uh oh!

Uh oh!

michaelzhang-ai commented Jul 2, 2026 •

edited

Loading

Uh oh!

michaelzhang-ai commented Jul 3, 2026 •

edited

Loading

Uh oh!

michaelzhang-ai commented Jul 3, 2026 •

edited

Loading

Uh oh!

HaiShaw commented Jul 3, 2026

Uh oh!

HaiShaw left a comment

Uh oh!

HaiShaw Jul 6, 2026 •

edited

Loading

Uh oh!

michaelzhang-ai Jul 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

michaelzhang-ai commented Jul 2, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Evidence it is the sparse prefill path

Modification

Accuracy Test

Checklist

CI States

Uh oh!

gemini-code-assist Bot commented Jul 2, 2026

Uh oh!

Fridge003 Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

michaelzhang-ai Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

michaelzhang-ai commented Jul 2, 2026

Uh oh!

Uh oh!

michaelzhang-ai commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelzhang-ai commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michaelzhang-ai commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HaiShaw commented Jul 3, 2026

Uh oh!

HaiShaw left a comment

Choose a reason for hiding this comment

Uh oh!

HaiShaw Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelzhang-ai Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

michaelzhang-ai commented Jul 2, 2026 •

edited by github-actions Bot

Loading

michaelzhang-ai commented Jul 2, 2026 •

edited

Loading

michaelzhang-ai commented Jul 3, 2026 •

edited

Loading

michaelzhang-ai commented Jul 3, 2026 •

edited

Loading

HaiShaw Jul 6, 2026 •

edited

Loading