Skip to content

[CPU] Avoid prefill CP predicates during decode graph capture - #39690

Merged
mingfeima merged 2 commits into
sgl-project:mainfrom
yanbing-j:yanbing/fix_prefill_CP_error
Sep 18, 2026
Merged

mingfeima merged 2 commits into
sgl-project:mainfrom
yanbing-j:yanbing/fix_prefill_CP_error

Conversation

@yanbing-j

@yanbing-j yanbing-j commented Sep 16, 2026 •

Copy link
Copy Markdown
Collaborator

Guard the DSA/MLA prefill context-parallel reduce-scatter checks so they only run for context-parallel extend batches. These predicates are irrelevant for decode, but were still traced during CPU torch.compile decode graph capture and regressed Qwen3.6-35B-A3B decode throughput.

This keeps decode graph capture on the hot path free of prefill-only CP runtime checks while preserving the existing reduce-scatter behavior for context-parallel prefill.

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): 🚫 Run #35186798049
Latest PR Test (Extra): ❌ Run #35186797885
Latest PR Test (AMD ROCm 10): ❌ Run #35186797991

Guard the DSA/MLA prefill context-parallel reduce-scatter checks so they
only run for context-parallel extend batches. These predicates are irrelevant
for decode, but were still traced during CPU torch.compile decode graph
capture and regressed Qwen3.6-35B-A3B decode throughput.

This keeps decode graph capture on the hot path free of prefill-only CP
runtime checks while preserving the existing reduce-scatter behavior for
context-parallel prefill.
@mingfeima mingfeima added intel cpu cpu backend performance optimization run-ci CI: run the baseline test suite on this PR labels Sep 17, 2026
@mingfeima
mingfeima marked this pull request as ready for review September 17, 2026 05:42
@mingfeima
mingfeima merged commit a407915 into sgl-project:main Sep 18, 2026
163 of 181 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cpu cpu backend performance optimization intel run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants