Skip to content

[Refactor] TCPCG: Consolidate attention eager regions and isolate Inkling policy (4/9) - #42288

Merged
Oasis-Git merged 13 commits into
sgl-project:mainfrom
Oasis-Git:refactor/tcpcg-04-attention-eager
Oct 17, 2026
Merged

Oasis-Git merged 13 commits into
sgl-project:mainfrom
Oasis-Git:refactor/tcpcg-04-attention-eager

Conversation

@Oasis-Git

@Oasis-Git Oasis-Git commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

This is step 4/9, split from the original TCPCG removal PR #41634.

Steps 1–3 are merged. Rebased onto upstream main (58cb1a2be6); the PR contains only step 4. Review the step-4 files.

Replace attention's TCPCG split-op adapters and layer-registry lookups with class-owned eager methods. On BCG replay, these methods read the live prepared batch through get_forward_batch() from step 2's runner-owned scope. The eager decorator and replay callbacks do not capture or rebind a batch argument.

Series

  1. Relocate shared graph tensor and DSA head-gate helpers — [Refactor][TCPCG] Relocate shared graph tensor and DSA head-gate helpers (1/9) #42285 (merged).
  2. Simplify BCG eager replay and publish the live batch in runner utilities — [Refactor][TCPCG] Bind live forward batches in BCG eager calls (2/9) #42286 (merged).
  3. Retire TCPCG backend selection, configuration, documentation, and tests — [Refactor][TCPCG] Retire backend selection and configuration (3/9) #42287 (merged).
  4. Consolidate attention eager regions and isolate Inkling policy — this PR.
  5. Migrate DSA and DeepSeek eager regions — [Refactor][TCPCG] Migrate DSA and DeepSeek graph breaks to eager methods (5/9) #42289.
  6. Migrate MoE and Mamba eager regions — [Refactor][TCPCG] Migrate MoE and Mamba graph execution to eager methods (6/9) #42290.
  7. Migrate diffusion eager wrappers and standardize names — [Refactor][TCPCG] Standardize diffusion eager graph wrappers (7/9) #42291; depends only on step 2.
  8. Remove obsolete runtime contexts, layer registries, split-op plumbing, and temporary compatibility after caller migration — [Refactor][TCPCG] Remove obsolete runtime registries and split-op plumbing (8/9) #42292.
  9. Simplify quantization paths separately because of their torch.compile implications — [Refactor][TCPCG] Simplify quantization paths after TCPCG retirement (9/9) #42293.

Changes

  • Consolidate dense attention, optional LSE, sparse handling, and extra keyword arguments in RadixAttention._eager_attention. Preserve main’s sparse-BCG fix: sparse attention also runs in an eager region and reads the live batch during replay.
  • Share output allocation, token slicing, temporary batch-state restoration, and padded-tail zeroing through attention/graph_utils.py. Preserve the newer eager padded-extend handling, including independent query/KV extents and tuple outputs.
  • Migrate linear attention, HPC RoPE/store-KV, and MLA BMM attention to class-owned eager methods. Preserve the newer linear-attention path that stays inside BCG when the backend supports it.
  • Keep Inkling's attention bypass and token handling in the model, without making the eager decorator nesting-aware.
  • Keep Cheng's ForwardContext unchanged. Publish the full-prefill flag and raw token count in a separate runner-owned prefill_graph_scope, with restoration on exit.

The legacy TCPCG context and layer registries remain temporarily for unmigrated callers until step 8. This changes attention dispatch and eager replay, while preserving backend computation and graph-owned output buffers.

Review fix: prefill scope

Require the runner-owned prefill scope before entering the migrated eager methods. Decode BCG also handles TARGET_VERIFY and DLLM_EXTEND, which count as extend but do not publish a prefill batch. These modes retain their original backend path. Apply the scope boundary to Radix, linear, HPC, MLA fusion eligibility, and Inkling eager/fusion gates. Add CPU regressions for decode BCG without a batch scope.

Review fixes: compiler compatibility and BCG test coverage

  • Skip the runner-only prefill ContextVar lookups while compiling. BCG/full prefill is uncompiled; compiled decode/target verification retains the ordinary backend path.
  • The shared BCG attention test runner now publishes both the live batch and prefill scope, and spies on migrated Radix/linear eager methods to prevent silent fallback coverage.
  • Compile the actual RadixAttention.forward entry for decode and target verification, both with the real absent scope and a sentinel that rejects context reads.
  • Isolated macOS PyTorch 2.10 CPU validation: the four compile regressions fail with the previous helpers and pass with the fix; all 34 Radix tests pass. Accelerator imports are shimmed; no NPU inference was run.
  • Current local CPU suite: 83 tests plus 27 subtests pass. A CPU harness executing the shared runner function with fixture dependencies isolated confirms padded eager dispatch and that removing the scope trips the new assertion. GPU backend suites remain unrun.

Validation

  • Pre-commit passed on all 14 files in the step-4 diff.
  • Attention, padded-extend, linear-attention, and Inkling fusion-gating CPU tests: 83 passed plus 27 subtests. These exercise real implementations with a macOS import shim for accelerator-dependent imports.
  • Radix attention's CI-style script entry (python file.py -f): 30 passed under the same shim.
  • CI collection sanity check: 2,243 test files / 2,741 registrations, including executable test-entry validation.
  • Added dense/LSE/sparse live-batch replay, nested scope restoration, and linear-attention in-graph dispatch coverage. Updated the GPU attention regression test for the new class method; it has not been run locally.
  • CUDA capture/replay, model accuracy, performance, and full CI remain unvalidated. This update is prepared for code review; it does not claim GPU validation.

CI States

Latest PR Test (Base): ✅ Run #39007966841
Latest PR Test (Extra): ❌ Run #39007966810
Latest PR Test (AMD ROCm 10): ❌ Run #39007966845

@Oasis-Git Oasis-Git added run-ci CI: run the baseline test suite on this PR bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) run-ci-extra CI: also run the extra suite (requires run-ci) and removed run-ci CI: run the baseline test suite on this PR labels Oct 13, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator Author

@Oasis-Git Oasis-Git removed the run-ci-extra CI: also run the extra suite (requires run-ci) label Oct 13, 2026
@Oasis-Git

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

@github-actions

This comment has been minimized.

@Oasis-Git

Copy link
Copy Markdown
Collaborator Author

@Oasis-Git Oasis-Git changed the title [Refactor][TCPCG] Consolidate attention eager regions and isolate Inkling policy (4/9) [Refactor] TCPCG: Consolidate attention eager regions and isolate Inkling policy (4/9) Oct 17, 2026
@Oasis-Git
Oasis-Git merged commit b234b5c into sgl-project:main Oct 17, 2026
202 of 231 checks passed
adarshxs added a commit to adarshxs/sglang that referenced this pull request Oct 26, 2026
Resolve conflicts and port the transformers backend to current main:

- pyproject: keep main's transformers==5.19.0 and uvicorn>=0.51 pins
  (drop this branch's 5.15.0 bump).
- model_config: keep main's EmbeddingGemma2 context clamp and apply the
  transformers embedding-plan clamp on top of it.
- lora_manager.set_lora_module: keep main's **kwargs passthrough together
  with the transformers LoRA adapter wrap.
- Delete the old models/transformers.py and port main's two changes to it:
  the embedding_rowwise TP-style alias (sgl-project#39012) into layers.py, and the
  pad_mm_tokens helper (sgl-project#44845) into MultiModalMixin.pad_input_ids. Export
  _normalize_tp_style from the package for test_hf_transformers.
- trtllm_mha_backend: drop this branch's causal= kwarg; main added the
  identical line, and the auto-merge produced a repeated keyword.
- Parallel linear layers take parallel_group instead of tp_rank/tp_size
  (sgl-project#42588): the fused QKV projection partitions on "attn_tp" and unfused
  K/V shards use ReplicatedParallelGroup("attn_tp", tp_size // kv_heads).
  TP unit tests build group stubs via sglang.test.parallel_groups.
- test_transformers_graph: drop the TCPCG fullgraph compile test, since
  the backend is retired and its custom attention op removed
  (sgl-project#42287, sgl-project#42288); give the hand-built PrefillCudaGraphRunner the
  attributes main's replay path now reads.
- test_embedding_model_spec: the support matrix has 10 rows (ModernBert
  and Roberta from this branch plus main's EmbeddingGemma2).
- test_transformers_fallback: follow main's removal of the skip-substrs
  test (sgl-project#43503).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) deepseek documentation Improvements or additions to documentation Multi-modal multi-modal language model npu piecewise-cuda-graph run-ci CI: run the baseline test suite on this PR speculative-decoding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant