Repository navigation
[PD] Transfer the DCP-replicated DSPARK draft KV in DCP1->DCP-N relayouts - #37709
Conversation
…outs The DCP1->DCP-N relayout (sgl-project#35762) covered only the owner-strided target MLA rows; DSPARK was left to a follow-up. The decode's draft pool is replicated across DCP ranks and indexes the shared allocator's widened virtual locs raw, so its rows need a second, non-strided plan: every chunk token to every rank, addressed at dst_pages[pos // (P*N)] * (P*N) + pos % (P*N) from the same page list the target plan collapses by dcp_size. - KVArgs.num_draft_entries marks the draft tail of kv_data_ptrs on both peers. - build_dcp_replicated_token_transfer_plan emits the widened-loc plan; the draft rows are page-contiguous on the prefill so they need no pack gather. - prepare_dcp_token_item_lens validates the draft tail at the widened page scale (page_size * dst_dcp_size) instead of rejecting it as a geometry mismatch; pack buffers size from target entries only. - mooncake sends both parts in one synchronous call; NIXL splits them into _part_{i}_{n} transfers reassembled by the existing chunk-part tracking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Ran HEAD ( Mooncake (default; IB uverbs not accessible in the pod → TCP fallback)
mean 0.9215, std 0.0038, min 0.915, max 0.925 NIXL (UCX)
mean 0.9295, std 0.0055, min 0.920, max 0.935 Accept length: 1.00 on every run / decode-log sample ( |
972e2fd to
0b4a285
Compare
Drop the replicated-plan helper and dest-page validators. One call fills both index maps so mooncake/nixl do not plan twice. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Ran HEAD (
Accept length is the dummy-draft bonus token only ( |
Replace formula-echo checks with small P/N cases so a swapped dest page size or a rank-ownership bug fails. Co-authored-by: Cursor <cursoragent@cursor.com>
|
/rerun-test test/registered/disaggregation/test_kimi_linear_pd_dcp4_dspark.py test/registered/disaggregation/test_kimi_linear_pd_dcp4.py |
|
Results for 🚀 |
|
/rerun-test test/registered/disaggregation/test_kimi_linear_pd_dcp4_dspark.py test/registered/disaggregation/test_kimi_linear_pd_dcp4.py |
|
Results for 🚀 |
|
/rerun-group disaggregation |
|
|
/rerun-group disaggregation |
|
Results for 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 |
…outs Cherry-pick of #37709 (net diff). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…red/e2e/ main now requires new registered tests to live under test/registered/<kind>/<subsystem>/ (scripts/lint/check_registered_tests.py). The test launches a PD pair and checks GSM8K plus that spec-verify runs on the decode side, so it is an e2e test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…virtual page test_second_chunk_crosses_dest_pages placed the chunk at send-range pos 4..7, which is exactly one virtual page (P*N=4), yet its draft golden mixed rows from virtual pages 4 and 6. Start the chunk at src_page_offset=1 (pos 2..5) so it really crosses the page boundary, and pin both target and draft goldens to the row mapping the plan builder (and main's target path) implement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
| "--speculative-draft-load-format", | ||
| "dummy", |
There was a problem hiding this comment.
Could this CI test detect a wrong draft mapping?
There was a problem hiding this comment.
Yeah this test is not really strong. We will train a separate drafter for Kimi Linear and use it in CI in the future
ShangmingCai
left a comment
ShangmingCai
left a comment
There was a problem hiding this comment.
Looks good. Verified and tested. No other comment.
|
/rerun-group disaggregation |
|
/rerun-test test/registered/disaggregation/test_kimi_linear_pd_dcp4_dspark.py test/registered/disaggregation/test_kimi_linear_pd_dcp4.py |
|
Results for 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 |
|
Results for 🚀 ⛔ |
|
/rerun-test test/registered/e2e/disaggregation/test_kimi_linear_pd_dcp4_dspark.py |
|
Results for 🚀 |
…ontract `send_kvcache_dcp` now returns one handle per transfer part (target + draft), and the worker extends its handle list with the result. The test merged from main (sgl-project#35762) still mocked a single string handle, so `extend` split it into characters and the chunk barrier polled 41 fake handles instead of 3. The fake `kv_args` also lacked `num_draft_entries`, which the pack path now reads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follow-up to #35762: ship DSpark draft KV from a DCP1 prefill to a DCP-N decode.
Design
Decode publishes one virtual page list. A virtual page is
P*Ntokens (P= physical page,N= decode DCP size). Each pool then lays that page out at its own width:Target: strided pick + #35762 pack. Draft: full copy, no pack.
Code
KVArgs.num_draft_entriesmarks the draft tail ofkv_data_ptrs.build_dcp_token_transfer_planfills both index maps in one call.prepare_dcp_token_item_lensscales draft dest pages byN. Pack buffers stay target-only._part_{i}_{n}notif).Out of scope: DCP>1 prefill, heterogeneous P/D TP × DSPARK.
Tests
test_dcp_pack.py: golden rows, chunk crossing, pack vs no-pack grouping, item-len scale.test_kimi_linear_pd_dcp4_dspark.py: Kimi-Linear TP4 DCP1 → DCP4 DSPARK, GSM8K.Speed Tests
8xB300,
Kimi-Linear-48B-A3B-Instruct, dummy DSPARK +SGLANG_SIMULATE_ACC_LEN=4(measured accept length 4.00). Same topology as #35762. OSL 32. ISL 128k/256k measured with SSE-framed TTFT (MiniLB can coalesce events;sglang.benchmark.servingtreating each HTTP chunk as one JSON reported Mooncake 128k TTFT=0).NIXL
Mooncake (TCP)
CI States
Latest PR Test (Base): 🚫 Run #34654691933
Latest PR Test (Extra): ⏳ Run #34661035645
Latest PR Test (AMD ROCm 10): ❌ Run #34654691978