Repository navigation
[AMD] [GLM5] Mark EAGLE verified on MI300X/MI325X (gfx942) in GLM-5.1 cookbook - #29313
Merged
Merged
Conversation
…-5.1 Follow-up to sgl-project#29194. EAGLE speculative decoding is now verified on gfx942 (MI300X/MI325X), not just gfx950: update the AMD note, emit the EAGLE flags for MI300X/MI325X in the deploy generator, and add the flags to the static AMD FP8/BF16 command blocks.
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
Bot
added
the
documentation
Raiden-Makoto
marked this pull request as ready for review
June 25, 2026 16:29
Raiden-Makoto
requested review from
JustinTong0323,
sogalin,
wisclmy0611 and
zijiexia
as code owners
June 25, 2026 16:29
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
HaiShaw
reviewed
Jun 25, 2026
| - **H100 and H200**: FP8 is the recommended deployment path. | ||
| - **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300. | ||
| - **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on MI355X (gfx950) and unverified on MI300X/MI325X (gfx942). | ||
| - **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on all AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950). |
Collaborator
There was a problem hiding this comment.
all AMD GPUs: --> AMD GPUs:
Address HaiShaw review: "all AMD GPUs:" -> "AMD GPUs:".
The aiter custom all-reduce kernel deadlocks during EAGLE target-verify at high concurrency on AMD GPUs (gfx942/gfx950), hanging the server. Add the flag to all AMD EAGLE server commands, emit it from the interactive deployment generator for AMD hardware, and document the requirement in the AMD GPU notes.
HaiShaw
approved these changes
Jun 26, 2026
sogalin
approved these changes
Jun 26, 2026
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Aug 24, 2026
… cookbook (sgl-project#29313) Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #29194, which enabled EAGLE for gfx950 (MI355X) in the GLM-5.1
cookbook but left gfx942 (MI300X/MI325X) marked "unverified" — both in the
prose and in the deploy generator, which excluded the EAGLE flags for those
GPUs. EAGLE speculative decoding has since been verified on gfx942.
Modifications
GLM-5.1.mdx: AMD note now states EAGLE is supported on all AMD GPUs(MI300X/MI325X (gfx942) and MI355X (gfx950)). Added the EAGLE flags to the
static AMD FP8 and BF16 server-command blocks.
glm-51-deployment.jsx: the deploy generator now emits the EAGLE flags onMI300X/MI325X as well (removed the gfx942 exclusion).
Important: EAGLE on AMD requires
--disable-custom-all-reduceEAGLE on AMD GPUs must be run with
--disable-custom-all-reduce. The aitercustom all-reduce kernel (
aiter::cross_device_reduce_2stage) deadlocks duringthe EAGLE target-verify step at high concurrency (reproduced hanging at c>=16
for the (4,1,5) config and c>=32 for (3,1,4) on gfx950/MI355X). The GPU kernel
busy-waits forever on cross-device signal flags and the server hangs with no
recovery. Disabling the custom all-reduce falls back to RCCL, which resolves the
hang with negligible performance impact. This was diagnosed via py-spy + rocgdb,
correlating stuck host-to-device copies with the spinning all-reduce kernel.
Accordingly:
GLM-5.1.mdx: the AMD GPU note now documents that EAGLE requires--disable-custom-all-reduceor the server will hang at high concurrency, andthe flag was added to all three AMD EAGLE server-command blocks (MXFP4 / FP8 / BF16).
glm-51-deployment.jsx: the deploy generator now emits--disable-custom-all-reducewhenever AMD hardware is selected with EAGLE.Checklist
CI States
Latest PR Test (Base): ✅ Run #28208347566
Latest PR Test (Extra): ❌ Run #28208347497