Repository navigation
[Fix] [AMD] GLM5: Fix MTP layer_quant_config in-place mutation and add nextn Quark-exclude unit test - #30565
Open
Raiden-Makoto wants to merge 10 commits into
Open
[Fix] [AMD] GLM5: Fix MTP layer_quant_config in-place mutation and add nextn Quark-exclude unit test#30565Raiden-Makoto wants to merge 10 commits into
Raiden-Makoto wants to merge 10 commits into
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Raiden-Makoto
force-pushed
the
tmp/eagle-mtp
branch
from
July 8, 2026 23:46
cb0127c to
ae4073f
Compare
…de tests GlmMoeDsaForCausalLMNextN._resolve_nextn_quant_config (sgl-project#30265) remapped layer_quant_config keys by mutating the shared quant_config in place, which can corrupt the main model's per-layer scheme selection (flagged in the sgl-project#30265 review). Copy quant_config and its nested dict before any mutation, and only copy when a remap/exclude is actually needed. Tests (the missing sgl-project#30265 coverage): - test/registered/unit/models/test_nextn_quark_exclude.py (CPU CI, base-a-test-cpu): base DeepseekV3ForCausalLMNextN passthrough plus the GLM-5.2 MTP exclude / layer_quant_config remap and no-mutation guarantees. - test/registered/amd/accuracy/mi35x/test_glm52_mxfp4_mtp_eval_mi35x.py (nightly-amd-8-gpu-mi35x-glm52-mxfp4-mtp): 8-GPU GLM-5.2-MXFP4 EAGLE-MTP GSM8K eval verifying the bf16 nextn layer loads under Quark MXFP4; wired into nightly-test-amd.yml and nightly-test-amd-rocm720.yml. Fix + CPU test originally by @fanxingran (sgl-project#30379); references sgl-project#29781. Co-authored-by: fanxingran <xingran.fan@amd.com>
Raiden-Makoto
force-pushed
the
tmp/eagle-mtp
branch
from
July 9, 2026 00:02
ae4073f to
8452c9e
Compare
5 tasks
Raiden-Makoto
marked this pull request as ready for review
July 9, 2026 14:35
Raiden-Makoto
requested review from
Fridge003,
HaiShaw,
Kangyan-Zhou,
bingxche,
ispobock and
merrymercy
as code owners
July 9, 2026 14:35
This was referenced Jul 10, 2026
This was referenced Jul 18, 2026
This was referenced Jul 27, 2026
HaiShaw
approved these changes
Aug 17, 2026
Collaborator
|
/tag-and-rerun-ci |
Contributor
Author
|
@amd-bot ci-status |
Contributor
Author
|
CI tests failed due to rate limit. |
Contributor
Author
|
/rerun-failed-ci |
1 similar comment
Contributor
Author
|
/rerun-failed-ci |
# Conflicts: # .github/workflows/nightly-test-amd-rocm720.yml
# Conflicts: # .github/workflows/nightly-test-amd-rocm720.yml
3 of 5 tasks
Contributor
Author
|
/rerun-failed-ci |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #30265 / #29781 (supersedes #30379). #30265 merged with two gaps flagged in its own review:
nightly=Trueand does not enable MTP; the GLM MTP e2e test is CUDA-only + FP8, not Quark MXFP4.GlmMoeDsaForCausalLMNextN._resolve_nextn_quant_configremapslayer_quant_configkeys by mutating the sharedquant_configin place, which can corrupt the main model's per-layer scheme selection (gemini-code-assist review on [AMD] Fix GLM-5.2 MTP Quark excludes #30265).The fix and the CPU test here originate from #30379 (@fanxingran, co-authored).
Modifications
python/sglang/srt/models/glm4_moe.py:_resolve_nextn_quant_configcopy.copysquant_config(and its nested dict) before any remap, and only copies when a remap/exclude is actually needed — no shared-state mutation.python/sglang/srt/models/deepseek_nextn.py: docstring on the base_resolve_nextn_quant_configdocumenting the subclass override contract.test/registered/unit/models/test_nextn_quark_exclude.py(new, CPU CIbase-a-test-cpu): baseDeepseekV3ForCausalLMNextNpassthrough (non-quark /None), GLM-5.2 MTPexclude_layers+layer_quant_configremap, coarse fused-expert prefix, and the no-mutation guarantee.test/registered/amd/accuracy/mi35x/test_glm52_mxfp4_mtp_eval_mi35x.py(new, nightlynightly-amd-8-gpu-mi35x-glm52-mxfp4-mtp): 8-GPU GLM-5.2-MXFP4 EAGLE-MTP GSM8K eval — GPU coverage that the bf16 nextn layer loads under Quark MXFP4 and MTP accepts drafts. Wired intonightly-test-amd.ymlandnightly-test-amd-rocm720.yml.Accuracy Tests
CPU unit test
test/registered/unit/models/test_nextn_quark_exclude.pyon this branch:test_layer_quant_config_keys_remapped_without_mutationfails on currentmain(exercises the #30265 in-place mutation); green after this change.pre-commit runpasses all hooks (isort/ruff/black/yaml/duplicate-job-name/registry validation).Nightly GPU eval thresholds are the reported
amd/GLM-5.2-MXFP4MI355x numbers: GSM8K accuracy floor 0.935 (#29781 measured 0.950 at EAGLE 5/1/6, the fixture's config) and accept-length floor 5.0 (#30265 measured 5.94).Speed Benchmarks
N/A — quant-config resolution runs once at model construction; no inference-path change.
Checklist
base-a-test-cpu)CI States
Latest PR Test (Base): 🚫 Run #33544291185
Latest PR Test (Extra): ❌ Run #33544290924
Latest PR Test (AMD ROCm 7.2): ❌ Run #33544291139