Repository navigation
model: add Muse Spark Open (text, vision and audio) - #30616
Conversation
Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid routing with a correction bias. Attention interleaves three sliding layers at a 2048 window with rope for one global layer without rope, with a sigmoid output gate and a weightless QK-norm. Residuals are norm preserving, their gate is folded at conversion time. Vision reuses the Muse Glimmer tower. Audio is a causal convolutional pyramid with no attention and no spectrogram, four stages folding 16, 6, 5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz. The quantizer keeps the latent projections and the attention gate at Q8_0, a little over 1% of the weights, since the whole residual stream passes through them and the low-bit types turned the output into a repeating loop. The model saver writes the post norm epsilon, and the arch is registered in test-llama-archs. Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
github-actions
Bot
added
model
| cb(cur, "result_output", -1); | ||
| res->t_logits = cur; | ||
|
|
||
| ggml_build_forward_expand(gf, cur); |
There was a problem hiding this comment.
No longer needed to forward expand
aldehir
left a comment
aldehir
left a comment
There was a problem hiding this comment.
Is the output format significantly different from glimmer that it requires a new parser?
Yes, Spark adds a third recipient Glimmer does not have: to=commentary, user facing content the model emits mid turn, typically before or between tool calls, terminated by <|eom|>. Glimmer only has to=self for the chain of thought and to=user for the final answer, so its parser treats everything that is not the final message as reasoning. On Spark that would swallow commentary into reasoning_content instead of surfacing it as content. The two parsers are otherwise close, the diff is about forty lines: the commentary rule, the aside alternation that lets analysis and commentary interleave, and the trigger regex which has to exclude commentary as well. Happy to merge them into one parameterised parser if you prefer that over two files, the rest is identical. |
| # The residual gate is norm preserving: the stream is scaled by alpha and | ||
| # the branch by beta, with alpha^2 + beta^2 = 1. Both depend on the weight | ||
| # alone, so they resolve here and the graph keeps a scale and an add. |
|
Ah, looks like it's taking some ideas from gpt-oss.
It's not a big deal, we can keep them separate for now. |
## Summary Studio support for the Muse Spark Open GGUFs (`unsloth/Muse-Spark-Open-816B-A42B-GGUF`, llama.cpp arch `muse-spark`, loaded through the prebuilt llama.cpp that carries ggml-org/llama.cpp#30616). - **Sampling defaults.** A new `muse-spark` family uses the model card settings: temperature 1.0, top_p 1.0, top_k disabled, and min_p 0.0. The Hugging Face generation_config.json lists top_k 64, but the model card recommends top_k disabled and its benchmarks use that, so the card wins. Before this, every Muse Spark id fell through to `default.yaml` (0.7 / 0.95 / min_p 0.01). - top_k is stored as -1, Studio's "disabled" value. The llama-server payload sends it as 0, which llama.cpp reads as no top-k cutoff. - The pattern sits next to `muse-glimmer`, so Glimmer keeps its own 0.95 / 64. - **Reasoning effort.** The Muse Spark chat template accepts `reasoning_effort` low, medium, high or xhigh (default xhigh). Any other value hits `raise_exception`, and the render fails. - Before this change, the request builder forwarded the OpenAI `none` sentinel as-is, and the default-effort path forwarded `none`, `minimal` and `max`. All of them failed. - Muse Spark cannot turn reasoning off. So for the `muse-spark` architecture only, `none` and `minimal` now map to `low` and `max` maps to `xhigh`. Inkling's numeric mapping and every other architecture are unchanged. ## Already handled by main (verified, no change needed) - **Variant menu.** On the real repo listing, the menu shows exactly the eight quants: UD-IQ1_S, UD-IQ1_M, UD-IQ2_XXS, UD-IQ2_S, UD-IQ3_XXS, UD-Q2_K_XL, UD-Q3_K_XL and UD-Q4_K_XL. - Each quant resolves to shard 1 plus its remaining shards. - The root `imatrix_unsloth.gguf` is never offered as a variant and never picked as the model. - **Reasoning detection.** Detection publishes the four effort levels and tool support. - The template opens reasoning with `<|start|>assistant to=self<|message|>`, so the existing recipient-channel parser handles the output: `self` is reasoning, `user` is the reply. - **Memory planner.** Tested on the real UD-IQ1_S metadata (62 blocks: 16 full-attention and 46 sliding-window layers; 163.6 GiB of weights). The planner fits 134144 tokens of context on a single B200. Larger context requests and smaller cards spread across 2 or 3 GPUs without errors. ## Testing - **New tests:** `tests/test_muse_spark.py`, 38 tests. 12 fail on main and all pass on this branch. - Sampling for the repo, repo:quant, cache snapshot, local .gguf and org variants. - The variant menu and shard resolution for every quant. - The imatrix exclusion. - Reasoning detection. - Every reasoning kwarg Studio can produce, rendered against the template's effort handling. - **Full backend suite** (`studio/backend/tests`, `-n 24`): main vs this branch comparison of failing test IDs is in progress; I will update here. - **Live check:** Studio's planned argv for UD-IQ1_S (`-c 134144`, flash attention on), run on a ggml-org/llama.cpp#30616 llama-server build on two B200s. - Effort high: 511 tokens, reasoning split into `reasoning_content`, answer `17 x 23 = 391`. - Effort low: 114 tokens, answer `391`; the system prompt renders `Reasoning strength: 8.` - Rendering with `reasoning_effort: "none"` fails with HTTP 500 `Unsupported reasoning_effort`. That is the failure the coercion prevents.
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
|
most tests are passed, should we merge this now? |
|
Good to go on my side. The only red is gpu-vulkan-nvidia-cm2, which also fails on master at cd0a5b6 and is unrelated to this PR, nothing here touches Vulkan. |
Thanks, that one is mine. muse-spark-moe is the only model failing, on state_load and sometimes cp_h/cp_d, with NMSE 1.17e-05 and 1.15e-05 against the 1.0e-05 threshold, at a different step each run. CPU backend only, nothing GPU in either log. It is a tolerance overshoot rather than a broken restore, most likely top-8 routing flipping an expert on a near tie. I'll fix CI. |
Overview
Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid routing with a correction bias. Attention interleaves three sliding layers at a 2048 window with rope for one global layer without rope, which is where the 1M context comes from, plus a sigmoid output gate and a weightless QK-norm. Residuals are norm preserving, their gate is folded at conversion time.
Vision reuses the Muse Glimmer tower. Audio is a causal convolutional pyramid with no attention and no spectrogram, four stages folding 16, 6, 5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz.
The quantizer keeps the latent projections and the attention gate at Q8_0, a little over 1% of the weights, since the whole residual stream passes through them and the low-bit types turned the output into a repeating loop without it.
Additional information
Validated against the transformers reference at cosine 1.000000000 on the text path and 0.999999926 on the audio tower, on the real weights. Tested on a Mac Studio M3 Ultra with Metal and on an RTX PRO 6000: vision on two images, audio on a 17s clip, tool calling, and perplexity on wikitext-2 and on recent llama.cpp sources the weights cannot have seen.
GGUFs: ggml-org/Muse-Spark-Open-816B-A42B-GGUF, Q8_0, Q4_K_S and an imatrix-calibrated Q2_K, with the mmproj in F16 and Q8_0.
Requirements