Repository navigation
ggml-cpu: re-hook llamafile MoE matmul on x86 (partial fix for #973) - #974
Conversation
Dense regression check ✅Sanity-bench on Llama-3.1-8B-Instruct-Q4_K_M (CPU-only, same Xeon Gold 6338 / Ice Lake-SP box, default
No regression. As expected since the hooks live in Test-plan item: |
Dense regression check — three-way: 0.9.3 vs 0.10.1 stock vs patchedUpdated table with stock 0.9.3 binary added (downloaded from the 0.9.3 release; same Xeon Gold 6338 / Ice Lake-SP box, same
Two things this tells us:
So the full picture of issue #973's CPU regression is two stacked bugs:
Test-plan item: |
Flash-attn investigation: confirmed second regression, fix recovers ~84%/82%Re-ran the same Qwen3-30B-A3B-Q4_K_M (MoE) and Llama-3.1-8B-Instruct-Q4_K_M (dense) benchmarks with Qwen3-30B-A3B-Q4_K_M (MoE, CPU-only, Xeon Gold 6338)
The two fixes are independent multipliers. Combined, they recover 84%/82% of 0.9.3 performance. Llama-3.1-8B-Instruct-Q4_K_M (dense, same box)
The MoE patch has zero effect on dense (as expected — hooks are perf-record on patched +
|
| Self % | Symbol | Bucket |
|---|---|---|
| 38.49% | mul_mat_qX_K_q8_K_T<DequantizerQ4K, 1> |
iqk MoE matmul |
| 23.74% | __kmp_wait_template |
OpenMP spin |
| 12.78% | mul_mat_qY_K_q8_K_T<DequantizerQ6K, 1> |
iqk MoE matmul |
| 11.26% | __kmp_hardware_timestamp |
OpenMP |
| 5.32% | tinyBLAS<...u16,float,float>::gemm<2,1> |
tinyBLAS |
| 2.45% | __kmp_wait_template (other variant) |
OpenMP |
| <1% | each: tinyBLAS float gemm, soft_max, barriers, … | misc |
ggml_compute_forward_flash_attn_ext_f16_one_chunk and ggml_vec_dot_f16 — the two hot symbols in the patched-but-fa=auto profile — drop out of the top entirely.
Bucketed comparison of the patched build with and without flash-attn:
| Bucket | + PR (fa=auto) | + PR (fa=off) |
|---|---|---|
| iqk MoE matmul kernels | 30.58% | 51.27% |
| Flash-attn f16 chunk + helpers | 28.23% | 0% |
| OpenMP overhead | ~28% | ~37% |
| tinyBLAS gemm | 0.54% | 6.14% |
The patched+-fa off profile is structurally identical to 0.9.3's (iqk matmul ~58%, threading ~29%, tinyBLAS ~6%). The remaining ~16% wall-clock gap is mostly OpenMP overhead being a few points higher than 0.9.3's pthread_pause_np/ggml_barrier threadpool.
Implications
- This PR (MoE matmul hooks) is necessary — without it, even
-fa offonly gets you 8.5%/34% (MoE). - A separate fix for the flash-attn f16 path is also needed to fully resolve Bug: CPU performance regression #973. Once that lands, full 0.9.3 throughput should be back.
- The two regressions are independent —
-fa offhelps even the unpatched build (8.47 → 10.98 t/s pp on MoE; 6.01 → ? on dense, not measured but expected similar).
Suggested options for the flash-attn follow-up (separate PR / issue):
- Short-term: change the in-tree default of
-fafromauto(which currently resolves to "on") toofffor the CPU backend insched_reserve. This is a one-line config change and would massively unblock CPU users today. Downside: may regress users who specifically wanted FA-on (e.g. for very long context). Could be conditioned on--gpu disable. - Long-term: write an AVX-512-VNNI-accelerated
ggml_vec_mad_f16/ fix thefp32 ↔ fp16inner loop, or land a llamafile-specific flash-attn implementation that uses iqk-style tiling.
I can open a follow-up issue tracking this next session if you like, or pause here.
Test-plan items: Sanity-bench a dense model — done (with three-way comparison plus the -fa off variant added). The new flash-attn finding is also captured in the updated PR description.
Correction & follow-up on flash-attn: 0.9.3 does support FA, but defaults OFFI claimed earlier "0.9.3 doesn't use this code path at all (predates upstream's flash-attn-ext)" — that was wrong. 0.9.3 has the So the flash-attn part of #973 is actually two compounding sub-issues. Confirmed by running both versions with FA explicitly enabled and disabled: Qwen3-30B-A3B-Q4_K_M (MoE)
Llama-3.1-8B-Instruct-Q4_K_M (dense)
So the picture is:
Implications for follow-upA clean fix for the flash-attn part of #973 likely requires both:
Should I open a separate issue for the flash-attn follow-up, or wait for maintainer input? |
aittalam
changed the title
The 0.10.0 "llamafile reloaded" refactor dropped the llamafile_mixmul and llamafile_mixmul_iqk hooks from ggml_compute_forward_mul_mat_id. Since then, every (expert × token × row) tuple in MoE matmul flows through type_traits_cpu[type].vec_dot (e.g. ggml_vec_dot_q4_K_q8_K, single-row AVX2 only), bypassing the batched/AVX-512-aware iqk kernels that classic llamafile used. On x86 this causes a 5-15× CPU-only regression on K-quant MoE models (Qwen3-30B-A3B-Q4_K_M reproduces it cleanly on both Alderlake AVX-VNNI and Ice Lake-SP AVX-512-VNNI). Dense matmul is unaffected because ggml_compute_forward_mul_mat retains its llamafile_sgemm hook. The patch re-introduces both 0.9.3 hooks on the new mul_mat_id: - a top-level llamafile_mixmul() early-out (handles f16/bf16/q4_0/q8_0) - an inner llamafile_mixmul_iqk() per-expert hook for the K-quant / Q8-vec-dot path (Q4_K, Q5_K, Q6_K, etc.), mirroring 0.9.3's ggml.c:11360. The submodule's bundled llamafile/sgemm.h only declares llamafile_sgemm, so the two functions are forward-declared inline in the patch to keep it self-contained. perf evidence on Xeon Gold 6338, Qwen3-30B-A3B-Q4_K_M: - ggml_vec_dot_q4_K_q8_K: 57.24% (unpatched) → <0.05% (patched) - mul_mat_qX_K_q8_K_T<Q4K,1> + Q6K variant: 8.74% → 30.58% - pp2048: 8.47 → 27.93 t/s (3.3× speedup, ~23% of 0.9.3 baseline) - tg32: 3.42 → 4.61 t/s (1.35× speedup, ~18% of 0.9.3 baseline) This PR is a partial fix for #973 — it addresses the MoE matmul leg only. A separate flash-attention regression (default flipped on->auto, and the FA implementation itself ~doubled in cost) accounts for the remaining gap and is tracked separately. Do NOT auto-close #973 on merge of this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
259b66f to
d6266ee
Compare
Threading investigation: OpenMP is NOT the bottleneck (residual ~16% gap is architectural)Goal: figure out why "PR +
The no-OpenMP build uses llama.cpp's own pthread-based threadpool fallback. Its perf profile (47,553 samples on the same generation workload):
OpenMP's Compare with 0.9.3 (which uses llamafile's classic threadpool):
That +10pp roughly matches the residual ~16% wall-clock gap. The cause is more barriers per token in the new llama.cpp — more model-architecture support means more graph nodes, each requiring a thread-sync barrier. Not OpenMP, not pthread overhead, not spin-wait policy. This is architectural (upstream llama.cpp's compute graph), not a llamafile-level bug. Plausibly fixable by:
Conclusion: OpenMP is NOT the bottleneck. The remaining ~16% gap is more sync points per token in the new graph, which is a deeper architectural concern. Won't address in this PR. Leaving OpenMP enabled (it's marginally better for |
Q8_0 MoE sanity-bench: outer
|
| Qwen3-30B-A3B-Q8_0 | flags | pp2048 (t/s) | tg32 (t/s) |
|---|---|---|---|
| stock 0.10.1 | default (fa=auto/on) | 19.85 ± 0.14 | 4.09 ± 0.07 |
| 0.10.1 + this PR | default (fa=auto/on) | 19.99 ± 0.28 | 4.12 ± 0.03 |
| stock 0.10.1 | -fa off |
42.01 ± 0.26 | 10.88 ± 0.13 |
| 0.10.1 + this PR | -fa off |
64.68 ± 0.52 | 10.89 ± 0.07 |
With -fa off (isolating the matmul effect from the FA regression tracked in #975): +54% pp improvement, tg unchanged (memory-bandwidth bound at Q8_0). With default FA on, the matmul advantage is fully masked by the FA bottleneck — same observation as for Q4_K_M (#974's other comment).
The +54% pp speedup on Q8_0 is smaller than the ~10× pp speedup we saw on Q4_K_M, but that's expected: Q8_0's per-row ggml_vec_dot_q8_0_q8_0 is already much faster than Q4_K's ggml_vec_dot_q4_K_q8_K (simpler kernel, less per-element work), so even the unpatched scalar fallback isn't as catastrophic. The patch still meaningfully helps via batched tinyBLAS dispatch.
perf-record confirms the outer hook fires (specifically, the path the inner Q4_K-style hook doesn't exercise)
30s capture on patched Q8_0 (fa=off), 47,533 samples. Top symbols:
| Self % | Symbol |
|---|---|
| 32.90% | tinyBLAS_Q0_AVX2<6, block_q8_0, block_q8_0, float>::gemm<2,1,1> |
| 25.04% | tinyBLAS_Q0_AVX2<0, block_q8_0, block_q8_0, float>::gemm<2,1,1> |
| 18.88% | __kmp_wait_template |
| 8.72% | __kmp_hardware_timestamp |
| 5.25% | ggml_vec_dot_q8_0_q8_0 (attention K×V residual) |
| 3.26% | tinyBLAS<...u16,float,float>::gemm<2,1> (f16 helper) |
| 1.68% | __kmp_wait_template (other variant) |
The 58% combined in tinyBLAS_Q0_AVX2<block_q8_0, block_q8_0, float> is the dispatch through llamafile_mixmul → MixMul → tinyBLAS_Q0_AVX2 — the path the top-level hook enables. Q4_K_M MoE took the inner llamafile_mixmul_iqk path instead (because Q4_K isn't in MixMul's type switch). So Q4_K_M and Q8_0 exercise different hooks; this bench confirms the outer one works end-to-end.
The two template specializations (<6, ...> and <0, ...>) reflect different chunk shapes selected by tinyBLAS based on matrix dimensions — both being hot confirms the dispatch is healthy.
Test-plan item: Sanity-bench a Q8_0 MoE — done. Updating PR body to check it off.
Q8_0 MoE — adding 0.9.3 baseline for three-way comparisonFollowing up on the Q8_0 bench above. Added 0.9.3 to make it a three-way comparison parallel to what we did on dense (Llama-3.1-8B).
Observations:
Reasonable to leave the ARM Mac test for a maintainer with the right hardware and mark this draft → ready for review. |
ARM Mac sanity ✅ — bench on Apple M4 Max (macOS 26.4.1, 16 cores)Got my hands on an M4 Max for the ARM check. cosmocc's APE binary runs natively (no rebuild), and the patch's hooks compile in for the aarch64 half too (verified earlier by Bench results (Qwen3-30B-A3B-Q4_K_M, CPU-only,
|
| build / flags | pp~2048 (t/s) | tg32 (t/s) | vs 0.9.3 default |
|---|---|---|---|
| 0.9.3 (default, n_threads=12) | 245.19 ± 0.16 | 53.30 ± 0.14 | — |
| stock 0.10.1 (default, n_threads=8) | 60.02 ± 0.09 | 29.09 ± 0.14 | 24% / 55% |
stock 0.10.1 (-t 12) |
87.02 ± 0.01 | 36.57 ± 0.21 | 36% / 69% |
| 0.10.1 + this PR (default, n_threads=8) | 160.71 ± 0.09 | 36.95 ± 0.33 | 66% / 69% |
0.10.1 + this PR (-t 12) |
229.05 ± 0.29 | 43.34 ± 0.50 | 93% / 81% |
Apples-to-apples patch effect at matched thread count (-t 12): +163% pp (2.63×), +19% tg. The patch is just as important on Apple Silicon as on Xeon. The hooks fire on ARM because cosmocc's aarch64 baseline doesn't set __ARM_FEATURE_SVE or __ARM_FEATURE_MATMUL_INT8, so the #if GGML_USE_LLAMAFILE guard stays defined.
Two side observations from the Mac data:
-
0.10.x's default thread count on Apple Silicon dropped from 12 → 8 between 0.9.3 (12 threads on M4 Max by default) and 0.10.1 (8 threads). This isn't directly caused by either the matmul or the FA regression in Bug: CPU performance regression #973/CPU flash-attn regression in 0.10.x (default flipped + implementation ~2× slower) #975 — it's a separate change in how upstream llama.cpp picks default threads. On a 12-perf-core + 4-efficiency-core M4 Max the right default is probably 12, not 8. Worth investigating in another follow-up. Setting
-t 12recovers a chunk of throughput on stock 0.10.1 too (60→87 pp). -
The patched build at
-t 12recovers 93% of 0.9.3 pp on Mac — better than the 84% we saw on Xeon. ARM doesn't suffer from the flash-attn regression here (because we explicitly passed-fa off); on default settings the Mac would also pay the FA cost, but the matmul fix is the bigger lever on either platform.
Smoke test
ssh studio # Apple M4 Max
~/llamafile-patched -m Qwen3-30B-A3B-Q4_K_M.gguf --gpu disable --server --port 8088 --host 127.0.0.1 -c 4096
# → answer to "Say hello" returned coherently via /v1/chat/completions (Qwen3 reasoning_content + content).
Bench script and protocol: tiny self-contained Python over the server's built-in timings.prompt_per_second / timings.predicted_per_second fields (auto-detects /v1/completions vs older /completion endpoint, cache-busts via per-run salt). Mirrors llama-benchy's pp~2048/tg32 protocol.
Test-plan item: ARM Mac sanity — done. Will tick it in the PR body next.
Clarification on Mac M4 Max numbers: thread-count is part of the gap@aittalam reviewing the Mac bench above made the right call to ask whether some of the Apple Silicon slowdown is just from fewer default threads (0.10.x defaults to 8 on M4 Max; 0.9.3 defaulted to 12). Math breakdown: Mac M4 Max, Qwen3-30B-A3B-Q4_K_M, CPU
So on Mac, this PR is still by far the biggest lever (~77% of the gap) but thread-count is a non-trivial ~15% and worth fixing independently. Linux Xeon is NOT affected. On x86 Linux, both 0.9.3 and 0.10.1 default to 16 threads (Xeon physical core count), so the Linux pp recovery numbers (84% with this PR + Root cause (on Apple Silicon)
0.9.3 had a llamafile-patched TrackingOpened #976 with full evidence + suggested fix. Trivial llamafile-side patch on Summary of issue #973's CPU regression on Apple Silicon (worst-case): three stacked, independent regressions
Each fix is small and independent. |
Raspberry Pi 5 sanity (4-core Cortex-A76, ASIMDDP, 4 GB)Tested on a Pi 5 with Used a minimal stand-in for
Observations:
So Pi 5 confirms: the patch is a no-op on dense ARM workloads — no regression, and no benefit expected. Good sanity check. |
Without this, MixMul::allocate_shared_memory() in tinyblas_cpu_mixmul.inc can fail (allocate() at line 372 returns nullptr when toto > params->wsize), making the outer llamafile_mixmul() hook in ggml_compute_forward_mul_mat_id a silent no-op. The hook was incidentally firing on Q8_0 MoE in our test (perf symbol tinyBLAS_Q0_AVX2<6, ...> = NCB|NCC = mixmul-instantiated) because the default 0.10.x work-buffer size happens to be large enough for that specific (model, prompt-shape, n_threads) tuple — but it's unreliable for other shapes. Mirror 0.9.3 ggml.c:17531, which took MAX(default, llamafile_mixmul_needs). This guarantees enough work-buffer is reserved for MixMul's shared allocations (quantized_thought, rowptr_result, rowptr_thought, rowptr_count). Identified by external code review of #974. Q4_K_M MoE (which goes through the inner llamafile_mixmul_iqk hook, not the outer one) is unaffected: pp=108.21 with fix vs 107.54 without, tg=16.56 vs 16.45 — within noise. Q8_0 MoE happens to be unchanged in our bench (pp=63.88 with fix vs 64.68 without, tg=10.81 vs 10.89) because the outer hook was already firing by coincidence. The fix makes the behavior deterministic across (model, prompt-shape, n_threads) combinations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addressed: missing
|
| metric | before fix | with fix | delta |
|---|---|---|---|
| Q4_K_M MoE pp2048 (-fa off) | 107.54 ± 0.31 | 108.21 ± 0.98 | +0.6% (noise) |
| Q4_K_M MoE tg32 (-fa off) | 16.45 ± 0.08 | 16.56 ± 0.11 | +0.7% (noise) |
| Q8_0 MoE pp2048 (-fa off) | 64.68 ± 0.52 | 63.88 ± 0.15 | −1.2% (noise) |
| Q8_0 MoE tg32 (-fa off) | 10.89 ± 0.07 | 10.81 ± 0.08 | −0.7% (noise) |
No perceptible perf change in either direction on our test box (Xeon Gold 6338) — confirming the outer hook was already firing for Q8_0 by coincidence, and Q4_K_M never used the outer hook anyway. The value of the fix is correctness for other (model, prompt, threads) combinations, not measurable perf gain on these specific runs.
Q8_0 MoE still produces coherent output (smoke-tested). make check still passes (re-run after this commit).
Bonus consequence
Without the fix, the outer hook is effectively non-deterministic — works for some workloads, silently falls through for others. With the fix, it's predictable. This also means F16 / BF16 / Q4_0 MoE models (the cases the outer hook handles that the inner doesn't) are now guaranteed to benefit from the patch, not just lucky.
Thanks for catching this — clearly worth fixing before merge.
Audit: surveyed all
|
| 0.9.3 [jart] piece | 0.10.x equivalent | Notes |
|---|---|---|
ggml_is_noop(op) skip in graph loop (ggml.c:15478, 17665) |
ggml_op_is_empty(op) in ggml-impl.h:90 + ggml-cpu.c:3040 |
Same op set (NONE, RESHAPE, TRANSPOSE, VIEW, PERMUTE). 0.10.x also adds TENSOR_FLAG_COMPUTE check and skips the trailing barrier — strict improvement. |
n_threads = n_tokens <= 2 ? n_threads : n_threads_batch (llama.cpp:15531, 15782) |
int n_threads = batched ? cparams.n_threads_batch : cparams.n_threads (llama-context.cpp:2175) |
batched set via ubatch.n_tokens > 1 at :1231. Slightly tighter threshold (>1 vs ≤2) but functionally equivalent for pp2048/tg32 workloads. |
llamafile_sgemm dense hooks (ggml.c:11081, 11142) |
ggml-cpu.c:1286, 1354 |
Already in current code, including the broadcast variant. |
GGML_MAX_NAME=128 for stable diffusion (ggml.h) |
Various places retain the larger NAME size | Need to verify still 128 not 64; quick spot-check shows 0.10.x uses GGML_MAX_NAME = 64 in ggml.h:212 — POTENTIAL LATENT ISSUE for SD users, but not part of #973 reproducer scope. |
3. Already tracked by sibling issues
- CPU flash-attn regression in 0.10.x (default flipped + implementation ~2× slower) #975 — flash-attn (default flipped + impl regressed)
- Apple Silicon CPU default thread count regressed 12 → 8 in 0.10.x (preprocessor vs runtime OS detection in upstream common.cpp) #976 — Apple Silicon CPU thread default (
hw.perflevel0.physicalcpu)
4. Lost / orphaned but NOT blocking the matmul fix
| Lost piece | Impact | Recommendation |
|---|---|---|
__target_clones("avx2") on ggml_compute_forward_out_prod_f32, set_f32, get_rows_back_*, diag_mask_f32, im2col_f32, conv_transpose_2d, ssm_scan_f32 |
Affects only ops used by training, vision, or SSM models. No-op for our LLM reproducer. | Restorable in a separate llamafile-files patch if SD / Mamba users hit slowdowns. Track separately. |
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS) + pthread_testcancel() per-op (ggml.c:17650, 17716) |
UX: lets a user interrupt mid-graph (mid-token). Without it, interrupts only land between tokens. Not a perf regression. | UX feature only; not blocking for #973. File as a separate UX/feature issue if needed. |
FLAG_trap SIGFPE trapping around SIMD vec ops (ggml-vector.inc:1935, llama.cpp:20188, 20200) |
Debugging convenience for finding NaN/Inf at the producing op. | Debug-only. Not blocking. |
llamafile_size(file) instead of stat for archive layouts (llama.cpp:3482, 19151, 19288) |
Affects when llamafile is loaded as part of a self-extracting .llamafile bundle. |
Probably handled differently in 0.10.x's loader path; needs spot-check. Not blocking #973. |
MIN(cpu_get_num_math(), 20) thread cap (common.cpp:290) |
Caps default thread count at 20 to avoid OpenMP overhead on huge machines. | Minor. Doesn't affect Xeon (16) or M4 Max (12). Could file as Apple Silicon thread default follow-up issue. |
// [jart] use two user prompts rather than merging system message (llama.cpp:19781) |
Chat template behavior tweak. | Chat-mode UX. Not blocking #973. |
Various FLAG_* binding in common.cpp (FLAG_threads, FLAG_seed, FLAG_prompt, etc.) |
Connects llama.cpp's argparse to llamafile's global FLAG_* — needed for llamafile-specific tools (CLI mode, chat mode) | Some bindings appear preserved (we see FLAG_* references in llamafile/); some not. Related to the auxiliary-binary link failures we noted earlier. Separate concern. |
5. Known auxiliary-binary build issues (not related to #973)
The build log shows undefined references in quantize/llama-bench/perplexity/imatrix/llava-quantize for FLAG_log_disable, FLAG_precise, llamafile_has, llamafile_help, llamafile_open_gguf, llamafile_unref, llamafile_check_cpu, tinylog. These auxiliary tools don't link llamafile.a properly. The main llamafile binary is unaffected and is what #973's reproducer exercises. Separate build hygiene issue.
Bottom line
Nothing in the audit looks like it would change this PR's matmul perf numbers materially. The three orphan jart pieces that DID directly affect #973's CPU regression are: matmul hooks (this PR), mixmul_needs (this PR, follow-up commit), and Apple Silicon thread default (#976). FA is separate (#975). All other [jart] losses are either preserved upstream under a different name, scoped to non-LLM ops, UX/debugging features, or auxiliary-binary build hygiene.
Ready to merge from my side.
…md_gemm) (#978) * llama-context: default -fa auto to off on CPU-only (partial fix for #975) Adds a llama.cpp patch making `-fa auto` resolve to OFF when no GPU devices are present (the CPU FA path is significantly slower than the non-FA path on x86; see #975 for benchmarks). Explicit `-fa on` / `-fa off` are unchanged, and the existing auto-resolution path still runs whenever a GPU backend is available (CUDA, ROCm, Metal, Vulkan). Recovers most of the user-visible CPU regression in 0.10.x that #974 left on the table. The second half of #975 — making the CPU FA kernel itself faster — is independent and will be a separate PR. * ggml-cpu: f32 VKQ accumulator in CPU flash-attn ext (partial fix for #975) When V is f16 and the hardware lacks native f16 FMA, the existing FA inner loop in ggml_compute_forward_flash_attn_ext_f16_one_chunk round-trips f16<->f32 every KV iteration via _mm512_cvtph_ps / _mm512_cvtps_ph (see simd-mappings.h GGML_F32Cx16_*) just to maintain the VKQ16 accumulator. perf-record on Xeon Gold 6338 (AVX-512F, no AVX-512-FP16) showed half the FA inner-loop work was these conversions and the convert+store extended the critical-path chain. This patch reorders the FA function to accumulate VKQ in f32 in that case, matching the existing non-f16-V branch. f16 V is dequantized once per KV iteration via the standard ggml_fp16_to_fp32_row helper, then ggml_vec_mad_f32 does an f32 FMA into VKQ32 with no convert-back. On hardware that has native f16 FMA (__AVX512FP16__, __ARM_FEATURE_FP16_VECTOR_ARITHMETIC, __riscv_zvfh) the existing f16 path is preserved -- changing it would lose half the SIMD lane count. Complements 43d62a6: that commit changes the default to -fa off on CPU-only; this commit makes the explicit -fa on path significantly faster on AVX-512F-without-FP16 / AVX2 / ARM-without-FP16-FMA. Both target #975. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * llamafile/fa: AVX-512F helpers for CPU flash-attention (partial fix for #975) Adds multi-arch helpers for the two hot per-element f16 ops in the CPU flash-attention inner loop, with runtime dispatch via the existing sgemm.cpp GemmFuncs struct. Routes ops.cpp's _one_chunk function to call them when V is f16, falling back to the upstream ggml helpers on false return. Why this matters ---------------- Cosmocc compiles llama.cpp's ggml-cpu/{ops,vec}.cpp with baseline x86_64 ISA so the APE binary stays portable across all CPUs. The result is AVX2-baseline codegen for ggml_vec_dot_f16 and ggml_fp16_to_fp32_row, while upstream's cmake -march=native build gets AVX-512. Vanilla llama.cpp at the *same submodule SHA* runs CPU flash-attention 4-8x faster per-token on the same Xeon Gold 6338 hardware because of this codegen difference, not any algorithmic difference. We can't add -mavx512f to the default-compiled ops.cpp/vec.cpp without breaking APE portability. Instead, this commit follows the iqk_mul_mat pattern: implement the hot helpers in llamafile/ with per-arch TARGET_ARCH flags (multi-arch fat-object), runtime-dispatched via sgemm.cpp's existing GemmFuncs struct based on X86_HAVE(AVX512F). Bench (Xeon Gold 6338, Qwen3-30B-A3B-Q4_K_M, --gpu disable, -fa on) ------------------------------------------------------------------- pp2048 tg32 before fa_helpers 27.91 7.08 after fa_helpers 28.82 18.20 <- decode beats upstream (17.34) upstream (cmake) 118.83 17.34 The tg gain comes from the patched _one_chunk function (decode path). pp2048 stays low because prompt processing uses ggml's tiled FA function which has its own per-element f16 loops not yet patched - that's a separate commit. Files ----- * llamafile/fa_helpers_amd_avx512f.cpp - AVX-512F implementation using _mm512_cvtph_ps + _mm512_fmadd_ps + _mm512_reduce_add_ps. Compiled with -Xx86_64-mavx512f -Xx86_64-mfma -Xx86_64-mf16c (per-target via cosmocc's -Xx86_64- prefix, doesn't leak to the aarch64 build). * llamafile/fa_helpers_unsupported.cpp - stubs returning false for CPUs without AVX-512F. * llamafile/sgemm.cpp - added fa_vec_dot_f16 + fa_fp16_to_fp32_row function pointers to GemmFuncs, initialized in the existing X86_HAVE(AVX512F) branch. * llamafile/sgemm.h - public API + per-arch declarations. * llamafile/BUILD.mk - TINYBLAS_CPU_FA_HELPERS_SRCS + AVX-512F build rule. * llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch - extended with extern "C" forward declarations and call sites in _one_chunk. CPUs without AVX-512F see no regression - they fall through to upstream's AVX2 helpers, which is what they already used. ARM82 / AVX2 variants of fa_helpers can be added in follow-up PRs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ggml-cpu: route tiled FA dequant through llamafile_fa_* (partial fix for #975) Extends llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch to cover the *tiled* flash-attention function (ggml_compute_forward_flash_attn_ext_tiled), which is the one prompt processing dispatches to. The previous commit only patched _one_chunk (used for decode), so tg recovered but pp2048 with -fa on stayed at ~28 t/s vs upstream's ~118. Two hot loops touched: * V-tile dequant (ops.cpp:8687-8693): direct substitution of ggml_fp16_to_fp32_row with llamafile_fa_fp16_to_fp32_row; falls back to upstream on false return. * K-tile transpose-and-dequant (ops.cpp:8622-8635): the destination is strided (K_f32[dk * KV_TILE_SZ + tk]), so we dequant the full DK row into a 4 KB stack scratch buffer with the SIMD helper, then do a scalar transpose-copy into K_f32. The transpose copy is cheap (DK loads + DK strided stores, all hot in L1). Stack buffer sized for DK <= 1024 covers all realistic attention head dims (Qwen3=128, DeepSeek-MLA=576). If the helper returns false or DK exceeds the scratch buffer, falls back to upstream's scalar table-lookup loop. CPUs without AVX-512F see no change (helper returns false → fallback fires → same code as before). APE binary remains portable; the AVX-512 instructions live only in llamafile/fa_helpers_amd_avx512f.o which is compiled with per-target TARGET_ARCH flags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tests: numerical-equivalence tests for llamafile_fa_* helpers (#975) Adds tests/fa_helpers_test.cpp comparing the new llamafile-side AVX-512 flash-attention helpers against upstream's ggml-cpu reference: - llamafile_fa_fp16_to_fp32_row vs ggml_fp16_to_fp32_row: asserts BIT-IDENTICAL output. Both use _mm*_cvtph_ps which is IEEE-754 deterministic per element, so equality should hold regardless of SIMD lane width. - llamafile_fa_vec_dot_f16 vs ggml_vec_dot_f16: asserts within ULP tolerance. The two implementations sum in different chunk widths (AVX2 8-wide vs AVX-512 16-wide), so reduction order differs and bit equality isn't guaranteed. Coverage: - Normal inputs (N(0, 0.1), typical post-LayerNorm activations). - Edge values (±0, ±INF, NaN, denormals, f16 min/max). - Adversarial alternating ±1 (worst case for reduction-order drift). - All zeros (must produce bit-exact 0.0f). - Dispatch wiring sanity check (output buffer must be untouched if the helper reports unsupported). Test sizes cover typical attention head dims (16, 32, 64, 80, 96, 128, 192, 256) plus odd tails (15, 17, 33, 129) and large (512, 1024). All inputs deterministic via fixed seeds. On CPUs without AVX-512F the helpers return false; the corresponding assertions are skipped and reported in the summary. The dispatch wiring test always runs. Wired into make check via tests/BUILD.mk. Test main() initializes ggml's f16<->f32 lookup table directly (skipping the graph-compute init dance) so the upstream reference works in a standalone test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * llamafile/fa: AVX-512 simd_gemm wrapper for tiled FA (closes #975 gap) Closes the remaining pp2048 gap for CPU flash-attention. The tiled FA function (ggml_compute_forward_flash_attn_ext_tiled) does two inline GEMMs per KV tile: KQ = Q @ K and VKQ += KQ @ V. Both go through ggml-cpu's static-inline simd_gemm in simd-gemm.h, which is compiled into ops.cpp.o at whatever ISA cosmocc's baseline gives (AVX2 with the GEMM_RM=6, GEMM_RN=2 tile config — 12 ymm-wide FMAs per inner iteration). Upstream's cmake -march=native build gets AVX-512 with GEMM_RM=4, GEMM_RN=4 (16 zmm-wide FMAs per inner iteration) which is ~2.7x peak FMA throughput on Skylake-X+ CPUs. Same fix shape as fa_helpers (#975 second commit). Adds: * llamafile/fa_simd_gemm_amd_avx512f.cpp — thin wrapper that includes simd-gemm.h and exposes llamafile_fa_simd_gemm_amd_avx512f. Compiled with -Xx86_64-mavx512f so the included static-inline simd_gemm picks the AVX-512 codegen path automatically. APE binary stays portable — AVX-512 instructions live only in this .o, routed at runtime only when X86_HAVE(AVX512F) is true. * llamafile/fa_helpers_unsupported.cpp — stub returning false. * llamafile/sgemm.h + sgemm.cpp — declarations + dispatcher entry. * llamafile/BUILD.mk — wires the new source + per-target AVX-512 flags. * llama.cpp.patches/patches/ggml_src_ggml-cpu_ops.cpp.patch — extended with two call sites in the tiled function (the KQ matmul and the VKQ accumulation), each with upstream fallback on false return. * tests/fa_helpers_test.cpp + tests/BUILD.mk — adds simd_gemm tests comparing the wrapper against a scalar reference on typical FA tile shapes (Q4 x DK128 x KV64 etc.), with ULP tolerance for FMA associativity. 3 new test functions; 9 total in the test binary. Bench (Xeon Gold 6338, Qwen3-30B-A3B-Q4_K_M, --gpu disable, -fa on) ------------------------------------------------------------------- pp2048 tg32 before this commit 29.28 17.97 after this commit 132.58 17.85 <- pp +4.5x, tg flat upstream llama.cpp 118.83 17.34 This PR (all 6 commits) now BEATS upstream on both pp (+11.6%) and tg (+2.9%) for CPU-only FA on this Xeon. The pp2048 gap relative to FA-off is also closed: -fa on 132.58 vs -fa off 116.56 — FA is genuinely a pp win, not just a tg win. Tests on Xeon: 11069 assertions pass (0 skipped); on Mac: 3 pass (73 skipped because Apple Silicon doesn't have AVX-512F, fallback to upstream's simd_gemm). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * tests/fa_helpers: address review feedback (3 fixes) 1. tests/fa_helpers_test.cpp:~159 — vec_dot_tol was multiplying by n twice, giving an O(n²·ε·max_abs) tolerance instead of the documented O(n·ε·max_abs) (Wilkinson-style summed-product bound). Fixed to `max(1e-5, 8 * n * eps * max_abs)`. Slightly tighter than before but still passes all 9 sub-tests on Xeon. 2. tests/fa_helpers_test.cpp:~270 — vec_dot_f16_adversarial_reduction_order was filling x and y with the same alternating-sign pattern, so every product was positive and the test didn't actually exercise the reduction-order sensitivity its name claimed. Fixed to use two different sign patterns: x[i] = (i & 1) ? -10 : +10 → [+, -, +, -, ...] y[i] = ((i / 2) & 1) ? -10 : +10 → [+, +, -, -, ...] so products alternate [+, -, -, +, +, -, -, +, ...], partial sums oscillate by ±100, and AVX2 (8-wide) vs AVX-512 (16-wide) reduction trees pair the cancelling terms in different orders — actually stressing the floating-point drift the test was supposed to cover. Tolerance adjusted to `max(1e-4, 4 * n * eps * scale²)`. Also drops the now-unused fill_adversarial_f16 helper. 3. llamafile/fa_simd_gemm_amd_avx512f.cpp:14-15 — header comment cited pre-patch line numbers for the simd_gemm call sites (8653 / 8724). Updated to current post-patch lines (8660 / 8740). Build clean and `make check` passes on Mac (3 PASSED, 73 skipped — no AVX-512F) and Xeon Gold 6338 (11069 PASSED, 0 skipped). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
The 0.10.0 "llamafile reloaded" refactor (#867) dropped the
llamafile_mixmulandllamafile_mixmul_iqkhooks fromggml_compute_forward_mul_mat_idin the llama.cpp submodule. Since then, MoE matmul on x86 falls through to single-rowggml_vec_dot_q4_K_q8_K(AVX2-only, no batching) instead of llamafile's batched, AVX-512-aware iqk kernels. This causes the 5–15× CPU-only regression on K-quant MoE models reported in #973 (Qwen3-30B-A3B-Q4_K_M reproduces cleanly on both Alderlake AVX-VNNI and Ice Lake-SP AVX-512-VNNI — it's an x86-wide bug, not AVX-VNNI-specific).This PR adds a new patch
llama.cpp.patches/patches/ggml_src_ggml-cpu_ggml-cpu.c.patchthat re-introduces both hooks (top-levelllamafile_mixmul+ innerllamafile_mixmul_iqk), mirroring the 0.9.3 shape. The fix applies cleanly to both the 0.10.1 submodule pin (5e9c63546) andmain's newer pin (7b8443ac7) — context at the patch sites is byte-identical.Evidence (Xeon Gold 6338, Ice Lake-SP, Qwen3-30B-A3B-Q4_K_M, CPU-only)
perf-record confirms the hook is firing:
ggml_vec_dot_q4_K_q8_K(57% of cycles unpatched) drops below the 0.05% threshold in the patched profile, whilemul_mat_qX_K_q8_K_T<DequantizerQ4K,1>(the iqk kernel) climbs from 5.89% to 23.34%.Why we don't recover the full 15× yet — confirmed: flash-attn
After the matmul fix, the new top symbol is
ggml_compute_forward_flash_attn_ext_f16_one_chunkat ~28% of cycles, withggml_compute_fp32_to_fp16andggml_vec_mad_f16as inlined hot children. 0.9.3 doesn't use this code path at all (predates upstream's flash-attn-ext). Confirmed via direct experiment (details): adding-fa offto the patched build recovers 84% of 0.9.3 pp and 82% of 0.9.3 tg on the Qwen3 MoE reproducer, and 83% / 85% on dense Llama-3.1-8B. The perf profile in that configuration is structurally identical to 0.9.3's.So the full picture of issue #973 is two stacked, independent regressions:
-fa offaloneThis PR fixes the MoE matmul leg. The flash-attn leg should be a separate PR — short-term option is flipping the default of
-fafromauto→offfor CPU-only inference; long-term option is fixing the f16 mad-loop or landing a llamafile-specific FA implementation.Full investigation write-up
Self-contained gist with the issue background, root cause analysis, perf-record evidence (including bucketed before/after tables for both kernels and threading), patch design notes, the second-regression finding, and follow-ups:
https://gist.github.com/aittalam/de0f8a494ff17d38c100f7ad08c0d7da
Refs #973 (partial fix — the matmul leg only; the flash-attn leg is being tracked separately and should NOT be auto-closed by this merge).
Test plan
.cosmocc/4.0.2/bin/make -j32clean build with patch applied.cosmocc/4.0.2/bin/make checkpasses (19/19extract_data_uristests)reasoning_content)ggml_vec_dot_q4_K_q8_Kdisappears from hot path and iqk kernels replace it-fa off, no tg change, no regression; perf confirms the top-levelllamafile_mixmul → MixMul → tinyBLAS_Q0_AVX2path is the one firing (different code path from the Q4_K_M inner-hook path). See Q8_0 bench comment.-fa off: smoke generation coherent, patch effect at matched threads (-t 12) is +163% pp / +19% tg, patched recovers 93% of 0.9.3 pp and 81% of tg. cosmocc's aarch64 baseline doesn't set__ARM_FEATURE_SVE/__ARM_FEATURE_MATMUL_INT8, so hooks fire on Apple Silicon too. See Mac M4 Max bench comment.All test-plan items checked. Ready for review when maintainers are ready.
🤖 Generated with Claude Code