Skip to content

Bug: CPU performance regression #973

Open
Open
Bug: CPU performance regression#973
Labels
bugmedium severity

Description

@Roysten
Roysten
opened on May 20, 2026

Contact Details

No response

What happened?

First of all thank you for this wonderful project.

I see wildly different performance between v0.9.3 and v0.10.1:

  • v0.9.3: 400 tokens predicted, 49 ms per token, 20.38 tokens per second
  • v0.10.1: 699 tokens / 2min 13s / 5.22 t/s

I start my model like this:
./llamafile-0.9.3 -m Qwen_Qwen3-30B-A3B-Q4_K_M.gguf --gpu disable
./llamafile-0.10.1-thin -m Qwen_Qwen3-30B-A3B-Q4_K_M.gguf --gpu disable

My CPU is: 12th Gen Intel Core i7-12700 (amd_avxvnni: AVX-VNNI).

Log are attached as .txt files because otherwise the maximum issue size was exceeded.

llamafile_logs_v0.9.3.txt
llamafile_logs_v0.10.1.txt

I report this for my Linux PC, but I can reproduce this also on a Windows system with a different CPU.

I wonder if the CPU improvements done by Justine are still being used: https://justine.lol/matmul/

Version

llamafile-0.10.1-thin

What operating system are you seeing the problem on?

Linux

Relevant log output

Activity
  1. Roysten
    added 
    bug
     
    medium severity
     on May 20, 2026
  2. aittalam commented on May 20, 2026 
    @aittalam
    aittalam
    on May 20, 2026
    Member
    Thank you @Roysten ! We tried to bring CPU optimizations back into the new llamafile, but (1) we have not been able to test them on many different CPUs and (2) they are now playing together with a different version of llama.cpp, so your feedback is very valuable.
    

    One thing I'd ask you, if that's ok, is to provide results from a benchmark so I can get a sense of how impactful the change is. I often use llama-benchy for that, which you can run as follows against a llamafile running in server mode:
    

    uvx llama-benchy --base-url http://localhost:8080 --model Qwen_Qwen3-30B-A3B-Q4_K_M --tokenizer Qwen/Qwen3-30B-A3B

    

    Another thing that might help understand the issue is whether you see the correct CPU architecture in the "compute" section when you run llamafile TUI (below the LLAMAFILE banner, under Software and Model).
    

    Thanks in advance for sharing! I'll try to delve deeper into this immediately but I'll surely have a better chance of success with this extra info 🙏
  3. Roysten commented on May 20, 2026 
    @Roysten
    Roysten
    on May 20, 2026
    Author
    Hi @aittalam, thank you for looking into it.
    

    CLI output v0.9.3:
    

    software: llamafile 0.9.3
model:    Qwen_Qwen3-30B-A3B-Q4_K_M.gguf
compute:  12th Gen Intel Core i7-12700 (alderlake)
server:   http://127.0.0.1:8080/

    

    CLI output v0.10.1:
    

    software: llamafile 0.10.1
model:    Qwen_Qwen3-30B-A3B-Q4_K_M.gguf
compute:  12th Gen Intel Core i7-12700 (amd_avxvnni: AVX-VNNI)
server:   http://127.0.0.1:8080

    

    I ran the benchmark command, but it only works for v0.10.1. For the v0.9.3 I get a HTTP 404 error in the warmup, it tries to access http://127.0.0.1:8080/chat/completions but it does exist.
    

    v0.10.1:
    

    [roysten@arch ~]$ uvx llama-benchy --base-url http://localhost:8080 --model Qwen_Qwen3-30B-A3B-Q4_K_M --tokenizer Qwen/Qwen3-30B-A3B
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
llama-benchy (0.3.7)
Date: 2026-05-20 18:23:30
Benchmarking model: Qwen_Qwen3-30B-A3B-Q4_K_M at http://localhost:8080
Concurrency levels: [1]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading text from cache: /home/roysten/.cache/llama-benchy/cc6a0b5782734ee3b9069aa3b64cc62c.txt
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (141518 > 131072). Running this sequence through the model will result in indexing errors
Total tokens available in text corpus: 141518
Warming up...
Warmup (User only) complete. Delta: 8 tokens (Server: 29, Local: 21)
Warmup (System+Empty) complete. Delta: 13 tokens (Server: 34, Local: 21)

Running coherence test...
Coherence test PASSED.
Measuring latency using mode: api...
Average latency (api): 0.58 ms
Running test: pp=2048, tg=32, depth=0, concurrency=1
  Run 1/3 (batch size 1)...
  No token_ids in response, using local tokenization
  Run 2/3 (batch size 1)...
  Run 3/3 (batch size 1)...
Printing results in MD format:



| model                     |   test |         t/s |    peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:--------------------------|-------:|------------:|------------:|-------------------:|-------------------:|-------------------:|
| Qwen_Qwen3-30B-A3B-Q4_K_M | pp2048 | 7.63 ± 0.02 |             | 268248.47 ± 763.04 | 268247.89 ± 763.04 | 269088.65 ± 779.54 |
| Qwen_Qwen3-30B-A3B-Q4_K_M |   tg32 | 2.34 ± 0.00 | 3.00 ± 0.00 |                    |                    |                    |

llama-benchy (0.3.7)
date: 2026-05-20 18:23:30 | latency mode: api

  4. aittalam commented on May 20, 2026 
    @aittalam
    aittalam
    on May 20, 2026
    Member
    My bad! I think the newer llama.cpp server serves both /chat/completions and /v1/chat/completions, while the old one looks like it only works with explicit /v1 prefix. I tried to run this
    

    uvx llama-benchy --base-url http://localhost:8080/v1 --model llama-3.1-8b-instruct --tokenizer meta-llama/Llama-3.1-8B-Instruct

    

    and it seems to be working for both (I just had another model available on my laptop 😅). These are the results I am getting on my Mac with --gpu disable:
    

    0.9.3:
    

    | model                 |   test |          t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:----------------------|-------:|-------------:|-------------:|-------------------:|-------------------:|-------------------:|
| llama-3.1-8b-instruct | pp2048 | 87.70 ± 4.25 |              | 23319.12 ± 1167.33 | 23318.20 ± 1167.33 | 23319.12 ± 1167.33 |
| llama-3.1-8b-instruct |   tg32 | 19.02 ± 0.57 | 20.67 ± 0.47 |                    |                    |                    |

    

    0.10.1
    

    | model                 |   test |          t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:----------------------|-------:|-------------:|-------------:|-------------------:|-------------------:|-------------------:|
| llama-3.1-8b-instruct | pp2048 | 59.10 ± 3.28 |              | 29084.73 ± 2366.15 | 29083.93 ± 2366.15 | 29084.73 ± 2366.15 |
| llama-3.1-8b-instruct |   tg32 | 18.01 ± 0.69 | 18.67 ± 0.47 |                    |                    |                    |

    

    So there definitely is some difference, especially in prompt parsing (pp2048), which we need to investigate. I saw the same even for GPU accelerated inference, so that might be a clue to a more general problem.
    

    Btw the first thing I noticed running this experiment is the following (left: 0.9.3, center: 0.10.0, right: 0.10.1):
    
Image

    i.e. 0.10.1 is not making the same use of the available cores. I tried to force a higher amount of threads with -t 12 and managed to make the pp2048 gap a bit closer
    

    0.10.1 with -t 12
    

    | model                 |   test |          t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:----------------------|-------:|-------------:|-------------:|------------------:|------------------:|------------------:|
| llama-3.1-8b-instruct | pp2048 | 69.37 ± 3.00 |              | 25908.09 ± 931.49 | 25907.28 ± 931.49 | 25908.09 ± 931.49 |
| llama-3.1-8b-instruct |   tg32 | 18.39 ± 0.58 | 19.00 ± 0.82 |                   |                   |                   |

    

    ... but I'll definitely look further into this!
  5. Roysten commented on May 20, 2026 
    @Roysten
    Roysten
    on May 20, 2026
    Author
    I modified the llamabency source code so it points to the v1 url, which gives the following results for v0.9.3:
    

    [roysten@arch llama_benchy]$ uvx llama-benchy --base-url http://localhost:8080 --model Qwen_Qwen3-30B-A3B-Q4_K_M --tokenizer Qwen/Qwen3-30B-A3B 
[transformers] PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
llama-benchy (0.3.7)
Date: 2026-05-20 20:15:36
Benchmarking model: Qwen_Qwen3-30B-A3B-Q4_K_M at http://localhost:8080
Concurrency levels: [1]
Loading text from cache: /home/roysten/.cache/llama-benchy/cc6a0b5782734ee3b9069aa3b64cc62c.txt
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (141518 > 131072). Running this sequence through the model will result in indexing errors
Total tokens available in text corpus: 141518
Warming up...
Warmup (User only) complete. Delta: 8 tokens (Server: 29, Local: 21)
Warmup (System+Empty) complete. Delta: 13 tokens (Server: 34, Local: 21)

Running coherence test...
Coherence test PASSED.
Measuring latency using mode: api...
Average latency (api): 16.87 ms
Running test: pp=2048, tg=32, depth=0, concurrency=1
  Run 1/3 (batch size 1)...
  No token_ids in response, using local tokenization
  Run 2/3 (batch size 1)...
  Run 3/3 (batch size 1)...
Printing results in MD format:



| model                     |   test |          t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:--------------------------|-------:|-------------:|-------------:|-------------------:|-------------------:|-------------------:|
| Qwen_Qwen3-30B-A3B-Q4_K_M | pp2048 | 95.23 ± 6.45 |              | 21533.76 ± 1394.68 | 21516.89 ± 1394.68 | 21533.76 ± 1394.68 |
| Qwen_Qwen3-30B-A3B-Q4_K_M |   tg32 | 17.85 ± 0.34 | 18.33 ± 0.47 |                    |                    |                    |

llama-benchy (0.3.7)
date: 2026-05-20 20:15:36 | latency mode: api

    

    I will try experimenting with the number of threads, to see if it makes any difference.
  6. aittalam commented on May 21, 2026 
    @aittalam
    aittalam
    on May 21, 2026
    Member
    Thank you! I noticed another thing from your very first post, that is instead of firing the alderlake intel optimized kernels 0.10.1 uses amd_avxvnni. This looks quite wrong and might be the reason for such a dramatic drop.
    
[update] alderlake was served by the amd_avxvnni kernel in 0.9.3 too, I'll look further into the issue
  7. aittalam
    added a commit that references this issue on May 22, 2026
    ggml-cpu: re-hook llamafile MoE matmul on x86 (partial fix for #973)
    d6266ee
  8. aittalam commented on May 22, 2026 
    @aittalam
    aittalam
    on May 22, 2026
    Member
    Tracking issue #975 opened for the flash-attention leg of this regression. PR #974 addresses the MoE matmul leg only; do not close this on its merge.
  9. aittalam commented on May 22, 2026 
    @aittalam
    aittalam
    on May 22, 2026
    Member
    Cross-link: also opened #976 for the Apple Silicon CPU thread-default regression (12→8 on M4 Max). That's a third compounding factor on Apple Silicon — independent from the matmul leg (#974) and the flash-attn leg (#975).
  10. aittalam
    added a commit that references this issue on May 22, 2026
    ggml-cpu: re-hook llamafile MoE matmul on x86 (partial fix for #973) (#…
    42a4006
  11. aittalam commented on May 22, 2026 
    @aittalam
    aittalam
    on May 22, 2026
    Member
    Hi @Roysten , sorry for the overload of info but I used Claude to help with this PR and it managed to be more verbose than me 😬
    
The good news is that I identified at least three possible issues to investigate and one of them already yielded a significant improvement (see #974). Another one relates to flash attention, which was off by default in 0.9.3 and is on in 0.10.*. I opened an issue for that and will tackle this next. Last but not least, the amount of threads used by default changed indeed on Mac, it looks like an easy fix but that shouldn't affect you.
    

    I chose to wait closing this issue until the FA part is fixed, or documented better. Thanks for opening this issue and helping make llamafile better! 🙏
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Metadata
Assignees
No one assigned
    Labels
    bugmedium severity
    Type
    No type
    Projects
    No projects
      Milestone
      No milestone
      Relationships
      None yet
      Development
      No branches or pull requests
      Issue actions