Ollama vs vLLM vs llama.cpp: Tokens Per Second on the Same GPU

You have one GPU and a 7B model to serve. Three engines will do the job: Ollama, vLLM, and llama.cpp. They are not interchangeable once you push real traffic through them, and the gap between them is much wider than the “they all run GGUF” folklore suggests.

So we settled the Ollama vs vLLM vs llama.cpp question the only way that means anything: same model, same GPU, same prompt, three engines, measured in tokens per second. Every number below came off rented GPUs in September 2026, running Qwen2.5-7B-Instruct on three cards in turn, an RTX 4090, an L40S, and an RTX 5090. If you want to reproduce any of it, the vLLM production install, the llama.cpp build, and the Ollama install are each covered separately.

The test setup

Everything ran on a single dedicated GPU per box, rented one at a time on vast.ai so the three cards were the only variable. Running all three engines on the same physical GPU is the whole point. Split the engines across different hardware and the comparison is worthless.

The model is Qwen2.5-7B-Instruct. vLLM served the AWQ 4-bit build; llama.cpp and Ollama served the same GGUF Q4_K_M weights. Both are 4-bit, though not identical on disk: the GGUF lands near 4.7 GB, the AWQ build near 5.6 GB because AWQ keeps the embedding and output layers in fp16 over a 152K-token vocabulary. Close enough that the memory story is fair. Context was pinned to 4096 tokens per slot with 64 parallel slots on all three engines, so nobody got an unfair batching advantage. The prompt was a fixed 512-token instruction, temperature 0, capped at 256 output tokens.

These were the launch configs:

One detail that bit hard: Ollama sizes its default context from available VRAM, and on a 46 GB card that tier lands at a 32K window, which it then multiplies by the parallel-slot count. That balloons the KV cache well past the card’s VRAM, so Ollama silently spilled part of the model to system RAM and ran at a crawl. Pinning a derived model to num_ctx 4096 fixed it and put all 29 layers back on the GPU. That is the config used for every Ollama number here.

Three numbers were captured per engine: time to first token, single-stream decode rate with one request in flight, and aggregate throughput as concurrency climbed from 1 to 64.

Single-user speed is a tie between vLLM and llama.cpp

With one request at a time, vLLM and llama.cpp decode at the same rate on every card. This is expected. Single-stream decode is memory-bandwidth bound, and a 4-bit 7B model reads the same weights per token no matter which engine drives it. On the RTX 4090 both sat at roughly 174 tokens per second. The L40S, with less memory bandwidth, ran near 136. The RTX 5090 pulled ahead to about 250, which tracks its far higher bandwidth.

Ollama is the odd one out, and only on two of the three cards. On the RTX 4090 it matched llama.cpp almost exactly. On the L40S and the RTX 5090 it decoded at roughly half the rate of raw llama.cpp serving the identical GGUF. Hold that thought, because it turns into the most useful finding in this whole test.

Under concurrency, vLLM pulls away from everything

This is where the engines stop looking alike. Throw concurrent requests at the endpoint and vLLM’s continuous batching and paged attention do exactly what they were built for. At 64 concurrent requests vLLM delivered 2.8 to 4.4 times the aggregate throughput of llama.cpp on the same GPU.

The 64-concurrency numbers tell the story cleanly:

llama.cpp scales too, just not as steeply. Its aggregate throughput climbs roughly seven to fourteen times from one request to 64 depending on the card, which covers modest concurrency but never approaches vLLM’s curve. If you are building an endpoint that fans out to many callers, this table is the argument for vLLM by itself. For a Kubernetes deployment of exactly this pattern, the vLLM on Kubernetes guide picks up where this leaves off, and SGLang is worth a look if you want to push batching further.

Time to first token

llama.cpp answers fastest cold, around 8 to 12 milliseconds to the first token across all three cards. vLLM sits higher, in the 16 to 25 millisecond range, because its scheduler is optimized for batched steady state rather than a single lonely request. Ollama tracked llama.cpp closely on the 4090 but posted the highest first-token latency of any engine on the L40S and 5090, an early symptom of the runtime issue below.

Do not read too much into vLLM’s cold-start number. Under the concurrency it is designed for, that scheduling overhead is amortized across the batch and stops mattering. For an interactive single-user chat on a laptop-class GPU, though, llama.cpp feeling snappier off the mark is real.

Ollama ran at half the speed of llama.cpp on two of three cards

Ollama serves GGUF weights through its own bundled runtime, close kin to llama.cpp, so the reasonable assumption is that it performs like llama.cpp. On the RTX 4090 it did. On the L40S and the RTX 5090 it did not, and the gap was not subtle: Ollama decoded at roughly half the speed of a from-source llama.cpp build serving the exact same GGUF on the exact same GPU.

That claim is easy to hand-wave, so here is the receipt. On the L40S, upstream llama.cpp reported 143 tokens per second decode from llama-bench. Ollama, using its own telemetry on the same card and the same weights, reported 61. The GPU was pinned at its maximum clock with no throttling flagged.

We chased the obvious explanations and ruled them out one by one. It was not thermal throttling, since the card held its top clock the whole run. It was not the pinned context, since a rebooted L40S ran raw llama-bench at full speed straight after. It was not flash attention, since toggling it on and off in upstream llama.cpp moved the number by three percent, not by half. All three boxes ran the same Ollama version and loaded the same CUDA runner.

The one thing that tracked the split was the GPU driver. The fast 4090 sat on a newer driver branch than the two slow cards. Ollama ships and links its own compiled runtime rather than using the llama.cpp you build, so the same box running both can hand you very different numbers. We did not fully isolate the root cause, and that is worth stating plainly rather than pretending to a cleaner answer than the data supports. If you are chasing an odd Ollama slowdown, checking your NVIDIA driver and CUDA setup is a sane first move.

Ollama’s concurrency behaviour on those two cards was rough too. Aggregate throughput barely climbed with more requests and on the RTX 5090 it actually went backwards past 32 concurrent, so treat the concurrent Ollama figures on the L40S and 5090 as a ceiling, not a promise. The single takeaway: if you care about tokens per second, do not assume Ollama’s convenience is free. Point ollama serve at a busy endpoint and you may be running at half speed for reasons that have nothing to do with the card’s raw horsepower.

What to run, by workload

The right engine depends entirely on how many requests hit it at once.

For a serving endpoint with real concurrency, vLLM is not a close call. It delivered several times the throughput of the others at 64 requests on every card. Pay the extra first-token latency and the AWQ conversion step; you get it back many times over under load.

For a single user, a scripting pipeline, or an interactive session on one workstation GPU, llama.cpp is the pick. It matches vLLM’s single-stream decode, answers first-token fastest, uses the least VRAM of the three, and has no batching machinery to get in the way. It is also the honest speed baseline that Ollama should hit but did not on two of our three cards.

Ollama earns its place on convenience: pull a model with one command, swap models on the fly, get an API without touching a launch flag. That is genuinely worth something for local development and for running models like Kimi K3 or DeepSeek V4 Flash on your own box. Just benchmark it on your own hardware before you trust it in front of users, because on this test it left half the GPU on the table for reasons no dashboard would have told you about. Once it is serving, keep an eye on it with DCGM and Grafana so a regression like this shows up as a graph instead of a support ticket.