Ollama vs llama.cpp vs vLLM on one 16GB desktop card

I run open-source models on my desktop, a machine with an RTX 5070 Ti and 16GB of VRAM. Everyone starts with Ollama because it's one command. The enthusiasts swear by llama.cpp. The serious deployment people run vLLM. I knew the engines perform differently, but I wanted real numbers: same model, same GPU, how big is the gap actually?

So I spent a few evenings benchmarking all three. Same prompts, same harness, same machine, one engine on the GPU at a time. I started with two models and realised partway through that the lineup was missing a third case, so I added it later. Glad I did, because it changed the conclusion.

The contenders

• Ollama is the one everyone starts with. One command and you're chatting. It was already running on my box as the backend for Open WebUI, so it represents "whatever the defaults give you". Underneath, it wraps llama.cpp.
• llama.cpp is that engine itself, but taken straight from the source: current master, compiled on my machine, with CUDA kernels tuned for this exact GPU generation. Comparing it with Ollama tells you what the convenience layer costs.
• vLLM is the server datacenters run. It's built around one idea, serving lots of requests at once, with clever memory management for the attention cache and a scheduler that keeps the GPU busy. The question was whether any of that pays off on a consumer card.

And the models are the real trick. Three of them, covering the three situations you can be in with a 16GB card:

• gpt-oss-20b, about 13GB at 4-bit. Fits, but snugly.
• Qwen3-Coder-30B, about 18.6GB at 4-bit. Doesn't fit at all.
• Gemma 4 12B, about 8GB at 4-bit. Fits with room to spare. This is the one I added later.

One honest detail before the numbers. My desktop session stayed running the whole time, which eats about 1GB of VRAM. That's the box as I actually use it, and that 1GB ends up playing a real part in the story.

Round one: the model that fits, snugly

Everything runs. Nobody crashes. And yet:

Same weights, same quantization, three wildly different outcomes. And yes, the datacenter engine really did 33 tokens a second. Seven times behind llama.cpp, on identical weights.

Here's what happened. vLLM couldn't load the model at all at first. Three out-of-memory crashes, because 13.8GB of weights plus its working space didn't fit next to my desktop's 1GB. The only way to get it running was telling it to park 3GB of the weights in system RAM. And the catch is that vLLM still does all its math on the GPU, so those parked gigabytes travel across the PCIe bus for every single token. VRAM moves data at roughly 900GB a second. PCIe moves it at 64. Generating tokens is basically a memory bandwidth game, and vLLM was forced to play it across the slow bus.

Hold that thought, because round three shows what this engine does when nothing is choking it.

llama.cpp beating Ollama by 36%, despite Ollama wrapping the same engine? Two boring reasons: a current build with kernels compiled for this exact GPU generation, and explicitly configured settings instead of shipped defaults. The convenience layer costs about a third of your single-user speed on this model. Now it's a measured number instead of a rumour.

Round two: the model that doesn't fit

An 18.6GB model on a 16GB card means something has to give, and each engine gives differently.

llama.cpp wins the single-user case again, and the way it does it is genuinely clever. Qwen3-Coder is a mixture-of-experts model. Every layer has 128 small expert networks, and each token only wakes up 8 of them. llama.cpp has a flag built for exactly this shape, --n-cpu-moe. Instead of streaming weights to the GPU, it sends the work to the weights: the expert layers live in system RAM and the CPU computes them right there, while the bandwidth-hungry part, attention and the cache, stays on the GPU. Since each token only wakes a few experts, the CPU barely works. An 18GB model doing 71 tokens a second on a 16GB card is, honestly, amazing.

Ollama handled the oversized model without being asked, it auto-splits between GPU and CPU. 67 tokens a second, perfectly usable. vLLM's generic offload managed 59 single-stream but again won time-to-first-token, prefill, and the 8-client column. A hint of things to come.

Round three: the model that fits comfortably, and the story flips

Gemma 4 12B leaves 6 or 7GB of headroom. For the first time, vLLM loaded natively: no offload, no compromises, all its machinery switched on.

Single user, the three are close: 91, 87, 81. You'd struggle to feel the difference in a chat window. But look at the 8-client column. vLLM served 419 tokens a second, nearly double llama.cpp and more than five times Ollama. The same burst of 8 requests took vLLM 4.9 seconds, llama.cpp 8.8, and Ollama 26.3. Sit with that spread for a second: same GPU, same model, and one engine returns the whole batch five times sooner than another.

That's what vLLM is for. Its poor showing in round one wasn't the engine, it was the offload. Give it resident weights and parallel traffic, and you can see exactly why datacenters run it.

The Ollama result deserves its own section

Across all three rounds, one pattern repeated: 8 clients got the same total throughput as 1. Out of the box, Ollama serves requests one at a time. It's not broken, it's a default, and it's fixable with a single environment variable, OLLAMA_NUM_PARALLEL. But I suspect almost nobody sets it, which means almost every multi-user Ollama box out there is quietly serving people from a queue. If your Ollama serves more than one person, or one person who fires parallel requests from scripts, set it today. It's the cheapest performance win in this whole post.

So which engine?

That's the wrong question, and getting the wrong question out of the way was the real result of these evenings. The right question is: does your model fit, and how many people are hitting it?

• Fits, one user: llama.cpp. Fastest at everything, and the gap is biggest exactly when the card is under pressure.
• Fits, many users or batch jobs: vLLM, and it's not close. The 419 vs 232 column is the whole argument.
• Too big, and it's a mixture-of-experts model: llama.cpp's expert offload. Nothing else is in the same neighbourhood, 71 against 33.5 on my hardest comparison.
• Want it running in five minutes: Ollama. Genuinely great at what it's for. Just set that env var.

On my own box, llama.cpp is now the serving engine behind everything single-user, Ollama stays as the quick way to pull and try models, and vLLM earned the job I originally doubted it could do here: batch work. My model eval suite fires eight requests at a time, which is exactly the traffic vLLM eats for breakfast.

Fine print

Same machine throughout: RTX 5070 Ti 16GB, Ryzen 7 9700X, 128GB DDR5, one engine on the GPU at a time, temperature 0, streamed responses, every prompt uniquely tagged so prefix caching couldn't cheat on prefill. Cross-engine model files aren't byte-identical where noted: the Qwen round compares Q4_K_M against AWQ, and vLLM's Gemma checkpoint is Google's official quantized release at 10.3GB against the 7.4GB file the others served, which explains part of its single-stream gap, decode speed follows bytes. vLLM ran handicapped in rounds one and two (eager mode, short context, forced by the memory squeeze) and unhandicapped in round three. And a headless box would hand every engine back my desktop's 1GB, which would likely have turned round one's "can't load" into "loads, barely". If you want the deeper writeup on how GGUF, expert offload, and paged attention actually work under the hood, tell me, that's a post of its own.