If you want to run a capable large language model on your own laptop, workstation, or server – with no API bills, no rate limits, and no data leaving your machine – llama.cpp is the tool that makes it practical. It is the open-source C/C++ inference engine that quietly powers most of the local-AI ecosystem: tools like Ollama (whose 0.30 release, out in October 2026 per Ollama’s own release notes, tightened GGUF compatibility by tracking llama.cpp’s engine more closely) and LM Studio are built on top of its ggml tensor library. This llama.cpp tutorial walks you through 12 hands-on steps – from cloning the repository and building from source, through CPU and GPU acceleration, to launching an OpenAI-compatible API server – so you can run an LLM locally in about 40 minutes.
llama.cpp has more than 118,000 GitHub stars and over 20,000 forks on GitHub, and is released under the permissive MIT license. For most of its history the project skipped traditional semantic versioning in favor of continuous build-tagged releases, from a contributor base that separate 2026 tracking puts past 900 contributors with more than 2,600 tagged releases; that changed on August 25, 2026, when GitHub began marking a numbered v0.3.0 release as the repository’s official “Latest” tag – a milestone the project has kept building on, with v0.4.0 landing on September 4, 2026 and v0.4.1 following as the new “Latest” release on September 14, 2026, adding support for the Maple 20B-A1B, Tencent Hy 4, and Spark2.5 architectures. Development has not slowed heading into the back half of 2026, either: an August 27, 2026 commit added new --video-* CLI arguments, and v0.4.1 shipped a breaking CLI change that removes the deprecated --mmap, --mlock, and --direct-io flags in favor of a single --load-mode flag (see Step 6), keeping that near-daily pace of backend and CLI changes going: v0.6.0 followed in October 2026 with an extended batch API and support for the GLM-5.3-Flash architecture, and v0.7.0, tagged shortly after and also in October 2026, is the newest release ggml-org lists as of this writing. By the end of this tutorial you will have built llama.cpp from source, downloaded a quantized GGUF model, run interactive chat from the command line, exposed a local API server, converted and quantized your own model, and benchmarked throughput. Every command below was verified against the official ggml-org/llama.cpp repository and rechecked in October 2026.
Quick answer: clone ggml-org/llama.cpp, build it with CMake (cmake -B build, then cmake --build build --config Release -j), download a GGUF model from Hugging Face, then run llama-cli for chat or llama-server for an OpenAI-compatible API – or skip straight to a model with the README’s own shortcut, llama cli -hf <repo> or llama serve -hf <repo>. About 40 minutes start to finish. The full path, step by step:
- Clone the llama.cpp repository
- Build llama.cpp from source with CMake (CPU)
- Enable GPU acceleration (CUDA, Metal, Vulkan, ROCm)
- Download a GGUF model
- Run your first inference with llama-cli
- Tune performance: GPU layers, context, and threads
- Start the OpenAI-compatible server (llama-server)
- Call the API from curl and Python
- Convert a Hugging Face model to GGUF
- Quantize a model with llama-quantize
- Benchmark throughput with llama-bench
- Run llama.cpp as a background service
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Is llama.cpp and Why Run an LLM Locally?
llama.cpp is a pure C/C++ implementation of large language model inference, created by Georgi Gerganov in March 2023 and now maintained by the ggml-org community. Its GitHub description is deliberately simple – “LLM inference in C/C++” – but that simplicity is the point. Because it has no heavyweight runtime dependencies like PyTorch or CUDA developer toolkits baked into the inference path, llama.cpp compiles to a handful of small native binaries that run almost anywhere: Linux, macOS, Windows, Raspberry Pi, Android, and even inside a browser via WebGPU.
The engine reads models in the GGUF file format (the successor to the older GGML format). GGUF packs the model weights, tokenizer, and metadata into a single portable file, and supports aggressive quantization – shrinking 16-bit weights down to 8, 6, 5, 4, 3, or even 2 bits so a model that needs a data-center GPU at full precision can fit on a gaming card or run on CPU RAM. That quantization story is exactly why so many people now run an LLM locally instead of paying per token in the cloud.
There are four practical reasons to choose a local llama.cpp setup over a hosted API:
- Privacy: prompts and documents never leave your hardware – critical for legal, medical, or proprietary code.
- Cost: after the one-time hardware investment, inference is free. No per-token charges, no surprise bills.
- Offline & control: it works on a plane or an air-gapped network, and you pin the exact model weights you tested against.
- Speed of iteration: no network round-trip, no API rate limits, and full access to low-level sampling and KV-cache settings.
That same privacy cuts both ways, and it is worth knowing the trade-off. Because a local model never phones home, there is no API-provider log of what was asked or generated – good for a compliance team, but it also means abuse is harder for defenders to spot. A security update from ExplainX on August 11, 2026 flagged this exact dynamic, noting that the North Korea-linked threat group Kimsuky has been observed running LLMs offline on its own servers, precisely to avoid the network-level visibility a hosted API call would create. It is a useful reminder that local inference is a neutral capability: the same offline operation that keeps a hospital’s patient records from leaving the building also removes a monitoring point that security teams have come to rely on.
It helps to understand where llama.cpp sits relative to the other popular local-LLM tools. The table below compares the three engines most US developers evaluate in 2026. If you want a deeper head-to-head on the two most common choices, see our dedicated vLLM vs Ollama breakdown.
| Engine | Language | Best for | Hardware focus | Model format | License |
|---|---|---|---|---|---|
| llama.cpp | C/C++ | Local/edge, CPU + consumer GPU, maximum portability | CPU, NVIDIA, Apple, AMD, Intel | GGUF (quantized) | MIT |
| Ollama | Go (wraps ggml/llama.cpp) | Beginner-friendly model management | CPU + GPU | GGUF (via llama.cpp) | MIT |
| vLLM | Python / PyTorch | High-throughput server clusters | NVIDIA/AMD data-center GPUs | Safetensors, AWQ, GPTQ | Apache-2.0 |
Prerequisites and System Requirements
Before you build llama.cpp, make sure your machine has the toolchain below. The project is intentionally lean, so the requirements are modest – you do not need a GPU at all to follow this llama.cpp tutorial, though one will dramatically speed up inference.
- Git – to clone the repository.
- CMake 3.14 or newer – llama.cpp’s only supported build system (the legacy Makefile path was retired in 2024).
- A C++17 compiler – GCC, Clang, or MSVC (Build Tools for Visual Studio on Windows).
- Python 3.9+ – only needed for the optional model-conversion steps (Steps 9–10).
- Optional GPU SDKs: NVIDIA CUDA Toolkit for CUDA builds, the Xcode command-line tools for Apple Metal, the Vulkan SDK for cross-vendor GPUs, or ROCm/HIP for AMD cards.
On Ubuntu 24.04 – the environment most 2026 self-hosting guides standardize on – a single apt command installs that entire toolchain in one pass:
# Ubuntu 24.04: Git, a C++17 toolchain, CMake, and Python in one command
sudo apt update
sudo apt install -y build-essential cmake git python3 python3-pipThat covers the CPU build path. If the same machine has an NVIDIA GPU, install the CUDA Toolkit separately (NVIDIA’s .deb repository or runfile installer) before moving on to Step 3, then build with -DGGML_CUDA=ON.
RAM and VRAM are the variables that decide which models you can actually run. A quantized model needs roughly its file size in memory, plus a few hundred megabytes to a couple of gigabytes for the context (KV cache). Use the table below as a planning guide – the numbers are approximate for the popular Q4_K_M quantization and will vary by architecture and context length.
| Model size | Q4_K_M file size (approx.) | Recommended VRAM | CPU-only RAM | Typical use case |
|---|---|---|---|---|
| 1–3B | 1–2.5 GB | 4 GB GPU | 8 GB | Edge devices, autocomplete, classification |
| 7–8B | ~4.5–5 GB | 6–8 GB GPU | 16 GB | General chat, summarization, coding helper |
| 13–14B | ~8–9 GB | 10–12 GB GPU | 16–32 GB | Stronger reasoning, longer context |
| 30–34B | ~18–20 GB | 24 GB GPU | 32–64 GB | High-quality assistant, RAG backends |
| 70B | ~40–43 GB | 2× 24 GB GPU | 64 GB+ | Near-frontier quality, offline workstations |
If you plan to run bigger models on a GPU, the card’s VRAM is the hard limit. Our RTX 5090 vs 4090 comparison is a useful reference for picking consumer hardware that maximizes the VRAM you can dedicate to local inference.
Step 1 – Clone the llama.cpp Repository
Everything starts with the source. Clone the official repository from the ggml-org organization (note: the project moved from the old ggerganov/llama.cpp path to ggml-org/llama.cpp, though the old URL still redirects). Current 2026 setup guides recommend a shallow clone with --depth 1, which pulls only the latest snapshot instead of the full commit history – noticeably faster since this tutorial only needs the current source, not years of past commits.
# Clone the repo and enter the directory (--depth 1 = fast, shallow clone)
git clone --depth 1 https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Optional: check the latest build tag you're on
git log -1 --onelineThe clone is a few hundred megabytes because it includes the bundled ggml library and example assets. You do not need to download any model yet – that happens in Step 4. If you would rather skip compilation entirely, the project also publishes pre-built binaries on its releases page for Windows, macOS, and Linux. As of October 2026, the GitHub project page documents an even broader set of install paths beyond source and pre-built binaries: Homebrew on macOS and Linux, Winget on Windows, Nix for reproducible installs, and the official Docker images covered in Step 12. Building from source still gives you the best hardware-specific performance tuned to your exact CPU and GPU, which is why this tutorial follows that path, but any of those package-manager routes will get the prebuilt llama-cli and llama-server binaries running in under a minute if you just want to try llama.cpp first.
Step 2 – Build llama.cpp from Source with CMake (CPU)
llama.cpp uses CMake exclusively. The two-command pattern below configures a build directory and compiles an optimized CPU build. This works on Linux, macOS, and Windows alike.
# Configure the build (creates a ./build directory)
cmake -B build
# Compile in Release mode, using all CPU cores (-j)
cmake --build build --config Release -j
# Same thing, written the way most 2026 Ubuntu 24.04 guides show it
cmake --build build --config Release -j$(nproc)The --config Release flag matters: a Debug build can be many times slower at inference. The trailing -j tells the build to use every available CPU core; current 2026 guides on Ubuntu 24.04 often spell this out explicitly as -j$(nproc), which is functionally identical to the plain -j used throughout this tutorial but pins the count to the nproc command’s output. When the build finishes, the binaries land in build/bin/. You should see the core tools listed below.
$ ls build/bin/
llama-cli llama-server llama-quantize
llama-completion llama-bench llama-perplexity
llama-simple llama-tokenize llama-embedding
...These renamed binaries are the modern interface. Older tutorials reference main and server – those were renamed to llama-cli and llama-server back in 2024, so if a guide still tells you to run ./main, it is out of date. llama-cli changed again in July 2026, when it was rebuilt into a thin client that launches llama-server in the background instead of running inference in the same process; the previous standalone, direct-inference binary was renamed llama-completion. Every llama-cli command in this tutorial still works exactly as shown, and if you specifically want that old no-server behavior – handy in a minimal script or container – llama-completion accepts the same flags. The most important tools for this llama.cpp tutorial are llama-cli (interactive inference), llama-server (the API server), llama-quantize (compression), and llama-bench (benchmarking).
Step 3 – Enable GPU Acceleration (CUDA, Metal, Vulkan, ROCm)
A CPU build works, but offloading layers to a GPU can increase throughput by an order of magnitude. To enable a backend you re-run the CMake configure step with a single flag, then rebuild. Pick the one that matches your hardware. This CPU (AVX2/AVX512), NVIDIA CUDA, AMD ROCm/HIP, and Apple Silicon (Metal) spread is still exactly the multi-backend matrix that 2026 llama.cpp tutorials converge on – there is no single “preferred” backend replacing this lineup, just the one that matches the hardware in front of you. A 2026 Ubuntu/GGUF setup guide gives the canonical CUDA path with the job count spelled out explicitly – cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc) – which is why the NVIDIA command below pins $(nproc) instead of a bare -j.
# NVIDIA (requires the CUDA Toolkit installed)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
# AMD (ROCm / HIP) – target your GPU architecture
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1030
cmake --build build --config Release -j
# Cross-vendor GPUs via Vulkan
cmake -B build -DGGML_VULKAN=ON
cmake --build build --config Release -jOn Apple Silicon there is nothing to configure: Metal is enabled by default on macOS, so the plain CPU build from Step 2 already uses the GPU and the unified memory architecture. (You would only pass -DGGML_METAL=OFF to explicitly disable it.) The table below summarizes the backend flags so you can match them to your machine.
| Hardware | Backend | CMake flag | Notes |
|---|---|---|---|
| NVIDIA GPU | CUDA | -DGGML_CUDA=ON | Requires CUDA Toolkit; fastest path on NVIDIA |
| Apple Silicon | Metal | (default on macOS) | No flag needed; uses unified memory |
| AMD GPU | HIP / ROCm | -DGGML_HIP=ON | Set -DGPU_TARGETS to your gfx arch |
| Any GPU | Vulkan | -DGGML_VULKAN=ON | Cross-vendor; great for mixed/old GPUs |
| Intel GPU | SYCL | (see SYCL docs) | For Intel Arc and integrated graphics |
| CPU only | AVX2/AVX512 | (default) | Auto-detected; no GPU required |
After a GPU rebuild, the same build/bin/ binaries are now GPU-aware. You activate offloading at runtime with the -ngl flag (covered in Step 6), not at build time – so one GPU-enabled build can still run purely on CPU if you set -ngl 0.
Step 4 – Download a GGUF Model
llama.cpp only runs GGUF files. The fastest way to get one is the built-in -hf flag, which downloads a model directly from the Hugging Face Hub. You do not need a separate download step or a Hugging Face token for public models. -hf downloads are now cached in the standard Hugging Face cache directory by default, the same location the huggingface_hub Python library and other Hugging Face-based tools use – so a model you already pulled for another app is reused instead of downloaded twice, and models you fetch with llama.cpp are visible to those other tools too.
# Pull a small instruct model straight from Hugging Face and chat
./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF
# Or pick a specific quantization from a repo that offers several
./build/bin/llama-cli -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_MAs of October 2026, the official README still leads with an even shorter form of this same command: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF for chat, and llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF to launch the API server covered in Step 7. That space-separated llama cli / llama serve naming is the project’s current documented entry point for the same -hf download-and-run flow used throughout this tutorial – if you spot the no-hyphen form in the README or a newer guide, it is the same tool, not a separate install.
You can also download a .gguf file manually from a model page on Hugging Face’s GGUF library and point llama.cpp at it with -m. Popular community quantizers such as bartowski, unsloth, and ggml-org publish ready-made GGUF files for almost every open model – Llama 3, Qwen, Gemma, Mistral, Mixtral, Phi, and DeepSeek among more than 50 supported architectures, with that coverage still being refined architecture by architecture: commit b10750 added recurrent-state rollback for Qwen4 speculative decoding back in August 2026.
Understanding Quantization Types (Q4_K_M, Q8_0, and friends)
The cryptic suffix on a GGUF filename tells you how heavily the weights were compressed. Lower bit-widths mean smaller files and faster inference but slightly lower quality. For most people, Q4_K_M is the sweet spot – it is the default recommendation in the community because it keeps almost all of the model’s quality at roughly a quarter of the full-precision size. You’ll see these tags in lowercase (q4_k_m, q5_k_m, q8_0) in scripts and config files and in uppercase (Q4_K_M) in filenames and docs – they refer to the same quant, and 2026 tooling still covers the same core ladder from q2_k up to q8_0. Within several of these families an _S (small) or _M (medium) suffix marks a smaller- or larger-footprint sub-variant at the same bit-width, such as Q3_K_S alongside the Q3_K_M shown below, so do not be surprised to see more than one option at the same nominal bit count.
| Quant | Bits (approx.) | 8B model size | Quality | When to use |
|---|---|---|---|---|
| F16 / BF16 | 16 | ~16 GB | Reference | Conversion source; max fidelity |
| Q8_0 | 8 | ~8.5 GB | Near-lossless | When VRAM is plentiful |
| Q6_K | 6.6 | ~6.6 GB | Excellent | High quality, moderate size |
| Q5_K_M | 5.5 | ~5.7 GB | Very good | Quality-leaning balance |
| Q4_K_M | 4.5 | ~4.9 GB | Great | Recommended default |
| Q3_K_M | 3.9 | ~4.0 GB | Usable | Tight VRAM budgets |
| Q2_K / IQ2 | 2–3 | ~3.2 GB | Degraded | Last resort to fit a model |
Where to Find Trustworthy GGUF Models
Because anyone can upload a GGUF file, it pays to know where the well-maintained ones live. The Hugging Face Hub is the central marketplace, and a handful of prolific quantizers have earned the community’s trust by publishing consistent, correctly templated files within hours of a new model’s release. When you browse Hugging Face’s GGUF library, filter by these reputable sources:
- ggml-org – the official llama.cpp organization’s own conversions, ideal for testing the latest features.
- bartowski – thorough quant ranges (from IQ2 to Q8_0) for nearly every popular model, with importance-matrix variants.
- unsloth – fast-turnaround quants known for fixed chat templates and “dynamic” quantization that preserves quality.
- The original model author – many labs (Google, Mistral, Qwen) now publish first-party GGUF files alongside their safetensors release.
For gated or private models, set an environment token before using the -hf flag: export HF_TOKEN=hf_your_token_here. llama.cpp reads that variable automatically and authenticates the download. Two practical sourcing tips: prefer files that include an embedded chat template (most modern quants do), and read the model card for the exact prompt format – a mismatch here is the single most common cause of garbled output. If you cannot find a GGUF for the model you want, Steps 9 and 10 show you how to make your own from any Hugging Face checkpoint.
Step 5 – Run Your First Inference with llama-cli
With a model in hand, run a one-shot prompt. Since the July 2026 rebuild, llama-cli is a thin client that launches llama-server in the background and talks to it, rather than running inference in the same process – the commands below behave identically either way. The -m flag points to your GGUF file, -p supplies the prompt, and -n caps the number of tokens generated.
# One-shot generation (CPU or auto-GPU)
./build/bin/llama-cli \
-m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-p "Explain what GGUF is in two sentences." \
-n 128For an interactive chat session – the experience most people want – add the -cnv (conversation) flag. llama.cpp applies the model’s built-in chat template automatically, so you simply type and press Enter.
# Interactive conversation mode
./build/bin/llama-cli \
-m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-cnv
> Hi! Give me three uses for a local LLM.
1. Private document summarization that never leaves your machine...
2. Offline coding assistance...
3. Building RAG pipelines without per-token API costs...The first run is slower because the model loads into memory; thanks to memory-mapped (mmap) loading, subsequent runs of the same file start almost instantly. If output looks like nonsense or repeats endlessly, your model file is likely missing a chat template or you picked too aggressive a quant – both covered in the troubleshooting section below.
Step 6 – Tune Performance: GPU Layers, Context, and Threads
This is where a local LLM goes from “works” to “fast.” Three flags do most of the heavy lifting: -ngl offloads model layers to the GPU, -c sets the context window, and -t controls CPU threads.
# Offload all layers to GPU, 8K context, flash attention on
./build/bin/llama-cli \
-m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-ngl 99 \ # offload as many layers as fit in VRAM
-c 8192 \ # context window (tokens)
-t 8 \ # CPU threads for any non-offloaded work
-fa \ # enable flash attention (saves memory)
-cnvSetting -ngl 99 is a common idiom: it tells llama.cpp to put up to 99 layers on the GPU, which in practice means “all of them” for most models. If a model is too big for your VRAM, lower the number to do a partial offload – say -ngl 20 – and the remaining layers run on CPU. The key flags you will reach for again and again are summarized here.
| Flag | Long form | Purpose |
|---|---|---|
-m | --model | Path to the GGUF model file |
-hf | --hf-repo | Download a model from Hugging Face |
-ngl | --n-gpu-layers | Number of layers to offload to GPU |
-c | --ctx-size | Context window in tokens |
-n | --predict | Max tokens to generate |
-t | --threads | CPU threads for generation |
-fa | --flash-attn | Enable flash attention |
-cnv | --conversation | Interactive chat mode |
--load-mode | – | Choose the model loading strategy (replaced --mmap/--mlock/--direct-io in v0.4.1) |
One more knob changed recently and is worth knowing about. Through v0.4.0, a separate --mlock flag pinned the model in RAM so the OS never swapped it to disk, keeping latency consistent on memory-pressured systems. v0.4.1 (September 14, 2026) removed that flag along with --mmap and --direct-io, folding all three into a single --load-mode flag that picks the loading strategy directly – update any saved command that still passes the old flags or it will fail to start. That consolidated flag has carried forward unchanged through v0.6.0 (which landed in October 2026 with an extended batch API and support for GLM-5.3-Flash) and into v0.7.0, the newest tagged release as of this writing, so the syntax above still applies. On consumer GPUs with limited VRAM, quantizing the KV cache with --cache-type-k q8_0 --cache-type-v q8_0 lets you fit a much longer context at a tiny quality cost.
Step 7 – Start the OpenAI-Compatible Server (llama-server)
The real power of llama.cpp for application developers is llama-server: a lightweight HTTP server that speaks the OpenAI chat API natively. Start it with your model and a port. This is the foundation of any local “llama-cpp server” deployment.
# Launch the API server with GPU offload and a 4K context
./build/bin/llama-server \
-m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8080 \
-ngl 99 \
-c 4096Once it boots, llama-server prints its listening address. The default port is 8080, and it ships with a built-in web chat UI – just open http://localhost:8080 in a browser to test the model with no extra software. This -ngl 99 / -c 4096 / --port 8080 combination is not specific to Llama 3 – it is the same baseline server invocation you will see recommended across 2026 guides for other popular GGUF models, including Qwen2.5-7B-Instruct-GGUF, which makes it a safe default to memorize. Behind the scenes the server exposes several endpoints.
main: server is listening on http://0.0.0.0:8080
main: loading model ...
llama_model_loaded: model size = 4.58 GiB
main: model loaded, slots: 1
# Web UI: http://localhost:8080
# OpenAI chat endpoint: POST /v1/chat/completions
# Native endpoint: POST /completion
# Health check: GET /health
# Embeddings: POST /v1/embeddingsBecause the server is OpenAI-compatible, any tool or SDK that talks to OpenAI can be pointed at it by changing the base URL – VS Code extensions, LangChain, LlamaIndex, and your own apps all work unchanged. Full options are documented in the official llama-server README.
As of October 2026, NVIDIA’s NemoClaw agent stack documents this exact server as a supported local-inference backend for its always-on AI agents. Its llama.cpp setup guide lays out two integration paths: attach the llama-server you just started as an authenticated, self-managed endpoint, or let NemoClaw provision and manage the deployment itself on qualified NVIDIA hardware (DGX Spark, Linux x86_64 with an NVIDIA GPU, or Windows WSL). Both paths route through the same OpenAI-compatible /v1 endpoint, so the client code in Step 8 does not change either way.
Step 8 – Call the API from curl and Python
With the server running, test it from the terminal first. The request body is identical to OpenAI’s – the only difference is the host. The model field can be any string; the server uses whatever model it loaded.
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local-llama",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What does the -ngl flag do in llama.cpp?"}
],
"temperature": 0.7
}'For application code, the official OpenAI Python SDK is the cleanest route. Install it with pip install openai, then override base_url to point at your local server. The api_key is required by the SDK but ignored by llama-server unless you enabled one.
from openai import OpenAI
# Point the OpenAI SDK at your local llama-server
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed",
)
response = client.chat.completions.create(
model="local-llama",
messages=[
{"role": "user", "content": "Write a haiku about local inference."}
],
)
print(response.choices[0].message.content)That is the entire integration. Any existing OpenAI codebase becomes a local-LLM codebase by swapping two lines. If you prefer a thicker Python wrapper with its own bindings, the separate llama-cpp-python package wraps the same engine and also offers an OpenAI-compatible server mode – useful when you want to embed inference inside a Python process rather than running a standalone binary. Python tooling moves fast here; our uv vs pip guide covers the fastest way to manage these dependencies.
Complete Project: A Streaming Local Chatbot in Python
Let’s assemble everything into one complete, working project: a multi-turn terminal chatbot that streams tokens in real time from your local llama-server. With the server from Step 7 running, save the script below as chat.py, run pip install openai, and execute it with python chat.py. It keeps full conversation history and prints the model’s reply word by word, exactly like a hosted chatbot – but entirely on your own machine.
from openai import OpenAI
# Connect to the local llama-server started in Step 7
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
# Conversation history, seeded with a system prompt
history = [{"role": "system", "content": "You are a helpful local assistant."}]
print("Local chatbot ready. Type 'exit' to quit.\n")
while True:
user = input("You: ")
if user.strip().lower() in {"exit", "quit"}:
break
history.append({"role": "user", "content": user})
# Stream the response token-by-token
stream = client.chat.completions.create(
model="local-llama", messages=history, stream=True
)
print("Bot: ", end="", flush=True)
reply = ""
for chunk in stream:
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
reply += delta
print("\n")
# Remember the assistant's turn for context
history.append({"role": "assistant", "content": reply})That is a genuinely useful application in roughly 25 lines. Because it talks to llama-server over the OpenAI-compatible API, you can swap the underlying model – Llama 3, Qwen, Gemma – without touching a single line of this code; just restart the server with a different -m file. From here you can extend it with retrieval (call the /v1/embeddings endpoint to ground answers in your own documents), persist history to disk, or wrap it in a web UI. This same pattern is how production teams put a local LLM behind an internal tool.
Step 9 – Convert a Hugging Face Model to GGUF
Sometimes the model you want has no community GGUF yet – for example, a model you fine-tuned yourself. llama.cpp ships Python scripts to convert Hugging Face (safetensors) checkpoints into GGUF. First install the conversion requirements, then run the converter.
# Install the Python dependencies for conversion
pip install -r requirements.txt
# Download a HF model (example uses git-lfs), then convert to GGUF
python convert_hf_to_gguf.py ./models/my-finetuned-model \
--outfile ./models/my-model-f16.gguf \
--outtype f16The --outtype f16 option produces a full-precision 16-bit GGUF. That file is large, but it is the recommended source for quantization in the next step – always quantize down from F16 rather than from an already-quantized file, to avoid compounding precision loss. The conversion script is named convert_hf_to_gguf.py; older guides reference a generic convert.py, which no longer exists in current builds.
Step 10 – Quantize a Model with llama-quantize
Now shrink that F16 file to a deployable size. The llama-quantize binary takes an input GGUF, an output path, and a target quantization type.
# Quantize the F16 model to the recommended Q4_K_M
./build/bin/llama-quantize \
./models/my-model-f16.gguf \
./models/my-model-Q4_K_M.gguf \
Q4_K_MThe tool prints each tensor as it processes it and reports the final size reduction. A typical 8B F16 model around 16 GB compresses to roughly 4.9 GB at Q4_K_M – a reduction of about 70% that still preserves most of the model’s quality.
[ 1/ 291] token_embd.weight - converting to q4_K .. size = 500.0 MiB -> 140.6 MiB
...
llama_model_quantize_internal: model size = 15317.02 MB
llama_model_quantize_internal: quant size = 4685.30 MB
main: quantize time = 48213.55 msFor the highest quality at very low bit-widths (IQ2/IQ3), pass an importance matrix generated with llama-imatrix from a small calibration dataset. For most users, the plain Q4_K_M or Q5_K_M output above is more than good enough.
Step 11 – Benchmark Throughput with llama-bench
To know whether your build and flags are actually fast, use llama-bench. It measures two key numbers: prompt processing speed (how fast the model ingests your input, shown as pp) and token generation speed (how fast it writes the answer, shown as tg), both in tokens per second.
# Benchmark prompt-processing and generation throughput
./build/bin/llama-bench \
-m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
-ngl 99The output is a clean table. The example below shows the format you will see – your actual tokens-per-second depends entirely on your CPU, GPU, and the model, so treat the numbers as illustrative rather than a hardware claim.
| model | size | backend | ngl | test | t/s |
| ------------------ | ---------- | ------- | --: | ----: | --------: |
| llama 8B Q4_K_M | 4.58 GiB | CUDA | 99 | pp512 | (varies) |
| llama 8B Q4_K_M | 4.58 GiB | CUDA | 99 | tg128 | (varies) |Run llama-bench before and after changing flags to measure the impact of -fa, KV-cache quantization, or a different -ngl value. It is the objective way to tune a local LLM rather than guessing. Prompt-processing (pp) speed dominates for long-document and RAG workloads, while generation (tg) speed is what you feel during chat.
Step 12 – Run llama.cpp as a Background Service (Production Tips)
For a persistent local LLM endpoint – on a home server or workstation – wrap llama-server in a systemd service so it starts on boot and restarts on failure; this remains the standard production pattern as of October 2026. Create /etc/systemd/system/llama.service:
[Unit]
Description=llama.cpp server
After=network.target
[Service]
ExecStart=/opt/llama.cpp/build/bin/llama-server \
-m /opt/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--host 0.0.0.0 --port 8080 -ngl 99 -c 8192 \
--api-key YOUR_SECRET_KEY
Restart=on-failure
User=llama
[Install]
WantedBy=multi-user.targetThen enable and start it:
sudo systemctl daemon-reload
sudo systemctl enable --now llama.service
systemctl status llama.serviceThree production hardening tips: set --api-key so the endpoint requires a bearer token; use --parallel N (or -np N) to handle multiple simultaneous requests by splitting the context into slots; and if you prefer containers, the project publishes official Docker images so you can run llama-server with docker run --gpus all ghcr.io/ggml-org/llama.cpp:server and skip the build entirely. Never expose an unauthenticated server to the public internet.
Quick Reference: Every Command in This Tutorial
Once you have run through Steps 1–12 a first time, you rarely need the full explanations again – just the command. Bookmark this table as a one-page cheat sheet for the whole llama.cpp workflow above.
| Step | Goal | Command |
|---|---|---|
| 1 | Clone the repo | git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp |
| 2 | Build (CPU) | cmake -B build && cmake --build build --config Release -j |
| 3 | Build with CUDA | cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc) |
| 4 | Pull a model from Hugging Face | ./build/bin/llama-cli -hf ggml-org/gemma-3-1b-it-GGUF |
| 5 | Interactive chat | ./build/bin/llama-cli -m model.gguf -cnv |
| 6 | Tuned local run | ./build/bin/llama-cli -m model.gguf -ngl 99 -c 8192 -t 8 -fa -cnv |
| 7 | Start the API server | ./build/bin/llama-server -m model.gguf --host 0.0.0.0 --port 8080 -ngl 99 -c 4096 |
| 8 | Test the API | curl http://localhost:8080/v1/chat/completions -d '{"messages":[...]}' |
| 9 | Convert HF model to GGUF | python convert_hf_to_gguf.py ./models/my-model --outfile model-f16.gguf --outtype f16 |
| 10 | Quantize to Q4_K_M | ./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M |
| 11 | Benchmark throughput | ./build/bin/llama-bench -m model.gguf -ngl 99 |
| 12 | Run as a background service | sudo systemctl enable --now llama.service |
What Changed in llama.cpp Tutorials: Old Commands vs. Current (2026)
If you followed an older llama.cpp guide and a command no longer works, you have not done anything wrong – the project renamed and relocated several pieces between 2023 and 2026. Every command earlier in this tutorial already reflects the current, ggml-org-era project as of October 2026; the table below collects those changes in one place so you can update old scripts, bookmarks, and Stack Overflow answers in a single pass.
| Old / pre-2024 | Current (2026) | Where it matters |
|---|---|---|
ggerganov/llama.cpp | ggml-org/llama.cpp | Repo path (old URL still redirects) – Step 1 |
Makefile (make) | CMake only | Build system, retired in 2024 – Step 2 |
./main | llama-cli | Interactive chat binary, renamed 2024 – Step 2 |
./server | llama-server | API server binary, renamed 2024 – Step 7 |
convert.py | convert_hf_to_gguf.py | Hugging Face to GGUF conversion script – Step 9 |
Manual .gguf download only | -hf flag pulls straight from the Hugging Face Hub | Model acquisition, cached in the standard HF cache dir – Step 4 |
Direct, in-process inference via llama-cli | llama-cli is a thin client that launches llama-server in the background; the old direct binary is now llama-completion | Rebuilt July 2026 – Step 2 and Step 5 |
--mmap / --mlock / --direct-io | --load-mode | Loading-strategy flags consolidated in v0.4.1, September 14, 2026 – Step 6 |
5 Common Pitfalls When Running llama.cpp
These are the mistakes that trip up almost everyone the first time they run a local LLM with llama.cpp. Avoiding them saves hours of confusion.
- Building in Debug instead of Release. If you forget
--config Release, inference can be several times slower. Always confirm you built the optimized configuration. - Leaving
-nglat 0 on a GPU build. A GPU-enabled build still runs on CPU until you tell it to offload. If performance feels like CPU-only, you probably forgot-ngl 99. - Choosing a quant that does not fit your VRAM. If the model plus context exceeds VRAM, llama.cpp silently falls back to slow CPU layers (or errors out). Match the quant to the hardware table above.
- Quantizing from an already-quantized file. Always convert to F16 first, then quantize down. Re-quantizing a Q4 file compounds precision loss and produces noticeably worse output.
- Ignoring the chat template. Garbled or never-ending output usually means the model’s chat template is missing or wrong. Use
-cnv(which auto-applies the template) or pass--chat-templateexplicitly.
Troubleshooting llama.cpp: 8 Common Errors and Fixes
When something breaks, the error message usually points straight at the cause. Here are the eight problems you are most likely to hit and how to resolve each one.
| Symptom / error | Likely cause | Fix |
|---|---|---|
CMake Error: Could NOT find CUDAToolkit | CUDA Toolkit not installed or not on PATH | Install the CUDA Toolkit and re-run the configure step; verify with nvcc --version |
error loading model: unknown (magic) | File is not valid GGUF (e.g., a safetensors download) | Download a real .gguf file or convert it in Step 9 |
cudaMalloc failed: out of memory | Model + context exceed VRAM | Lower -ngl, use a smaller quant, or reduce -c |
| Output is gibberish or never stops | Wrong/missing chat template | Use -cnv or set --chat-template for the model |
illegal instruction (core dumped) | Binary built for newer CPU instructions (AVX512) than your CPU has | Rebuild on the target machine; CMake auto-detects supported instructions |
| Very slow generation on GPU build | -ngl not set, so layers run on CPU | Add -ngl 99 to offload to the GPU |
bind: address already in use | Port 8080 is taken by another process | Pick a different --port or stop the conflicting service |
| GGUF version mismatch / fails to load | Model made with a much newer or older GGUF spec | Update llama.cpp (git pull + rebuild) or re-download the model |
A general rule: when in doubt, run git pull in your llama.cpp directory and rebuild. The project regularly ships multiple builds within a single day – the latest tagged release as of October 2026 is still v0.4.1, which landed September 14, 2026 – so staying current fixes a surprising number of model-loading and backend issues automatically. That release is also the reason a saved --mlock or --mmap command suddenly errors out: both were removed in favor of --load-mode (see Step 6).
Advanced llama.cpp Tips: Speculative Decoding, Grammars, and Embeddings
Once the basics work, llama.cpp has powerful features that most tutorials skip. These are worth graduating to.
Speculative decoding for faster generation
Pair a large “target” model with a tiny “draft” model that proposes tokens the big model then verifies in batches. When the draft guesses well, generation speeds up at no quality cost. Pass a draft model with -md (model-draft) alongside your main -m. It works best when the two models share a tokenizer family – for example an 8B target with a 1B draft.
Constrained output with GBNF grammars and JSON schema
llama.cpp can force the model’s output to match a grammar. Pass --grammar-file schema.gbnf, or use the server’s response_format with a JSON schema, to guarantee parseable structured output – invaluable for function calling and data extraction where you cannot tolerate a stray token. This is built into the engine, so it works offline with any model.
Two more advanced capabilities round out the toolkit: embeddings, generated via llama-embedding or the server’s /v1/embeddings endpoint, let you build a fully local RAG pipeline; and multimodal support (the mtmd stack) lets vision models like Qwen-VL and LLaVA accept images when you supply a projector file with --mmproj. The mtmd stack is moving beyond vision, too: early August 2026 commits added support for Qwen3-TTS speech synthesis and a new chunk save/load function for multimodal sessions, so audio is becoming a first-class citizen alongside images. That expansion continued through the month – the repository’s August 27, 2026 commit added new --video-* CLI arguments, extending the same mtmd groundwork toward video input alongside the existing image and audio support. With embeddings plus chat both served locally, you can build an entire private AI assistant without a single external API call – a stack worth comparing against hosted options in our Claude vs ChatGPT vs Gemini roundup.
llama.cpp vs Ollama: Which Should You Use?
Because Ollama is built on top of llama.cpp’s ggml engine, the question is not which is “faster” – they share the same core – but which abstraction fits your workflow. Ollama optimizes for convenience: a one-line install, a curated model registry, and ollama run llama3 simplicity. llama.cpp optimizes for control: you choose the exact build flags, the exact quant, the exact sampling and KV-cache settings, and you get every new feature the day it merges rather than after a wrapper exposes it.
Choose Ollama if you want the fastest possible “just works” path and don’t need low-level tuning. Choose llama.cpp directly if you care about squeezing maximum performance from specific hardware, need features like speculative decoding or custom grammars before wrappers expose them, or are deploying a lean production server without an extra management layer. Many teams use both: Ollama on developer laptops for convenience, and raw llama-server in production for control. For a full performance-oriented comparison of local serving engines, see our vLLM vs Ollama analysis, and for the broader open-model landscape our Grok vs ChatGPT coverage.
Frequently Asked Questions About llama.cpp
Is llama.cpp free to use?
Yes. llama.cpp is open source under the permissive MIT license, which allows commercial use, modification, and redistribution. There are no usage fees – your only cost is the hardware you run it on. The models you load have their own separate licenses, so always check the license of any model you download from Hugging Face.
Do I need a GPU to run llama.cpp?
No. llama.cpp was originally famous precisely because it runs LLMs efficiently on CPU using AVX2/AVX512 instructions. Small models (1–8B at Q4_K_M) run comfortably on a modern laptop with 16 GB of RAM. A GPU dramatically increases speed and lets you run larger models, but it is optional for following this llama.cpp tutorial.
What is the difference between llama.cpp and Ollama?
Ollama is a user-friendly wrapper built on top of llama.cpp’s ggml engine. llama.cpp is the underlying inference library and gives you direct, low-level control over build flags, quantization, and runtime settings. Ollama trades some of that control for a simpler model-management experience. They share the same core, so neither is fundamentally faster.
What does the GGUF format do?
GGUF is the single-file model format llama.cpp requires. It bundles the weights, tokenizer, and metadata into one portable file and supports quantization down to as low as 2 bits. It is the successor to the older GGML format and is now the de facto standard for local LLM distribution, with first-class support on the Hugging Face Hub.
Which quantization should I choose?
For most users, Q4_K_M is the recommended default – it preserves nearly all of a model’s quality at about a quarter of the full-precision size. If you have plenty of VRAM and want maximum fidelity, use Q6_K or Q8_0. If you are squeezing a model into tight memory, drop to Q3_K_M, accepting some quality loss.
Can llama-server replace the OpenAI API?
For chat, completions, and embeddings, yes. llama-server implements an OpenAI-compatible API at /v1/chat/completions and related paths, so most apps and SDKs work by simply changing the base URL to http://localhost:8080/v1. Provider-specific features (like OpenAI’s hosted tools) are not replicated, but the core chat and embeddings surface is a drop-in replacement.
What models can llama.cpp run?
More than 50 text architectures, including Llama 3, Qwen, Gemma, Mistral, Mixtral, Phi, Falcon, and DeepSeek, plus multimodal models like LLaVA and Qwen-VL – and, as of early August 2026, Qwen3-TTS for local speech synthesis through the mtmd stack. The list keeps growing: v0.4.1, released September 14, 2026, added Maple 20B-A1B, Tencent Hy 4, and Spark2.5. (Newer architectures such as DeepSeek V4 are still being upstreamed from community forks as of August 2026 and are not yet in mainline builds.) Any model with a community GGUF on Hugging Face works immediately; others can be converted with convert_hf_to_gguf.py as shown in Step 9.
What is the difference between llama-cli and the newer llama cli command?
Nothing runs differently under the hood. llama-cli and llama-server are the hyphenated binaries this tutorial builds from source in Step 2, and every command shown here continues to work exactly as documented. llama cli -hf <repo> and llama serve -hf <repo> are the space-separated forms the official README still leads with as of October 2026 – a shorter entry point that pairs the same -hf download-and-run flow with a friendlier command shape. If you spot the no-hyphen version in the README, a forum post, or a newer guide, treat it as the same tool addressed a different way, not a required migration.
Why does llama-cli start a server in the background now?
Because the project rebuilt it that way in July 2026. Running llama-cli now launches llama-server in the background and talks to it as a thin client, instead of running inference directly inside the same process the way it always had before. Every llama-cli command in this tutorial still works exactly as written – the change is internal. If you specifically need the old direct, no-server behavior, for example inside a minimal script or container image, use llama-completion, the renamed version of the original standalone inference binary; it takes the same flags.
What is the latest version of llama.cpp?
v0.4.1 is still the latest tagged release as of October 2026. It was tagged on September 14, 2026 and is listed on GitHub as the repository’s official “Latest” release. It followed v0.4.0 on September 4, 2026, which in turn followed v0.3.0 on August 25, 2026 – the first time the project marked a numbered release as “Latest” after years of continuous build-tagged releases with no semantic versioning. v0.4.1 added support for the Maple 20B-A1B, Tencent Hy 4, and Spark2.5 architectures, and it is a breaking release for scripts: the old --mmap, --mlock, and --direct-io flags are gone, replaced by a single --load-mode flag (see Step 6). The underlying code still moves faster than the tags, so if you want the newest optimizations rather than a stable snapshot, pull master directly – the next question shows how.
How do I update llama.cpp to the latest build?
Run git pull in your cloned directory, then rebuild with cmake --build build --config Release -j. The project ships continuous build-tagged releases and commits land almost daily, so pulling regularly keeps you current with new model support and bug fixes. As of October 2026, GitHub still marks v0.4.1 (September 14, 2026) as the repository’s “Latest” tag, so grabbing that release instead of pulling master is a quicker way to land on a stable, tested snapshot that already includes support for Maple 20B-A1B, Tencent Hy 4, and Spark2.5, plus the newer --load-mode flag in place of the removed --mmap/--mlock/--direct-io options.
Real-World Use Cases for a Local llama.cpp Setup
Running an LLM locally is not just a hobbyist exercise – it solves concrete problems that hosted APIs cannot. Here are the workloads where a llama.cpp deployment genuinely shines in 2026:
- Private document analysis: summarize contracts, medical records, or internal reports without sending a single byte to a third party – the only way to stay compliant in regulated industries.
- Offline coding assistance: point a VS Code or JetBrains extension at llama-server’s OpenAI endpoint for autocomplete and chat that works on a plane, behind a firewall, or in an air-gapped environment.
- Local RAG pipelines: combine the embeddings endpoint with a vector store to build a question-answering system over your own knowledge base, at zero marginal cost per query.
- High-volume batch jobs: classify, tag, or extract data from millions of records where per-token API pricing would be prohibitively expensive.
- Edge and on-device AI: because llama.cpp runs on a Raspberry Pi, a phone, or a tiny mini-PC, it powers assistants that ship inside hardware products with no cloud dependency.
The common thread is sovereignty: when the model runs on hardware you control, you own the data, the uptime, and the cost curve. As open-weight models keep closing the gap with proprietary frontier systems – a trend we track across our AI coverage – the case for a local llama.cpp deployment only gets stronger. For many teams, the right architecture in 2026 is a hybrid: a hosted frontier model for the hardest 5% of queries, and a local llama.cpp server handling the other 95% privately and for free.
Conclusion: Your Local LLM Stack Is Ready
You now have a complete, working local LLM setup: you cloned and built llama.cpp with CMake, optionally enabled GPU acceleration, downloaded a quantized GGUF model, ran interactive inference with llama-cli, tuned it with GPU offload and context flags, served an OpenAI-compatible API with llama-server, called it from curl and Python, and even converted, quantized, and benchmarked your own model. That is a genuinely production-capable private AI stack running entirely on your hardware – no API keys, no per-token bills, and no data leaving your machine.
The next step is to build something on top of it: wire llama-server into a coding assistant, stand up a local RAG pipeline using the embeddings endpoint, or experiment with speculative decoding to push generation speed further. Because llama.cpp ships new builds daily, keep your clone updated to ride the steady stream of new model support and optimizations. Local AI in 2026 is no longer a compromise – with llama.cpp, it is a serious, free, and private alternative to the cloud.
