A local model that answers in 40 tokens per second instead of 20 is not a hardware upgrade, it is a software decision. Speculative decoding pairs a small “draft” model with the larger model you actually want answers from, and lets the big model verify several guessed tokens in one pass instead of generating them one at a time. By September 2026 this stopped being a research curiosity: Ollama shipped native MTP-based speculative decoding in v0.32.6, vLLM rebuilt its entire speculative config around a single JSON block in v0.30.0, and llama.cpp added EAGLE-3 and DFlash draft modes as first-class flags. This tutorial sets up speculative decoding across all three runtimes, plus LM Studio, in 13 steps, using only draft and target model pairs that ship real, verified support. Budget about 90 minutes if you already have a working local LLM setup, longer if you are building llama.cpp from source for the first time.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What Speculative Decoding Actually Does to Your Token Generation
Standard autoregressive decoding generates one token, feeds it back into the model, generates the next token, and repeats. Every single step pays the full cost of running the large model forward, even though most of those steps are cheap in terms of raw compute and expensive only because of memory bandwidth. Speculative decoding breaks that pattern. A smaller “draft” model proposes a short run of candidate tokens, and the large “target” model checks all of them in a single forward pass, accepting the ones that match what it would have generated anyway and rejecting the rest. Because verification is parallel while generation is sequential, this trade lands squarely in favor of the target model’s idle memory bandwidth.
The technique was formalized in research on speculative decoding for transformers and later extended by projects like EAGLE, Medusa, and n-gram draft-and-verify approaches, and it has become one of the more approachable ways to speed up self-hosted inference without buying a bigger GPU. It works especially well for structured or repetitive output such as code, JSON, and long factual answers, where the draft model’s guesses have a high chance of matching the target model. It works less well for creative writing at high temperature, where token-by-token unpredictability lowers the acceptance rate and can make speculative decoding a net loss.
Three separate mechanisms now fall under the same “speculative decoding” umbrella, and this tutorial covers all three because different runtimes default to different ones:
- Standalone draft model – a genuinely separate, much smaller model from the same family proposes tokens (for example, a 0.6B Qwen model drafting for an 8B Qwen model).
- Multi-Token Prediction (MTP) – the target model ships its own lightweight prediction head that drafts for itself, so there is no second model to download at all.
- EAGLE / EAGLE-3 heads – a small, trained “draft head” attached to the target model’s hidden states, which typically reaches a higher acceptance rate than an independent draft model.
The economics matter as much as the mechanics here. Every token an API-hosted model generates for you carries a per-token price tag, but every token a self-hosted model generates costs whatever your electricity bill works out to, regardless of how fast or slow it runs. Speculative decoding does not change that cost structure directly, but it changes the throughput you get out of hardware you already own, which is exactly the lever self-hosters actually control. A single RTX-class GPU that felt marginal for real-time chat at 20 tokens per second stops feeling marginal once well-matched speculative decoding pushes it meaningfully higher on the same card, same power draw, same rent or mortgage payment.
How Speculative Decoding Compares to Quantization and Batching
Speculative decoding is not a replacement for the other speed techniques circulating in the local LLM community, it stacks on top of them. Quantization (running a model at Q4_K_M instead of full precision) shrinks memory footprint and reduces the bytes that need to move per token, which speeds up generation for a different reason than speculative decoding does — quantization reduces the cost of each step, while speculative decoding reduces the number of sequential steps that pay full cost. Running both together is standard practice: quantize the target model to fit your VRAM budget, then add a draft model on top for additional throughput.
Continuous batching, the technique vLLM and similar production servers use to serve many concurrent requests efficiently, interacts with speculative decoding in the opposite direction — it can reduce the benefit rather than compound it. Batching already keeps the GPU’s compute units busy across multiple users’ requests, which is the same idle capacity speculative decoding is designed to exploit for a single request. At high concurrency, that headroom is already spoken for, so the marginal gain from adding a draft model shrinks. This is why speculative decoding tends to show its largest gains for single-user or low-concurrency local setups, and a smaller, still-real but less dramatic gain in busy multi-tenant serving. Hugging Face’s own generation strategies documentation covers how these decoding-time techniques relate to each other in more architectural detail if you want the underlying theory.
Prerequisites: Hardware, Software, and Model Versions
Before touching a config file, confirm you have the following. Version numbers below reflect what shipped as of late September 2026, and speculative decoding support moves fast enough that it is worth checking each project’s release notes before you start.
- A CUDA-capable GPU (or Apple Silicon for the Ollama MLX path) with enough VRAM to hold the target model plus a small draft model at once — see the sizing table in Step 1.
- llama.cpp built at or after build b11100 (September 22, 2026), which carries the
--spec-typeflag family used throughout this guide. - vLLM v0.30.0 or newer if you want server-side speculative decoding through the unified
--speculative-configJSON interface (the older--speculative-modelflag was deprecated). - Ollama v0.32.6 or newer for native MTP speculative decoding; the current release at publication is v0.34.4 (September 23, 2026).
- LM Studio 0.4.14 or newer if you prefer a GUI toggle over CLI flags.
- Python 3.10+ and a Hugging Face account/token if you plan to pull draft models or convert EAGLE heads yourself.
Step 1: Confirm Your VRAM Budget for Two Models at Once
The single most common reason a speculative decoding setup fails on the first try is running out of VRAM, because you are now loading two models instead of one. Draft models are small by design, typically in the 0.5B–4B range, but they still add real memory overhead on top of your target model’s footprint. Check your available VRAM first.
nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csvUse the table below as planning guidance rather than a hard minimum — actual usage shifts with context length, quantization, and KV-cache settings.
| Target model class | System RAM | Target VRAM | Add for draft model | Typical disk (quantized) |
|---|---|---|---|---|
| 7B-class target | ~16 GB | ~8 GB | +1–2 GB (0.5B–1B draft) | 6–10 GB |
| 27B-class target | 32–64 GB | 16–24 GB | +2–4 GB (1B–4B draft) | 20–35 GB |
| 70B-plus target | 64–128 GB | 48–80 GB+ | +3–6 GB (4B-class draft) | 45–100 GB |
If you are already running close to your VRAM ceiling with the target model alone, either drop to a smaller draft model, quantize the draft more aggressively than the target (a Q4 draft next to a Q6 target is a common combination), or use MTP mode, which needs no second model file since the drafting head is baked into the target checkpoint. On a single consumer GPU in the 16–24 GB range, the realistic sweet spot is a 7B–14B target with a sub-1B draft, or a 27B-class target quantized down far enough to leave 2–4 GB of headroom. Anything larger genuinely needs the 48 GB-plus workstation or multi-GPU tier, at which point the draft model’s footprint becomes a rounding error next to the target’s.
Step 2: Build or Update llama.cpp to the Latest Release
The --spec-type flag family and the EAGLE-3/DFlash/DSpark draft modes are recent additions, so an old llama.cpp checkout will simply not recognize the flags in this guide. Build from source to get the current speculative decoding code path, documented in the project’s own speculative decoding reference.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
./build/bin/llama-server --versionDrop -DGGML_CUDA=ON if you are building for Apple Silicon or CPU-only; llama.cpp will pick up Metal automatically on macOS. Confirm the build succeeded and that llama-server --help lists --spec-type before moving on.
Step 3: Pick a Verified Draft and Target Model Pair
This is where most speculative decoding setups quietly fail without erroring out: a mismatched tokenizer between draft and target model tanks the acceptance rate, so speedup disappears even though the server runs fine. The safest path is staying inside one model family, or using a model that ships its own MTP head so there is nothing to mismatch.
| Target model | Draft option | Mechanism | Notes |
|---|---|---|---|
| Qwen3.6 / Qwen3.8 family | Built-in MTP head | MTP | Ships a drafting head with the checkpoint; no separate download needed |
| Gemma 4 | Built-in MTP head | MTP | Supported via Ollama’s MLX runner and llama.cpp’s draft-mtp mode |
| DeepSeek-V3 / DeepSeek-R1 | Built-in MTP layers | MTP | Enable with vLLM’s "method": "mtp" config, no draft model field required |
| GLM-4.7 | Built-in MTP layers | MTP | Same vLLM MTP config path as DeepSeek-V3/R1 |
| MiMo-V2.6-Pro | MiMo-V2.6-Flash | Standalone draft model | Same architecture family, smaller Flash variant drafts for Pro |
| Qwen3-4B (custom) | AngelSlim Qwen3-4B EAGLE3 head | EAGLE-3 | Requires converting the EAGLE head checkpoint to GGUF yourself |
Same-Family Pairs vs Built-In MTP Drafters
If your target model already ships an MTP head, use it before reaching for a standalone draft model — it is one less file to download, one less tokenizer to worry about, and generally the path the model’s own team tuned for. If you are working with a model family that has no MTP head, pick a standalone draft roughly eight to ten times smaller than the target, from the exact same family, sharing the exact same tokenizer. A Qwen draft for a Qwen target works. A Llama draft for a Qwen target does not, unless you explicitly enable a cross-vocabulary bridge like vLLM’s Token-Level Intersection setting, covered in Step 7.
Before launching anything, it is worth confirming the tokenizer match programmatically rather than assuming it from the model names. Two checkpoints from the same family can still diverge if one was fine-tuned with an expanded vocabulary or custom special tokens. A quick sanity check with the Hugging Face transformers library catches this before you spend time debugging a low acceptance rate later:
from transformers import AutoTokenizer
target = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-8B-Instruct")
draft = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-0.6B-Instruct")
print(target.get_vocab() == draft.get_vocab()) # should print TrueIf that prints False, do not proceed with a standalone draft pairing — either find a draft explicitly released alongside the target, fall back to the target’s MTP head if it has one, or use vLLM’s heterogeneous-vocabulary mode and expect a lower acceptance rate.
Step 4: Download the GGUF Files
Pull both the target and draft GGUF files with the Hugging Face CLI, or with Ollama’s pull command if you are going the Ollama route covered in Step 8. Keep the draft model’s quantization at or below the target’s — a Q4_K_M draft paired with a Q6_K or Q8_0 target is a common, safe combination that keeps the draft cheap without dragging down its accuracy too far.
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-8B-Instruct-GGUF qwen3-8b-q6_k.gguf --local-dir ./models
huggingface-cli download Qwen/Qwen3-0.6B-Instruct-GGUF qwen3-0.6b-q4_k_m.gguf --local-dir ./modelsStep 5: Launch llama-server With Classic Draft-Model Speculation
With both GGUF files in place, point llama-server at the target model with -m and the draft model with -md (also written --model-draft in the full flag form). Set --spec-type draft-simple for a plain standalone draft model.
./build/bin/llama-server \
-m ./models/qwen3-8b-q6_k.gguf \
-md ./models/qwen3-0.6b-q4_k_m.gguf \
--spec-type draft-simple \
--spec-draft-n-max 16 \
--spec-draft-n-min 4 \
--spec-draft-p-split 0.10 \
--ctx-size 32768 \
--port 8080 \
-ngl 99--spec-draft-n-max caps how many tokens the draft model proposes in a single burst (the llama.cpp default is 3; 16 is a common starting point for well-matched pairs). --spec-draft-n-min is a floor that stops the draft from bailing out too early on a weak run of guesses, and --spec-draft-p-split controls the probability threshold used to decide whether to keep speculating deeper into a branch. Leave --spec-draft-p-min at its default of 0.00 unless you are specifically tuning against a benchmark.
| Flag | Default | Purpose |
|---|---|---|
-md / --model-draft | none | Path to the draft model GGUF file |
--spec-type | none | Selects the mechanism: draft-simple, draft-eagle3, draft-dflash, draft-dspark, draft-mtp, or an n-gram mode |
--spec-draft-n-max | 3 | Maximum tokens the draft proposes per burst |
--spec-draft-n-min | 0 | Minimum tokens before the draft can give up on a weak run |
--spec-draft-p-split | 0.10 | Probability threshold for continuing to speculate deeper |
--spec-draft-p-min | 0.00 | Minimum greedy probability accepted from the draft |
Step 6: Switch to EAGLE3 for a Higher Acceptance Rate
A trained EAGLE-3 head generally accepts more of its own proposed tokens than an independent, much-smaller draft model, because it was trained directly against the target model’s hidden states rather than being a general-purpose small model repurposed as a drafter. The tradeoff is an extra conversion step, since EAGLE heads are usually distributed as Hugging Face checkpoints rather than ready-made GGUF files.
Converting a Hugging Face EAGLE Head to GGUF
python convert_hf_to_gguf.py AngelSlim/Qwen3-4B_eagle3 \
--target-model-dir Qwen/Qwen3-4B \
--outtype bf16 \
--outfile Qwen3-4B-eagle3.gguf
./build/bin/llama-server \
-m Qwen3-4B.gguf \
-md Qwen3-4B-eagle3.gguf \
--spec-type draft-eagle3 \
--ctx-size 16384 \
--port 8080Note that this EAGLE-3 head is trained specifically against Qwen3-4B; it will not transfer to a different target model size or family. If you switch target models, you need a matching EAGLE head or you fall back to a standalone draft model or MTP.
Step 7: Configure Speculative Decoding in vLLM
vLLM v0.30.0 consolidated speculative decoding into a single --speculative-config JSON argument; the older separated --speculative-model and --num-speculative-tokens flags are deprecated. This is the path to use if you want an OpenAI-compatible production server rather than a single-user llama.cpp instance. Full reference is in the vLLM speculative decoding documentation.
vllm serve Qwen/Qwen3-8B \
--speculative-config '{
"method": "draft_model",
"model": "Qwen/Qwen3-0.6B",
"num_speculative_tokens": 5
}'For a model that ships built-in MTP layers, such as DeepSeek-V3, DeepSeek-R1, or GLM-4.7, skip the draft model entirely and just set the method to mtp:
vllm serve deepseek-ai/DeepSeek-V3 \
--speculative-config '{
"method": "mtp",
"num_speculative_tokens": 4
}'If you want to pair a draft model from a different family than your target — something llama.cpp does not support cleanly — set use_heterogeneous_vocab: true inside the same JSON block. This enables vLLM’s Token-Level Intersection algorithm, which reconciles mismatched tokenizers instead of rejecting the pair outright. Expect a lower acceptance rate than a same-family pair, but it is a legitimate option when no matching small model exists in your target’s family.
Step 8: Turn On Native MTP Speculative Decoding in Ollama
Ollama v0.32.6, released August 5, 2026, added MTP-head-based automatic speculative decoding along with improved OpenAI protocol compatibility for /v1/chat/completions. In Ollama’s design, speculative decoding is attached to the model definition rather than passed as a server-launch flag — you declare a DRAFT directive inside a Modelfile and Ollama handles the rest.
# Modelfile
FROM qwen3.6:8b
DRAFT qwen3.6:0.6bollama create qwen3.6-spec -f Modelfile
ollama run qwen3.6-specFor models with a native MTP head — Qwen3.6 and Gemma 4 both ship one — Ollama can enable speculative decoding automatically without a separate DRAFT line at all once you are on v0.32.6 or newer. Gemma 4’s MTP path currently runs through Ollama’s MLX runner, so it applies to Apple Silicon rather than the CUDA backend. Verify your installed version before relying on automatic detection, since the flag naming changed more than once during 2026’s release cycle.
ollama --version
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen3.6-spec",
"messages": [{"role": "user", "content": "Write a Python function that reverses a linked list."}]
}'Ollama’s OpenAI-compatible endpoint lives at http://localhost:11434/v1, so any existing OpenAI SDK client code can point at your speculative-decoding-enabled model with just a base URL change. If you plan on exposing this endpoint beyond localhost, set an OLLAMA_API_KEY environment variable first — see Step 12 for hardening this further.
Step 9: Use the One-Toggle Path in LM Studio
If a CLI is not your preferred workflow, LM Studio made MTP speculative decoding a stable, fully supported feature in version 0.4.14 (May 2026), promoting it out of the experimental toggle it started as. Download the target model through LM Studio’s model browser, open its load settings, and enable the speculative decoding switch — LM Studio will either use the target’s built-in MTP head automatically or prompt you to pick a compatible draft model from the same family if one is available locally.
LM Studio also exposes an OpenAI-compatible local server, so once speculative decoding is toggled on for a loaded model, every downstream tool pointed at LM Studio’s local API benefits without any client-side changes.
Step 10: Benchmark Tokens Per Second Before and After
Do not trust a speculative decoding setup until you have measured it against the same target model running without a draft. Run the same prompt at the same context length both ways and compare wall-clock generation time. llama.cpp’s built-in llama-bench tool handles this cleanly.
# Baseline, no speculation
./build/bin/llama-bench -m ./models/qwen3-8b-q6_k.gguf -p 512 -n 256
# With speculative decoding
./build/bin/llama-bench -m ./models/qwen3-8b-q6_k.gguf \
-md ./models/qwen3-0.6b-q4_k_m.gguf --spec-type draft-simple -p 512 -n 256Speedup depends heavily on how well-matched your draft and target are and on the nature of the prompt. Code generation, JSON extraction, and other structured outputs tend to see the largest gains, commonly in the 30–60% range for well-tuned standalone draft pairs, with EAGLE-3 and MTP setups sometimes reaching notably higher acceptance rates on repetitive text. Open-ended creative writing at high temperature typically sees the smallest gains, and can occasionally run slower than no speculation at all once verification overhead is accounted for. Treat any number you read online, including in this guide, as a starting expectation rather than a guarantee for your specific hardware and prompts.
Run the benchmark at least three times per configuration and average the results — a single run can be skewed by thermal throttling, background processes, or a prompt that happened to land on an unusually easy or unusually hard sequence for the draft model to predict. Record both the raw tokens-per-second figure and the acceptance rate llama-bench reports alongside it, since the two numbers together tell you whether a disappointing speedup is a draft-model-quality problem (low acceptance rate) or a configuration problem (high acceptance rate but still slow, which usually points back to VRAM pressure or an oversized --spec-draft-n-max).
Step 11: Tune Draft Length and Acceptance Rate
Once the baseline is working, the main lever you have is how many tokens the draft model proposes before the target model verifies. Too short and you leave speedup on the table; too long and you waste compute on guesses the target model was always going to reject.
- Start with
--spec-draft-n-max 16for llama.cpp and watch the acceptance rate reported in server logs. - If acceptance sits comfortably above 70%, try raising
--spec-draft-n-maxfurther — the draft model is reliable enough to guess longer runs. - If acceptance drops below 40%, lower
--spec-draft-n-maxor reconsider your draft/target pairing; a weak draft is actively slowing you down at that point. - In vLLM, adjust
num_speculative_tokensthe same way — lower it for creative, high-temperature workloads and raise it for deterministic, structured generation. - Greedy or low-temperature sampling produces the highest acceptance rates across every runtime; high-temperature creative sampling produces the lowest.
Step 12: Lock Down the Endpoint Before Exposing It Beyond Localhost
By default, llama-server, vLLM, and Ollama all bind to a local address and assume trusted callers. If you plan to reach the server from another machine on your network, add authentication before opening the port. Ollama’s OpenAI-compatible layer supports an OLLAMA_API_KEY environment variable for exactly this purpose, and llama-server accepts an --api-key flag.
export OLLAMA_API_KEY="a-long-random-string"
./build/bin/llama-server -m ./models/qwen3-8b-q6_k.gguf -md ./models/qwen3-0.6b-q4_k_m.gguf \
--spec-type draft-simple --host 0.0.0.0 --port 8080 --api-key "a-long-random-string"If you need remote access from outside your own network, put the server behind a reverse proxy with TLS rather than exposing the raw port directly — readers who have already set up a multi-model AI router with LiteLLM can drop a speculative-decoding-enabled local endpoint in as just another backend behind the same auth layer.
Step 13: Automate Startup With a Script or systemd Unit
Once your flags are dialed in, wrap them in a systemd unit so the speculative-decoding server survives reboots and restarts automatically if it crashes.
[Unit]
Description=llama-server with speculative decoding
After=network.target
[Service]
ExecStart=/home/user/llama.cpp/build/bin/llama-server \
-m /home/user/models/qwen3-8b-q6_k.gguf \
-md /home/user/models/qwen3-0.6b-q4_k_m.gguf \
--spec-type draft-simple --spec-draft-n-max 16 \
--host 127.0.0.1 --port 8080 -ngl 99
Restart=on-failure
User=user
[Install]
WantedBy=multi-user.targetSave this as /etc/systemd/system/llama-spec.service, then enable it with sudo systemctl enable --now llama-spec. Check status and logs with systemctl status llama-spec and journalctl -u llama-spec -f.
Common Pitfalls When Setting Up Speculative Decoding
- Mismatched tokenizers. Pairing a draft model from a different family than the target is the single most common cause of a working-but-useless setup — the server runs, but the acceptance rate is near zero, so you get no speedup and pay extra VRAM for nothing.
- Draft model too close in size to the target. A draft that is only two or three times smaller than the target spends nearly as much compute proposing tokens as the target spends verifying them, eating into or erasing the net gain.
- Ignoring VRAM headroom. Loading a second model on an already-full GPU forces partial CPU offload, which can make the “optimized” setup slower than the unoptimized baseline.
- Testing only on creative writing. If your benchmark prompt is open-ended fiction at high temperature, you will see a small or negative speedup and may wrongly conclude speculative decoding does not work — test on code or structured extraction first.
- Using an outdated llama.cpp, vLLM, or Ollama build. Flag names for speculative decoding changed more than once across 2026 releases; a command copied from an older tutorial or forum post may silently fail to parse on a newer build, or vice versa.
- Forgetting to quantize the draft model appropriately. A full-precision draft model next to a heavily quantized target defeats the purpose — the draft should generally be quantized at least as aggressively as the target.
- Assuming the same flags work across every runtime. llama.cpp’s
--spec-typefamily, vLLM’s--speculative-configJSON, and Ollama’s ModelfileDRAFTdirective are three unrelated interfaces to the same underlying idea — copying a llama.cpp command into a vLLM deployment script will simply fail to parse. - Skipping the acceptance-rate check after a model update. Updating either the draft or target model, even to a point release, can shift acceptance rate enough to change whether speculation is still worth the added complexity — re-run the Step 10 benchmark after any model swap, not just at initial setup.
Troubleshooting Speculative Decoding Problems
Work through these checks in order when speculative decoding does not behave as expected.
- Server refuses to start with an “unknown argument” error. Your llama.cpp, vLLM, or Ollama build predates the flags in this guide — rebuild or update to the versions listed in the prerequisites.
- No speedup observed at all. Check the server logs for the acceptance rate; if it is near zero, your draft and target likely do not share a tokenizer.
- Output quality changed after enabling speculative decoding. With standard verification, output should be mathematically identical to running the target model alone. If it is not, you are likely running a non-verified sampling mode — double-check you have not accidentally switched to a lossy speculative sampling variant.
- Out-of-memory error on load. Drop to a smaller draft model, quantize the draft more aggressively, or reduce
--ctx-sizefor both models. - vLLM rejects the old
--speculative-modelflag. That flag is deprecated as of v0.30.0 — migrate to the unified--speculative-configJSON syntax shown in Step 7. - Ollama Modelfile
DRAFTdirective is ignored. Confirm you are on Ollama v0.32.6 or later withollama --version; earlier builds do not recognize the directive. - Speedup is inconsistent between runs. Acceptance rate varies with prompt content — code and structured data accept more draft tokens than open-ended prose, so run-to-run variance is expected, not a bug.
- Multi-GPU setup is slower than single-GPU. Draft model round-trips add latency across device boundaries; keep the draft model on the same GPU as the first layers of the target model rather than splitting it across devices.
- EAGLE-3 conversion script fails. Confirm the
--target-model-dirpath points at the exact same base model the EAGLE head was trained against — EAGLE heads are not portable across model sizes or fine-tunes.
Advanced Tips for Multi-GPU and Production Setups
For production serving through vLLM, set draft_tensor_parallel_size explicitly rather than letting it inherit the target model’s parallelism — a tiny draft model rarely benefits from being sharded across multiple GPUs the way a 70B target does, and forcing it onto a single device cuts communication overhead. Combine speculative decoding with continuous batching carefully: verification cost scales with batch size, so the net benefit of speculation shrinks as concurrent request volume grows. For high-throughput multi-tenant serving, benchmark with a realistic concurrent request count rather than the single-request numbers from Step 10, since the two workloads behave differently.
If you are running a mixture-of-experts model like DeepSeek-V3 or GLM-4.7, prefer their native MTP heads over a standalone draft model whenever possible — MoE architectures complicate what “eight to ten times smaller” means for a compatible draft, and the model’s own team already tuned the MTP head against the specific expert routing behavior. Teams running a self-hosted image generation stack alongside a text model should keep the two workloads on separate GPUs where possible, since speculative decoding’s benefit depends on the target GPU having predictable memory bandwidth availability, which a second heavy workload on the same card will disrupt.
Monitor acceptance rate over time rather than checking it once at setup. A draft model tuned against one workload — say, a customer support bot mostly answering short factual questions — can see its acceptance rate drift if the workload shifts toward longer, more open-ended responses. Log the acceptance rate alongside your usual latency metrics so a regression shows up in monitoring rather than as a mysterious complaint that “the server feels slower than last month.” For teams running several speculative-decoding-enabled models behind one gateway, keep a per-model record of which draft or MTP configuration is attached, since it is easy to lose track of which target model is paired with which draft once you are managing more than two or three endpoints.
Finally, revisit your configuration whenever you update the target model’s weights, even for a minor point release. A fine-tune or safety patch can shift the target model’s token distribution enough to change how often it agrees with a draft model that scored well against the previous checkpoint. This is less of a concern for native MTP heads, which typically ship and version together with the target model, and more of a concern for standalone draft models pulled from a separate repository on its own release schedule.
A Complete Working Config You Can Copy
Here is a full, working reference setup combining everything above: a Qwen3.6 8B target model with its 0.6B counterpart as draft, served through llama-server with authentication, and wrapped in a systemd unit for persistence.
# 1. Build llama.cpp
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc)
# 2. Download models
huggingface-cli download Qwen/Qwen3.6-8B-Instruct-GGUF qwen3.6-8b-q6_k.gguf --local-dir ./models
huggingface-cli download Qwen/Qwen3.6-0.6B-Instruct-GGUF qwen3.6-0.6b-q4_k_m.gguf --local-dir ./models
# 3. Test manually before installing as a service
./build/bin/llama-server \
-m ./models/qwen3.6-8b-q6_k.gguf \
-md ./models/qwen3.6-0.6b-q4_k_m.gguf \
--spec-type draft-simple --spec-draft-n-max 16 --spec-draft-n-min 4 \
--ctx-size 32768 --host 127.0.0.1 --port 8080 --api-key "a-long-random-string" -ngl 99
# 4. Confirm it answers
curl http://127.0.0.1:8080/v1/chat/completions \
-H "Authorization: Bearer a-long-random-string" \
-H "Content-Type: application/json" \
-d '{"model": "qwen3.6-8b", "messages": [{"role": "user", "content": "Write a haiku about GPUs."}]}'Once this responds correctly and llama-bench confirms a real speedup over the no-draft baseline, install the systemd unit from Step 13 so it runs unattended. If you later want this model reachable from a broader agent stack, pair it with the same router pattern used in a production embeddings pipeline or fall it back behind a cloud model such as Grok 4.7 or Claude Fable 5.1 for requests your local hardware cannot handle. For a broader view of which open-weight models are worth pairing with a draft in the first place, the site’s open source LLM rankings and the AI models hub track which checkpoints ship native MTP support as new releases land.
Frequently Asked Questions
Does speculative decoding change the model’s output?
No, when implemented correctly with standard rejection sampling, the target model’s verification step guarantees output distributionally identical to running it alone without a draft. Speculative decoding changes speed, not quality.
Can I use any small model as a draft for any large model?
No. The draft and target need to share a tokenizer for standard implementations in llama.cpp and Ollama. vLLM’s Token-Level Intersection mode relaxes this for cross-family pairs, but same-family pairs still perform better.
Do I need a second GPU to run a draft model?
No. Draft models are intentionally small (typically 0.5B–4B parameters) and are designed to share the same GPU as the target model, adding only a few gigabytes of VRAM overhead.
Which is easier to set up, MTP or a standalone draft model?
MTP is generally easier, since it requires no second model download and no tokenizer-matching decisions — the drafting head ships inside the target checkpoint. It is only available for model families that specifically trained one in, such as Qwen3.6, Gemma 4, DeepSeek-V3/R1, and GLM-4.7.
Why did speculative decoding make my server slower?
Usually a mismatched or oversized draft model, a batch size high enough that verification overhead outweighs the savings, or a workload like high-temperature creative writing where acceptance rates are naturally low.
Does Ollama support speculative decoding out of the box?
As of v0.32.6 (August 5, 2026) and later, yes, through native MTP support for compatible models and the DRAFT Modelfile directive for standalone draft models. Earlier Ollama versions do not recognize either mechanism.
Is speculative decoding worth setting up for casual chatbot use?
It depends on your workload. For code assistance, structured data extraction, and factual Q&A, the acceptance rate is usually high enough to notice. For open-ended creative chat, the gain is smaller and sometimes not worth the added setup complexity of managing two models.
Related Coverage
- Microsoft Agent Framework Setup: 13 Steps, 90 Min [2026]
- Set Up OpenAI Embeddings API: 13 Steps, 90 Min [2026]
- Veo 3.1 API Setup: Migrate From Sora 2 in 12 Steps [2026]
- Grok 4.7 API Setup: 12 Steps, $2/$6 per 1M Tokens [2026]
- Hy Image 3.5 API Setup: 12 Steps, $0.024/Image [2026]
