Instructions to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0 # Run inference directly in the terminal: llama cli -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0 # Run inference directly in the terminal: llama cli -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Use Docker
docker model run hf.co/xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
- Ollama
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with Ollama:
ollama run hf.co/xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
- Unsloth Desktop
- Pi
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with Docker Model Runner:
docker model run hf.co/xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
- Lemonade
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Run and chat with the model
lemonade run user.gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf-Q4_0
List all available models
lemonade list
- Hermes Agent
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf:Q4_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Gemma 4 26B A4B-it QAT assistant (drafter), exact Q4_0 GGUF (unofficial)
This is an unofficial build, made and published independently of Google. It is a llama.cpp
GGUF of Google's speculative-decoding drafter for Gemma 4 26B A4B-it QAT,
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant,
with every weight matrix stored at 4 bits on the grid quantization-aware training produced. It is
251,939,648 B, against 461,785,920 B for the Q8_0 file llama.cpp's converter writes. Google's
model card is kept unchanged in ORIGINAL_README.md.
Use it with a Gemma 4 26B A4B-it QAT target, for example
google/gemma-4-26B-A4B-it-qat-q4_0-gguf.
Google's card says the drafter must match the target's QAT precision.
The 26B target is 14.4 GB, so on an 8 GB card its experts go to system RAM with -ncmoe
(the experts of the first N of 30 layers stay on the CPU). The setting this was measured at:
llama-server -m gemma-4-26B_q4_0-it.gguf -ngl 99 -ncmoe 21 -fa 1 -c 131072 -ctk q4_0 -ctv q4_0 -t 8 \
-md gemma-4-26B-A4B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2
Measured on llama.cpp build 7182ef71e only.
| Q8_0 (llama.cpp converter) | This GGUF | |
|---|---|---|
| File size | 461,785,920 B | 251,939,648 B |
| VRAM added by the drafter (nvidia-smi) | 508 MiB | 308 MiB |
| Generation with drafter, 26B target, RTX 3050 | 35.6 tok/s (1.20x) | 36.8 tok/s (1.24x) |
| Drafted tokens accepted | 0.657 | 0.654 |
Without a drafter the target runs at 29.6 tok/s. This file is 1.03x the Q8_0 one at the same acceptance.
| drafted tokens per step | 1 | 2 | 3 |
|---|---|---|---|
| Generation, x no drafter (greedy) | 1.25x | 1.24x | 1.01x |
| Generation, x no drafter (temperature 0.8) | 1.25x | 1.22x | 0.61x |
| Drafted tokens accepted | 0.760 | 0.654 | 0.579 |
Per prompt at 2 drafted tokens: code 1.51x, facts 1.43x, explanation 1.17x, chat 1.16x, story 1.12x, summary 1.12x.
The gain is smaller than on the dense 12B (2.01x for its drafter on the same card). With the experts on the CPU, each drafted token the target checks routes to its own 8 of 128 experts, which the CPU then reads; that likely limits the gain here (inferred, not measured). Use 1 or 2 drafted tokens, not 3.
Measured on an RTX 3050 OEM (8 GB) in an i7-5820K desktop (6 cores, quad-channel DDR4), llama.cpp
7182ef71e, target google/gemma-4-26B-A4B-it-qat-q4_0-gguf with -ncmoe 20 -t 8, flash
attention, f16 KV cache, -c 8192: six prompts (code, chat, story, explanation, facts, summary),
256 tokens, greedy, median of three runs, geometric mean over prompts. No drafter ran first and
last, at 29.6 and 28.7 tok/s.
On an 8 GB card the drafter costs context. With a q4_0 KV cache, the target loads at 131,072 tokens with 21 expert layers in RAM and this drafter; without a drafter, 262,144 fits with 22 in RAM. 262,144 does not load with the drafter even at 24.
Greedy text changes. With the drafter, greedy replies differ from the target's greedy replies without a drafter in 18 of 18 at 2 drafted tokens (12 of 18 at 1). The target still checks every drafted token; the differences likely come from rounding in the batched check (inferred, not measured). Answer quality with this drafter has not been measured on the 26B; for the 12B's drafter, built the same way, GSM8K and BFCL scored level with and without it.
What is in it
- Built from
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistantat revision9537141: 4 transformer layers of width 1024 (3 sliding-window, 1 full attention), with queries only; keys and values come from the target's cache. Two projections join it to the target's 2816-wide hidden state. The output head is the tied 262,144-token embedding. The drafter itself is dense; only the target is a mixture of experts. - Every weight matrix as Q4_0 (the embedding, 4 x attention q/o, 4 x MLP gate/up/down, both projections), each block's trained step recovered: largest magnitude divided by 8, 7, … 1, whichever puts all 32 values on an integer level, refined by least squares and stored as the fp16 block scale. No block was off the grid. 96.6–97.0% of values per tensor reproduce the bf16 source exactly once rounded back to bf16.
- Metadata (hyperparameters, tokenizer, chat template) is llama.cpp's converter output,
general.file_typeset to Q4_0. Norms and scales stay F32. - Text only. The drafter has no vision or audio parts.
Files
| file | bytes | SHA-256 |
|---|---|---|
gemma-4-26B-A4B-it-qat-assistant-q4_0-exact.gguf |
251,939,648 | 6cee35a091e6b7572be5b972fc994cde0c95dfa392466bdac0c184fe9060090b |
evidence/ holds the build script (gguf_exact.py), its per-tensor log and report
(build.log, build-report.json), the speed run's script and per-prompt table (spec_ab.py,
speed-summary.txt) and the converter log. The full run is in
local-llamacpp-3050oem-26b-q4_0.
License
Apache 2.0, as Google's source. Gemma 4 terms: https://ai.google.dev/gemma/docs/gemma_4_license
- Downloads last month
- 989
4-bit
Model tree for xbill9/gemma-4-26B-A4B-it-qat-assistant-q4_0-exact-gguf
Base model
google/gemma-4-26B-A4B-it-assistant