GRPO is the training method behind most of the reasoning-model headlines of the past year, and as of October 26, 2026 you can run it on a single GPU with open tools. This tutorial walks through 13 steps that take you from an empty folder to a LoRA adapter that learns to solve arithmetic word problems by reasoning first and answering second. The stack is Hugging Face TRL, PEFT, vLLM and one base model, with no reward model and no human labels.
The key fact for planning is simple. TRL ships GRPO next to SFT, DPO and reward training, so you do not need a second framework to try it. The trainer class is GRPOTrainer, and Hugging Face’s October 2026 post-training curriculum reportedly still lists it as the trainer for reward-function reinforcement learning.
Why bother now? Open-weight releases keep getting bigger, and the cheapest way to make a small model good at one narrow job is reinforcement learning against a check you can write in code. If you have read about how often GLM-5.3 reasons about its own training, you already know that reward design is where things get interesting. Here you will write the rewards yourself, test them offline, train, and then measure whether the model improved.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What GRPO Does and How It Differs From PPO
Group Relative Policy Optimization, or GRPO, came out of the DeepSeekMath work and later became a core ingredient in DeepSeek-R1 style reasoning training. The idea is short. For each prompt, the model writes a group of answers. A reward function scores every answer. The trainer then compares each answer to the average of its own group and pushes the model toward the above-average ones.
PPO, the older workhorse of RLHF, needs a separate value network (the critic) to estimate how good a state is. That critic adds a second large model to the memory bill and a second set of knobs to tune. GRPO replaces the critic with group statistics. The advantage of an answer is its reward minus the group mean, divided by the group standard deviation. No critic, no extra weights, and that is why one GPU is enough for small models.
advantage_i = (reward_i - mean(rewards_in_group)) / (std(rewards_in_group) + eps)
# Example group of 4 answers to one prompt:
# rewards = [2.5, 0.0, 2.5, 0.5]
# mean = 1.375, std ~ 1.14
# advantages = [ +0.99, -1.21, +0.99, -0.77 ]
# The two correct answers get pushed up, the two weak ones pushed down.Readers who search for the GRPO formula or for GRPO vs PPO usually want exactly this picture. Two consequences matter in practice. First, if every answer in a group gets the same reward, the standard deviation is zero and that group teaches the model nothing. Second, the reward function is the whole curriculum. Whatever you score is what the model learns, including your mistakes.
TRL is now in the v1.x generation, and GRPOTrainer remains an actively supported trainer for reinforcement-learning post-training. TRL’s documentation lists vLLM-powered generation, PEFT/LoRA and vision-language training as supported in GRPOTrainer. That combination of LoRA plus vLLM is what makes this tutorial possible on one card. If you want the original GRPO paper for the derivation, read the DeepSeekMath paper alongside this guide and compare its notation with the formula above.
The October 2026 Stack: Versions That Work Together
GRPO tutorials age fast because TRL ships fast. A recipe from last spring may fail on today’s release, and a recipe from today may silently train wrong on an older one. TRL v1.14.2, published October 6, 2026, reportedly fixed two cases of silently incorrect training and three crashes, with GRPO, RLOO and Distillation affected. Treat anything older than that release as a version to skip.
The table below lists the pins this tutorial uses and the reason for each. The TRL version comes from its own GitHub release page. For PEFT and vLLM, the text after the table explains what secondary sources report, so confirm both on the package index before you pin.
| Component | Pin in this tutorial | Why that pin |
|---|---|---|
| TRL | 1.16.0 | Released Oct. 15, 2026 with AWS Trainium support and a prefix-tree speedup for AsyncGRPO |
| PEFT | 0.21.x | Matches the 0.21 line in the PEFT docs |
| vLLM | 0.31.x | The newest line TRL’s supported range reportedly reached in October |
| Transformers | 5.18.0 or later | PEFT and TRL release notes cite 5.18.0 as the floor for a recent compatibility fix |
| Python | 3.11 | PEFT’s release line requires 3.10 or later |
| Base model | Qwen/Qwen3-4B-Instruct-2507 | Small enough for one card, with an instruct format that works with chat prompts |
Two details sit behind those pins. One secondary source reports that TRL raised its supported vLLM ceiling from 0.30.0 to 0.31.0 on October 7, 2026. A Hugging Face community article from early October also describes LoRA and QLoRA runs on Qwen3-4B-Instruct-2507 on a single NVIDIA GB10 workstation with TRL, which is why that model is a sensible default here.
One more current-state note. Hugging Face now offers two GRPO trainers: the synchronous GRPOTrainer and the experimental AsyncGRPOTrainer, which splits generation and training for agent-scale jobs. According to a Hugging Face post dated September 10, 2026, LoRA support for the async trainer, including adapter-only sync to vLLM, ships in TRL v1.14. This tutorial uses the synchronous trainer because it fits one GPU and is simpler to debug. The async path returns under advanced tips.
Bigger newcomers do not change the recipe, they change the scale. Reflection’s 501B Beam model was announced on October 16, 2026. Hugging Face’s Beam post describes LoRA and GRPO recipes, but its examples run on multi-node H200 hardware. Treat them as proof the method scales, not as something to copy onto your desktop.
Prerequisites and Hardware Budget
You need a Linux machine (or WSL2) with an NVIDIA GPU, a working driver, and Python 3.11. Budget about 40 GB of free disk for the model, checkpoints and a vLLM cache. A free Hugging Face account helps for downloads, though the base model used here is open. Basic Python and some comfort with the command line are enough, since every step ships with code you can paste.
Sizing is where most first attempts fail, so here is the arithmetic. A 4-billion-parameter model in bfloat16 takes roughly 8 GB for weights alone. With the colocated vLLM engine, you hold a second copy of those weights for fast generation, plus a KV cache for every sequence in the group. These are my sizing estimates from that arithmetic, not vendor specs or measured benchmarks, so expect to adjust them on your card.
| GPU memory class | Suggested profile | Settings to change |
|---|---|---|
| 24 GB | Tight. Use a smaller base model or the reduced profile | Not listed |
| 40 to 48 GB | Default profile in this tutorial | Not listed |
| 80 GB | Comfortable, room for longer reasoning | max_completion_length 768, num_generations 16 |
| No local GPU | Rent one for the run | Hugging Face lists on-demand GPU hardware starting at $0.60/hour, and larger cards cost more |
NVIDIA’s current GeForce flagship is the RTX 5090, and it suits this job, but any recent CUDA card with enough memory works. If you only have a laptop, skip ahead to the cost section and rent a card for the training run.
Steps 1 to 3: Environment, Project Layout and Version Check
Step 1: Create an isolated environment. Use uv or plain venv. A clean environment matters more for GRPO than for most tutorials, because TRL, vLLM and PyTorch pin each other tightly and a stray package can swap a CUDA build under you.
mkdir grpo-arith && cd grpo-arith
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
# Install vLLM first so it picks the matching torch build
pip install "vllm==0.31.*"
pip install "trl==1.16.0" "peft>=0.21,<0.22" "transformers>=5.18" datasets accelerateInstalling vLLM first lets it choose the PyTorch build it was compiled against. If pip reports a conflict, do not force it. Read the message, then relax the pin that conflicts instead of the one that matters most (vLLM).
Step 2: Lay out the project. Four small files keep the pieces testable on their own. The data generator, the rewards, the trainer and the evaluator each get a file.
grpo-arith/
make_data.py # builds data/train.jsonl and data/eval.jsonl
rewards.py # three reward functions, testable without a GPU
train.py # GRPOConfig + GRPOTrainer + LoRA
evaluate.py # greedy accuracy for base vs adapter
data/
runs/Step 3: Verify the versions and the GPU. Run this before anything else. It takes ten seconds and catches the two commonest failures, a CPU-only torch and a TRL older than the October fixes.
python - <<'EOF'
import torch, trl, peft, transformers, vllm
print("torch", torch.__version__, "cuda", torch.version.cuda, "gpu", torch.cuda.is_available())
print("trl", trl.__version__, "| peft", peft.__version__)
print("transformers", transformers.__version__, "| vllm", vllm.__version__)
if torch.cuda.is_available():
p = torch.cuda.get_device_properties(0)
print(p.name, round(p.total_memory / 1e9, 1), "GB")
from trl import GRPOConfig
fields = GRPOConfig.__dataclass_fields__
print([f for f in fields if "vllm" in f])
EOFExpected output looks like the block below. Your numbers will differ, so read it for shape rather than exact values. You want gpu True, a TRL at or above 1.14.2, and a list of vLLM-related fields that includes use_vllm and vllm_mode.
torch 2.x.x cuda 12.x gpu True
trl 1.16.0 | peft 0.21.x
transformers 5.x.x | vllm 0.31.x
NVIDIA ... 48.0 GB
['use_vllm', 'vllm_mode', 'vllm_gpu_memory_utilization', ...]That last line is a habit worth keeping. Argument names drift between TRL releases, and printing the real field list on your pinned version beats trusting any tutorial, including this one. If vllm_mode is missing on your version, check the TRL docs for the current name of the colocate switch before you continue.
Steps 4 and 5: Build a Dataset With Checkable Answers
GRPO needs prompts and a way to score answers. It does not need target reasoning text. That makes it a good fit for tasks with a verifiable result: math, code that passes tests, structured extraction, anything a script can grade. We generate arithmetic word problems with integer answers because a Python function can check them exactly.
Step 4: Write the generator. The prompt column holds chat messages and the answer column holds the gold integer as a string. TRL passes every extra column to your reward functions as keyword arguments, which is how the correctness reward will see the gold answer.
# make_data.py
import json
import random
from pathlib import Path
SYSTEM = (
"You solve arithmetic word problems. Think step by step inside "
"<reasoning></reasoning> tags, then give only the final integer "
"inside <answer></answer> tags."
)
def shop(rng):
crates, per, mon = rng.randint(4, 20), rng.randint(6, 24), rng.randint(5, 40)
tue = rng.randint(5, 40)
total = crates * per
text = (f"A store receives {crates} crates with {per} bottles each. "
f"It sells {mon} bottles on Monday and {tue} on Tuesday. "
f"How many bottles are left?")
return text, total - mon - tue
def trip(rng):
s1, h1, s2, h2 = rng.randint(30, 90), rng.randint(1, 5), rng.randint(30, 90), rng.randint(1, 5)
text = (f"A van drives {s1} km per hour for {h1} hours, then {s2} km per hour "
f"for {h2} hours. How many kilometers does it drive in total?")
return text, s1 * h1 + s2 * h2
def savings(rng):
week, weeks = rng.randint(8, 40), rng.randint(4, 20)
spend = rng.randint(5, week * weeks // 2)
text = (f"Maya saves {week} dollars a week for {weeks} weeks, spends {spend} dollars "
f"on a book, and then a gift doubles what she has left. "
f"How many dollars does she have now?")
return text, (week * weeks - spend) * 2
def build(n, seed):
rng = random.Random(seed)
rows = []
for _ in range(n):
text, ans = rng.choice([shop, trip, savings])(rng)
rows.append({
"prompt": [{"role": "system", "content": SYSTEM},
{"role": "user", "content": text}],
"answer": str(ans),
})
return rows
if __name__ == "__main__":
Path("data").mkdir(exist_ok=True)
for name, n, seed in [("train", 2000, 1), ("eval", 200, 999)]:
with open(f"data/{name}.jsonl", "w") as f:
for row in build(n, seed):
f.write(json.dumps(row) + "\n")
print("wrote data/train.jsonl and data/eval.jsonl")Step 5: Generate and inspect. Always read a few rows by eye. A generator bug that produces negative answers or impossible quantities will poison every reward downstream, and you will blame the trainer.
python make_data.py
head -n 2 data/train.jsonl
wc -l data/*.jsonlwrote data/train.jsonl and data/eval.jsonl
{"prompt": [{"role": "system", "content": "You solve arithmetic word problems. ..."}, {"role": "user", "content": "A store receives 12 crates with ..."}], "answer": "187"}
200 data/eval.jsonl
2000 data/train.jsonlThe train and eval files use different random seeds, so the evaluation set is not a copy of the training set. With only three templates the numbers differ but the problem types repeat. This project therefore shows that GRPO works, not that the model learned general math. Swap in your own task once the pipeline runs.
Steps 6 and 7: Write Reward Functions and Test Them Offline
Step 6: Write three small rewards. Splitting the reward into parts gives you a log line per part, so you can see which behavior is improving. Each function takes the batch of completions plus any dataset columns and returns one float per completion. For chat-format prompts, each completion arrives as a list holding one assistant message, which the helper unwraps.
# rewards.py
import re
FORMAT_RE = re.compile(
r"^\s*<reasoning>.+?</reasoning>\s*<answer>\s*-?\d+\s*</answer>\s*$",
re.DOTALL,
)
ANSWER_RE = re.compile(r"<answer>\s*(-?\d+)\s*</answer>")
def _text(completion):
if isinstance(completion, list):
return completion[0]["content"]
return completion
def format_reward(completions, **kwargs):
"""0.5 when the whole completion follows the reasoning-then-answer layout."""
return [0.5 if FORMAT_RE.match(_text(c)) else 0.0 for c in completions]
def answer_tag_reward(completions, **kwargs):
"""0.2 when an integer answer tag exists anywhere, a dense early signal."""
return [0.2 if ANSWER_RE.search(_text(c)) else 0.0 for c in completions]
def correctness_reward(completions, answer, **kwargs):
"""2.0 when the last answer tag equals the gold integer."""
scores = []
for completion, gold in zip(completions, answer):
found = ANSWER_RE.findall(_text(completion))
scores.append(2.0 if found and found[-1] == str(gold) else 0.0)
return scoresThe weights encode priorities. Correctness is worth four times the format reward, so the model cannot get rich by formatting alone. The tag reward exists because early in training a 4B instruct model often gets the layout wrong, and a group where every answer scores zero gives no gradient. A small dense reward breaks that tie.
Good reward design follows three habits. Score outcomes you can verify, not styles you can only guess at. Keep each reward function small enough that you can explain its failure modes in a sentence. And give partial credit only where partial progress is real, because every partial credit is a door the model can walk through without solving the task.
Step 7: Unit-test the rewards without a GPU. This is the cheapest debugging you will do. Feed the functions a perfect answer, a right answer in the wrong layout, a wrong answer, and garbage. If the numbers surprise you, fix the rewards before you spend an hour of GPU time on them.
python - <<'EOF'
from rewards import format_reward, answer_tag_reward, correctness_reward
def msg(t):
return [{"role": "assistant", "content": t}]
cases = {
"perfect": msg("<reasoning>12*3=36, 36-6=30</reasoning><answer>30</answer>"),
"right-no-fmt": msg("The result is <answer>30</answer> I think."),
"wrong": msg("<reasoning>guess</reasoning><answer>31</answer>"),
"garbage": msg("I do not know."),
}
batch = list(cases.values())
gold = ["30"] * len(batch)
for name, f in [("format", format_reward), ("tag", answer_tag_reward)]:
print(name, f(batch))
print("correct", correctness_reward(batch, gold))
EOFformat [0.5, 0.0, 0.5, 0.0]
tag [0.2, 0.2, 0.2, 0.0]
correct [2.0, 2.0, 0.0, 0.0]Read that output closely. The perfect answer scores 2.7 in total. The right-answer-wrong-layout case scores 2.2, so correctness still dominates. The wrong-but-formatted answer scores 0.7, which is the number to watch. If a model can reliably earn 0.7 by writing nonsense in the right tags, the trainer will find that shortcut quickly. That is reward hacking in miniature, and it is why we weighted correctness so heavily.
Steps 8 and 9: Configure GRPOConfig and the LoRA Adapter
Step 8: Set the LoRA adapter. LoRA trains small low-rank matrices beside the frozen weights, which keeps optimizer state tiny and lets you save a checkpoint measured in megabytes. Rank 32 with all-linear targets is a reasonable starting point for a 4B model. Because LoRA learns more slowly per step than full fine-tuning, the learning rate sits higher than a full-parameter GRPO run would use.
One change from this month matters here. According to TRL’s 1.15.0 release notes (October 8, 2026), attaching a PEFT adapter directly to lm_head now raises an error, and the fix is modules_to_save=["lm_head"] if you need that layer trained. We skip lm_head entirely, so the config below is safe.
Step 9: Build the trainer arguments. Every value below has a reason, and the comment beside each line gives it in a few words.
# train.py
from datasets import load_dataset
from peft import LoraConfig
from trl import GRPOConfig, GRPOTrainer
from rewards import answer_tag_reward, correctness_reward, format_reward
BASE = "Qwen/Qwen3-4B-Instruct-2507"
data = load_dataset("json", data_files={
"train": "data/train.jsonl",
"eval": "data/eval.jsonl",
})
peft_config = LoraConfig(
r=32,
lora_alpha=64,
lora_dropout=0.0,
target_modules="all-linear", # attention + MLP projections, not lm_head
task_type="CAUSAL_LM",
)
args = GRPOConfig(
output_dir="runs/grpo-arith",
learning_rate=2e-5, # LoRA tolerates a higher rate than full fine-tuning
lr_scheduler_type="cosine",
warmup_ratio=0.05,
per_device_train_batch_size=8, # 8 completions per device per micro-step
gradient_accumulation_steps=4, # 8 * 4 = 32 completions per optimizer step
num_generations=8, # group size, must divide the batch above
max_completion_length=384, # room for reasoning, short enough to stay cheap
temperature=1.0, # keep diversity inside each group
beta=0.0, # no KL penalty, no reference model in memory
loss_type="dapo", # pinned so a default change cannot alter the run
mask_truncated_completions=True, # do not reward answers cut off mid-thought
max_steps=150, # the 70-minute path, raise for better results
bf16=True,
gradient_checkpointing=True,
use_vllm=True,
vllm_mode="colocate", # vLLM shares the training GPU
vllm_gpu_memory_utilization=0.35,
logging_steps=5,
save_steps=50,
log_completions=True,
report_to="none",
seed=42,
)
trainer = GRPOTrainer(
model=BASE,
reward_funcs=[format_reward, answer_tag_reward, correctness_reward],
args=args,
train_dataset=data["train"],
eval_dataset=data["eval"],
peft_config=peft_config,
)
if __name__ == "__main__":
trainer.train()
trainer.save_model("runs/grpo-arith/final")Three settings deserve a longer note. Setting beta=0.0 drops the KL penalty, which means TRL does not load a frozen reference copy of the model. That saves several gigabytes and matches the direction recent GRPO recipes have taken. It also removes a guardrail, so watch for drift in the logs. Pinning loss_type matters because one unconfirmed report says the trainer’s default moved to the DAPO loss in August 2026. Treat that as a rumor. Pinning the value removes the question.
The third is the batch arithmetic. TRL needs the number of completions generated per step to divide evenly by num_generations. Here 8 per device times 4 accumulation steps gives 32, and 32 divided by 8 is 4 prompts per optimizer step. Change one number and you must recheck the others. Mismatches are the single most common startup error in GRPO runs.
Step 10: Launch Training and Read the Logs
Step 10: Start the run. A plain python train.py works for one GPU. Use a terminal multiplexer such as tmux so a dropped SSH session does not kill a half-finished run.
tmux new -s grpo
source .venv/bin/activate
python train.py 2>&1 | tee runs/train.logStartup takes a few minutes. The trainer loads the model, boots a colocated vLLM engine, and compiles kernels. Then the step logs begin. The block below shows the shape of a healthy early log. The numbers are illustrative, not a benchmark, and your values will move differently.
{'loss': 0.012, 'reward': 0.94, 'reward_std': 0.71,
'rewards/format_reward/mean': 0.11, 'rewards/answer_tag_reward/mean': 0.14,
'rewards/correctness_reward/mean': 0.69,
'completions/mean_length': 241.3, 'completions/clipped_ratio': 0.08,
'frac_reward_zero_std': 0.12, 'learning_rate': 2e-05, 'epoch': 0.08}
...
{'loss': 0.004, 'reward': 1.88, 'reward_std': 0.52,
'rewards/format_reward/mean': 0.43, 'rewards/answer_tag_reward/mean': 0.19,
'rewards/correctness_reward/mean': 1.26,
'completions/mean_length': 188.9, 'completions/clipped_ratio': 0.01,
'frac_reward_zero_std': 0.31, 'learning_rate': 1.1e-05, 'epoch': 1.2}Here is what to read in that log, in priority order.
- reward and the per-function means. The overall mean should climb, and the correctness mean should climb faster than the format mean once layout is learned.
- reward_std. This is the learning signal. If it falls to near zero early, groups are all-good or all-bad and the model stops learning.
- frac_reward_zero_std. The share of groups with identical rewards. Values rising toward 0.5 or higher mean your task is getting too easy or too hard for the sampling temperature.
- completions/clipped_ratio. The share of answers cut off at the length limit. Keep it under about 10 percent, or raise
max_completion_length. - completions/mean_length. A steady rise toward the cap means the model is rambling to chase reward. A collapse to a few tokens means it skips reasoning.
Because log_completions=True, the trainer also prints sample completions with their rewards. Read five of them every fifty steps. Numbers can look fine while the model learns a trick, and only the text shows you the trick. Checkpoints land every 50 steps in runs/grpo-arith, so a bad stretch costs you minutes, not the whole run.
Steps 11 and 12: Evaluate Against the Base Model and Serve the Adapter
Step 11: Measure improvement on held-out problems. Training reward tells you the model got better under sampling on the training distribution. The evaluation script runs greedy decoding on the 200 held-out problems for both the base model and the adapter, so you get an honest before-and-after comparison.
# evaluate.py
import json
import sys
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
from rewards import ANSWER_RE, FORMAT_RE
BASE = "Qwen/Qwen3-4B-Instruct-2507"
adapter = sys.argv[1] if len(sys.argv) > 1 else None
tok = AutoTokenizer.from_pretrained(BASE, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="cuda")
if adapter:
model = PeftModel.from_pretrained(model, adapter)
model.eval()
rows = [json.loads(line) for line in open("data/eval.jsonl")]
correct = formatted = 0
for i in range(0, len(rows), 16):
batch = rows[i:i + 16]
texts = [tok.apply_chat_template(r["prompt"], tokenize=False, add_generation_prompt=True) for r in batch]
enc = tok(texts, return_tensors="pt", padding=True).to("cuda")
with torch.no_grad():
out = model.generate(**enc, max_new_tokens=384, do_sample=False)
for r, seq in zip(batch, out):
reply = tok.decode(seq[enc["input_ids"].shape[1]:], skip_special_tokens=True)
found = ANSWER_RE.findall(reply)
correct += bool(found and found[-1] == r["answer"])
formatted += bool(FORMAT_RE.match(reply))
n = len(rows)
print(f"{'adapter' if adapter else 'base'}: accuracy {correct / n:.1%} | format {formatted / n:.1%} | n={n}")python evaluate.py
python evaluate.py runs/grpo-arith/final
# Illustrative output shape, your numbers will differ:
# base: accuracy 62.0% | format 18.5% | n=200
# adapter: accuracy 81.5% | format 96.0% | n=200Do not expect the exact gap above. Treat it as the shape you hope for: the format rate jumps first because it is the easiest thing to learn, and accuracy rises more slowly. If format hits 90 percent but accuracy stays flat, the model learned the layout and nothing else. That points back to Step 7 and to harder training problems.
Step 12: Serve the adapter or merge it. You have two options. For quick experiments, serve the base model once with vLLM and load the adapter on top, which also lets you hot-swap several adapters. Hugging Face documents this pattern for production with many LoRA adapters served from one endpoint, using settings such as SM_VLLM_ENABLE_LORA=true, SM_VLLM_MAX_LORA_RANK=16 and SM_VLLM_MAX_LORAS=4. Our adapter has rank 32, so raise the rank limit to match.
# Option A: base model plus adapter, OpenAI-compatible server
vllm serve Qwen/Qwen3-4B-Instruct-2507 \
--enable-lora --max-lora-rank 32 \
--lora-modules arith=runs/grpo-arith/final \
--max-model-len 4096
# Option B: merge once, serve a normal checkpoint
python - <<'EOF'
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen3-4B-Instruct-2507"
m = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)
m = PeftModel.from_pretrained(m, "runs/grpo-arith/final").merge_and_unload()
m.save_pretrained("merged-arith")
AutoTokenizer.from_pretrained(base).save_pretrained("merged-arith")
EOF
vllm serve ./merged-arith --max-model-len 4096curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "arith",
"messages": [
{"role": "system", "content": "Think inside reasoning tags, then answer inside answer tags."},
{"role": "user", "content": "A van drives 60 km per hour for 3 hours, then 40 km per hour for 2 hours. How many kilometers in total?"}
]
}'For Option A, the model name in the request is the adapter name you passed to --lora-modules. For Option B, use the path you served. Merge when you want one artifact to ship, and keep the adapter separate when you plan to train several variants on the same base.
Step 13 and the Complete Project: One Command From Scratch
Step 13: Make the run reproducible. A GRPO result you cannot repeat is a story, not a result. Pin the versions, fix the seed (done in the config), commit the four files, and write the exact commands in a script. The runner below executes the whole project from a fresh clone.
#!/usr/bin/env bash
# run_all.sh - complete project, fresh machine to evaluated adapter
set -euo pipefail
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install "vllm==0.31.*"
pip install "trl==1.16.0" "peft>=0.21,<0.22" "transformers>=5.18" datasets accelerate
mkdir -p runs
pip freeze > runs/requirements.lock
python make_data.py
python -c "from rewards import *; print('rewards import ok')"
python train.py 2>&1 | tee runs/train.log
python evaluate.py | tee runs/eval-base.txt
python evaluate.py runs/grpo-arith/final | tee runs/eval-adapter.txtThat is the complete working project: a data generator, three reward functions, a LoRA GRPO trainer, an evaluator and a runner. It comes to roughly 200 lines of Python. Everything you would change for your own task lives in two places, the generator (what the prompts are) and rewards.py (what counts as good).
The 70-minute budget in the title covers setup, data, reward tests and a 150-step run on a 40 to 48 GB card, plus a first evaluation. Longer runs take longer, and your GPU, driver and disk speed set the clock. Plan an unattended block for anything past 150 steps.
Common Pitfalls That Waste GPU Hours
Every one of these has cost someone a night of compute. The list is ordered by how often they bite first-time GRPO users.
1. Rewards the model cannot earn yet
If the base model scores zero on nearly every prompt, every group has zero variance and nothing trains. Run the base model through your rewards first (Step 11 with no adapter). Aim for a base accuracy somewhere between 20 and 70 percent. Too low, add dense partial rewards or easier problems. Too high, make the task harder.
2. Reward hacking through the format
The model optimizes the score, not your intent. A format reward that is too generous teaches tag spam. Keep format rewards small relative to correctness, and read sampled completions regularly. A quick look at raw text beats any dashboard for catching this.
3. A batch size that does not divide by num_generations
The trainer raises an error at startup if the completions per step do not split into whole groups. Do the multiplication by hand whenever you change the batch, accumulation steps or group size.
4. A vLLM version outside TRL’s supported window
TRL moves its supported vLLM range often and drops old releases without ceremony. One TRL change dropped vLLM 0.20.2 support, and reports say further removals of 0.20.x and 0.21.0 followed during October. Install the vLLM version your TRL release was tested against, and read the release notes when you upgrade either one.
5. Pinning an old TRL and trusting the numbers
The October 6 correctness fixes mean an older install can train wrongly without crashing. Reported issues included multiple end-of-turn token IDs and completion handling in GRPO. A smooth-looking run on an old pin proves nothing.
6. Attaching the adapter to lm_head
Since TRL 1.15.0 this raises an error. If you copied a LoRA config from an older guide with lm_head in target_modules, remove it and use modules_to_save where needed.
7. Letting vLLM starve training
In colocate mode both engines share one card. Set vllm_gpu_memory_utilization too high and training runs out of memory on the first backward pass. Set it too low and generation stalls on a tiny KV cache. Start on the low side on a 48 GB card and move in steps of 0.05.
8. Judging success on training reward alone
Training reward rises when the model learns the task and when it learns your loopholes. Only the held-out evaluation separates the two. Run Step 11 before you claim a win.
Troubleshooting: 10 Symptoms and Fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| CUDA out of memory right after vLLM starts | vLLM reserved too much of the shared GPU | Lower vllm_gpu_memory_utilization by 0.05, or cut per-device batch to 4 |
| Out of memory during the first backward pass | Long completions plus a large group | Reduce max_completion_length and num_generations, keep gradient checkpointing on |
| Error about generations and batch size | Completions per step not divisible by num_generations | Recompute per-device batch times accumulation, then adjust |
| Reward stays flat near zero | Task too hard, or rewards unreachable | Test the base model offline, add dense partial rewards, simplify problems |
| reward_std drops to near zero | Groups are uniformly right or wrong | Raise temperature slightly, mix in harder or easier prompts |
| Format reward high, accuracy flat | Model learned layout but not the task | Lower the format weight, raise problem difficulty, train longer |
| Reward climbs then collapses | Learning rate too high or reward hacking | Halve the learning rate, inspect sampled completions, restore a saved checkpoint |
| ImportError or vLLM crash on start | vLLM version outside TRL’s range, or a torch mismatch | Reinstall vLLM first, then TRL, and match the pins in the stack table |
| Error mentioning lm_head and PEFT | Adapter targets lm_head (TRL 1.15.0 and later) | Remove lm_head from target_modules, use modules_to_save if you need it trained |
| TypeError: unexpected keyword in reward function | Function signature lacks **kwargs or the answer column | Accept **kwargs and name the dataset column exactly, here answer |
Two extra checks solve many other cases. First, if a step takes minutes instead of seconds, confirm that use_vllm=True took effect and that you did not fall back to slow Hugging Face generation. Second, if the fused LM-head path complains about Triton, install Triton. TRL’s v1.15.0 notes describe the fused LM head as on by default and as needing Triton for GPU scoring. The same notes report it allows sequences up to 6.9 times longer on the same GPU by avoiding the full batch-by-sequence-by-vocabulary logits tensor.
Advanced Tips: Scaling Beyond the Single-GPU Recipe
Move to AsyncGRPOTrainer when generation dominates. The synchronous trainer idles the GPU while vLLM generates. The async trainer overlaps the two, and since TRL v1.14 it supports LoRA with adapter-only synchronization to vLLM. Hugging Face positions it for agent-scale jobs. It is still experimental, and an issue opened in the TRL repository proposes adding a quantization_config option to it, so expect API movement. Pin the version and re-read the docs before you switch.
Use a curriculum. Start with two-step problems, watch frac_reward_zero_std, and add three-step and four-step problems once it climbs. This keeps groups diverse. You can build it with a generator that takes a difficulty argument and a second training run that resumes from the first adapter.
Try environment-based RL. Hugging Face published its RL environments announcement on September 28, 2026, and its October 20 curriculum reportedly lists GRPOTrainer for reward-based RL and for environments. Coding-agent examples have moved to a Harbor integration in OpenEnv, replacing the older opencode examples. If your task needs tools or a sandbox, that is the road, though it needs more infrastructure than one GPU.
Test smaller or newer base models. NVIDIA’s Nemotron 3.5 Bolt E4B 4GB, a small open model, was reportedly released on October 23, 2026 and is a candidate for constrained hardware. Nothing in the sources I checked confirms it works with this exact TRL recipe, its chat template, or LoRA plus vLLM training, so run ten steps before you commit. The same warning applies to any model released in the last week.
Add a held-out reward audit. Write a second, stricter checker that the trainer never sees, and run it on samples during evaluation. If the training reward says 90 percent and the audit says 60, you found a loophole. This takes ten lines and catches more problems than any hyperparameter sweep.
Compare the loss variants. Run the same seed with loss_type="grpo" and loss_type="dapo", then compare held-out accuracy and completion length. A community article also describes sequence-level importance sampling (GSPO) through importance_sampling_level="sequence" as available since August 2026. I found that in a single article, so confirm it in your TRL docs first.
Watch completion length as a cost lever. Reasoning length drives both generation time and memory. If mean length creeps up with no accuracy gain, add a small length penalty or lower max_completion_length. Short, correct reasoning is cheaper to train and cheaper to serve afterward.
What a GRPO Run Costs and When to Rent a GPU
Money here is mostly time on a card. Hugging Face’s own pages list on-demand GPU hardware starting at $0.60 per hour, a Pro account at $9 a month, and dedicated inference starting at $0.033 per hour. Those are entry rates, and a 40 to 48 GB card costs more than the cheapest option. Check the current rate for the exact GPU before you start a run.
A sensible budget plan has three parts. Use a cheap or free machine for Steps 1 through 7, since none of them need a GPU except the version check. Rent the large card only for training and evaluation. Then shut it down. Save the adapter and the log to the Hub or to local disk first, because a stopped rental often wipes its disk.
If you already run models locally, much of the setup carries over from our AI and machine learning coverage of inference stacks. The training side adds reward design, which no hardware upgrade replaces.
Frequently Asked Questions About GRPO
What does GRPO stand for?
Group Relative Policy Optimization. It is a reinforcement-learning method that scores a group of answers per prompt and uses the group average as the baseline, so it needs no separate critic model.
How is GRPO different from PPO?
PPO trains a value network to estimate expected reward. GRPO drops that network and normalizes rewards within each prompt’s group. The result is less memory use and fewer moving parts, at the cost of generating several answers per prompt.
Can I run GRPO on a single consumer GPU?
Yes for small models, with LoRA and a shortened completion length. A 4B model is comfortable on a 40 to 48 GB card in my sizing and tight on 24 GB. Smaller models fit more easily.
Do I need a reward model or human labels?
No. You need a function that scores a completion, and for tasks like math or code that function can be a few lines of Python. That is the main appeal of GRPO over classic RLHF.
Why is my reward stuck at zero?
Usually because the base model never earns reward, so all answers in a group tie. Add a dense partial reward, make the task easier, or raise the sampling temperature. Test the rewards against the base model offline first.
Should I use GRPOTrainer or AsyncGRPOTrainer?
Start with GRPOTrainer. Move to the async trainer when generation time dominates or when you train agents with environments. The async trainer is still described as experimental, so expect interface changes.
How does this tutorial differ from the original GRPO paper setup?
The original work trained at far larger scale with different infrastructure and used a KL penalty against a reference model. This tutorial trims the idea to one GPU with LoRA, a colocated vLLM engine and beta=0.0. The core mechanism, group-relative advantages, is the same.
Will this exact code keep working?
Pin the versions in Step 1 and it should. TRL, vLLM and PEFT all shipped notable changes in October 2026 alone, so unpinned installs drift quickly. When you upgrade, read the release notes first and rerun the reward unit tests.
