Skip to content

About

No description, website, or topics provided.

Resources

Stars

52 stars

Watchers

1 watching

Forks

Repository files navigation

♟️ Official implementation of "Understanding Reasoning Pretraining to Post-training" Paper

Paper

Overview

We study how pretraining shapes what post-training (SFT + RL) can achieve, using chess as a fully controllable, verifiable testbed. The pipeline (figure above) is:

  • (a) Pre-to-post training pipeline — pretrain on human chess games, SFT on synthetic tree-search reasoning traces, then RL in a verifiable puzzle environment.
  • (b) Scaling analysis — a pretraining↔RL scaling law relating post-RL reward to pretraining loss and pretraining tokens.
  • (c) Mechanism analysis — is RL just sharpening the SFT policy, or discovering new behaviors? (ground-truth amplification vs. tail discovery).

📢 Updates

  • 2026-10-16 — RL now fixes the trainer–inference mismatch by default (fp32 LM head on both the trainer and vLLM, and vLLM's own log-probs as π_old in the PPO ratio); configs it can't support (e.g. top-p < 1, async or SGLang rollout, LoRA) stop at startup with an error naming the flag to set to false.
  • 2026-09-22 — Fixed how SFT applies the positional-encoding (YaRN) context extension. The block_size/rope_scaling change was applied after the model was built, so SFT runs with a longer context than the 1,024-token pretraining context trained with plain RoPE extrapolation while the saved config claimed YaRN. The config is now patched before the model is instantiated, so training and inference both use YaRN (factor 3, 1,024 → 3,072 tokens). In our reruns this improves SFT, most clearly for the 20M and 50M models. Thanks to Jens Tuyls for spotting it.

🗺️ Repository structure

Each training stage is self-contained and has its own README with environment setup, data, and quick-start commands. Start here:

Stage Directory What it does
1️⃣ Pretraining pretraining/ Pretrain Qwen3-style models (20M–1B) on tokenized human chess games.
2️⃣ SFT sft/ Generate chain-of-thought (CoT) data via tree search, then run multi-turn supervised fine-tuning on a pretrained checkpoint.
3️⃣ RL rl/ Multi-turn GRPO post-training in a verifiable chess environment, built on a fork of verl.

🔧 Supporting code

Directory Purpose
data_preprocessing/ Download / decontaminate / tokenize human games and puzzles.
cot_analysis/ Analyze and score the generated chain-of-thought reasoning.
policy_evolution/ Measure how the policy evolves across pretraining → SFT → RL.
download/ Fetch released HF checkpoints.
llm_tokens/ Chess tokenizer / token utilities.
results/ Output location for runs and evaluations.

🚀 Quick start

# 1️⃣ Pretrain
cd pretraining && bash run_pretrain.sh          # see pretraining/README.md

# 2️⃣ SFT (CoT generation + fine-tuning)
cd sft && bash run_sft.sh                        # see sft/README.md

# 3️⃣ RL (multi-turn GRPO)
cd rl/verl/8_gpu_bash && bash sweep_multi_turn.sh   # see rl/README.md

Each script is driven by environment variables (data paths, GPU counts, W&B entity, checkpoint specs) — see the per-stage READMEs for the full list of knobs.


📚 Citation

If you find this work useful, please cite:

@article{pre2post-chess,
  title   = {Understanding Reasoning from Pretraining to Post-Training},
  author  = {Shen, Jingyan and Li, Ang and Rahman, Salman and Sun, Yifan and
             Goldblum, Micah and Telgarsky, Matus and Izmailov, Pavel},
  journal = {arXiv preprint arXiv:2607.16097},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2607.16097}
}

About

No description, website, or topics provided.

Resources

Stars

52 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages