Jump to content

Post-training of large language models

Add links
From Wikipedia, the free encyclopedia

Post-training of large language models is a term used in recent technical literature for training applied to a large language model (LLM) after its initial large-scale pretraining. The scope varies among surveys. Supervised fine-tuning or instruction tuning and preference-based alignment are common elements, while broader treatments differ on whether reasoning-focused training, parameter-efficient adaptation, distillation, tool integration, or inference-time scaling are included.[1][2]

In training-centered usage, a pretrained or base model is further optimized with structured data such as instruction-response pairs, comparisons between candidate responses, step-level judgments, or outcomes that can be checked automatically. Practical systems may combine several stages, for example supervised training followed by preference optimization or reinforcement learning.[3][4][5][6]

Scope and terminology

[edit]

Recent surveys use post-training as an umbrella label for several lines of research, including instruction tuning, preference alignment, reasoning-oriented training, efficiency methods, and distillation. Their boundaries differ, so the label does not imply a single objective or a universally agreed sequence of stages.[1][2]

Some surveys include increased computation at inference time in a broad account of post-training, while others center the term on additional training after pretraining.[1][2] Post-training in this sense is also distinct from post-training quantization, which lowers the numerical precision of an already trained network rather than teaching instruction following, preferences, or reasoning behavior.[7]

Development

[edit]

The methods now described as post-training emerged through several lines of work.[1][2] Multitask instruction tuning trained models on collections of tasks written as natural-language prompts. T0 converted supervised datasets into prompted tasks and evaluated generalization to tasks excluded from its training mixture.[8] Later FLAN experiments studied instruction tuning across larger task mixtures, model families, model sizes, and chain-of-thought data.[3]

Preference-based training developed alongside instruction tuning. Work on abstractive summarization trained a reward model from human comparisons and optimized a language model against that reward.[9] InstructGPT used supervised fine-tuning on demonstrations, reward-model training on ranked outputs, and reinforcement learning from human feedback.[10] Reviews of reinforcement learning from human feedback describe the approach as a combination of feedback collection, reward learning, and policy optimization, with limitations at each stage.[4][11]

Direct preference optimization (DPO), introduced in 2023, reformulated a commonly used preference-learning objective as a classification loss and avoided the separately trained reward model and online policy-gradient stage used in a common reinforcement learning from human feedback pipeline.[12] Kahneman-Tversky optimization (KTO) was subsequently proposed for data labeled desirable or undesirable without requiring matched response pairs.[13]

Methods

[edit]

Supervised fine-tuning and instruction tuning

[edit]

In supervised fine-tuning (SFT), the model is trained to predict a target response for a given input. Instruction tuning is a form of SFT in which examples are organized as natural-language instructions paired with desired outputs. Research on instruction tuning examines task mixtures, model scale, response construction, data quality, and generalization to unseen tasks.[3]

Instruction data can be written by people, converted from existing task datasets, generated synthetically, or assembled from several sources. Self-Instruct used a language model to generate instructions, inputs, and outputs, filtered invalid or near-duplicate examples, and then used the resulting data for fine-tuning.[14] Results from individual recipes do not establish a fixed amount or composition of data that works for every model or task; outcomes vary with the base model, task distribution, and training procedure.[3]

SFT often appears at the beginning of a multi-stage pipeline. It can establish response formats and basic instruction following before preference optimization or reinforcement learning, and it can provide a policy that generates outputs for later comparison or verification.[10][6]

Preference learning and alignment

[edit]

Preference learning uses judgments about candidate outputs rather than only a single target response. Feedback may be collected as rankings, pairwise comparisons, or labels such as desirable and undesirable. The model is then trained to increase the probability of preferred behavior, commonly subject to a constraint that limits divergence from a reference policy.[4][10][12]

Reward modeling and reinforcement learning from human feedback

[edit]

A common reinforcement learning from human feedback (RLHF) pipeline begins with SFT, collects comparisons between model outputs, trains a reward model to predict those comparisons, and optimizes the language-model policy to increase the predicted reward.[10][9][4] The reward model is a learned proxy for the collected feedback rather than a direct measurement of a universal human preference. The resulting behavior depends on the annotators, instructions, sampling process, and aggregation method used to construct the feedback data.[10][11]

Direct preference optimization

[edit]

DPO learns directly from preferred and rejected responses. Its derivation expresses the optimal policy for a class of reward-regularized objectives in a form that can be fitted with a binary classification loss.[12] DPO still depends on the coverage and quality of comparison data and on choices such as the reference policy and regularization strength.[12][15]

Related objectives use different feedback assumptions. KTO, for example, was designed for examples labeled desirable or undesirable without a matched alternative response.[13] These methods are alternatives within preference-based training, not evidence that one objective is uniformly preferable across datasets and applications.[13][11]

AI-generated feedback

[edit]

Feedback can also be generated or assisted by another language model. Reinforcement learning from AI feedback (RLAIF) trains a reward model using preferences produced by an AI labeler. In experiments on summarization and dialogue tasks, RLAIF produced results comparable to the studied RLHF baselines.[16] AI-generated feedback reduces dependence on human labels, but its results depend on the capabilities and prompts of the labeler model and on the criteria used to guide its judgments.[16][11]

Reasoning-focused training

[edit]

Some recent surveys include reasoning-focused methods within post-training.[2][1] These methods use worked solutions, generated rationales, step-level feedback, or automatically checked outcomes to train multi-step problem solving.

The Self-Taught Reasoner (STaR) iteratively generated rationales, retained rationales that led to correct answers, and fine-tuned on the retained examples.[17] Process supervision instead evaluates intermediate reasoning steps. In one study of mathematical reasoning, process-supervised reward models outperformed outcome-supervised reward models in the tested setting.[5] Such results are specific to the studied tasks and models and do not establish that one reasoning-training recipe is uniformly preferable.[2]

Parameter-efficient adaptation

[edit]

Full fine-tuning updates all model parameters and can require substantial accelerator memory. Parameter-efficient methods train a smaller set of added or selected parameters while leaving most base-model weights fixed. Low-rank adaptation (LoRA) represents weight updates with trainable low-rank matrices, reducing the number of trainable parameters and the storage needed for separate task adaptations.[18]

Quantized low-rank adaptation (QLoRA) combines low-rank adapters with a quantized frozen base model. Its design backpropagates gradients through a frozen 4-bit quantized model into LoRA adapters, reducing memory use during fine-tuning.[19] LoRA and QLoRA describe how an update is parameterized; they can be used with instruction tuning or preference optimization rather than constituting a separate behavioral objective.[18][19]

Distillation

[edit]

Knowledge distillation trains a student model to reproduce selected behavior from a teacher model. In LLM post-training, the transferred signal may include token probabilities, generated responses, or instruction-following behavior.[1] MiniLLM used an on-policy distillation objective based on reverse Kullback-Leibler divergence to train smaller generative language models from larger teachers.[20] Distillation can reduce deployment cost or consolidate behavior learned by a larger model, but the student remains dependent on the teacher outputs and on the coverage and filtering of the distillation data.[20][1]

Data and evaluation

[edit]

Post-training datasets differ by stage. SFT uses target responses, preference optimization uses comparisons or desirability labels, process supervision uses judgments on intermediate steps, and some reinforcement-learning settings use outcomes that can be checked by rules, tests, or answer matching.[3][4][5] Synthetic data can expand coverage, but filtering is used to remove invalid, repetitive, or low-quality generations.[14]

Evaluation is matched to the intended effect of each stage. Instruction following can be assessed on held-out tasks and human judgments; preference models can be compared on held-out comparisons; reasoning systems can be evaluated using answer correctness and, where available, judgments on intermediate steps.[3][4][5] Multi-stage pipelines may use development benchmarks during training and separate unseen evaluations afterward; some also apply benchmark decontamination to reduce overlap between training and evaluation data.[6]

No single score measures all post-training goals. Improvements in instruction following, preference fit, reasoning correctness, diversity, safety, or domain performance can move differently under the same update. Comparisons are also sensitive to the base model, training data, prompts, decoding settings, and evaluator.[4][11][15]

Limitations

[edit]

Preference-based methods depend on imperfect feedback. Human labels can be inconsistent, unrepresentative, or difficult to obtain for tasks whose correctness is hard to judge, while AI-generated labels depend on the labeler model and its instructions.[4][11][16] Optimization can also overfit a learned or implicit preference signal: performance measured by the training objective may continue to improve while quality under an external evaluator plateaus or deteriorates.[15]

Sequential stages can interfere with previously learned behavior. Recent post-training surveys discuss catastrophic forgetting, capability regressions, and trade-offs among alignment, diversity, and general task performance.[1][2] These concerns depend on the base model, data, objective, and ordering of stages rather than following uniformly from every post-training method.

Multi-stage pipelines increase computational and engineering cost and make causal attribution difficult. Final behavior may depend on the base model, data curation, stage order, reward design, sampling, and evaluation choices.[6][4] Limited disclosure of post-training datasets and recipes can hinder independent reproduction and comparison.[6]

See also

[edit]

References

[edit]
  1. 1 2 3 4 5 6 7 8 Tie, Guiyao; Zhao, Zeli; Song, Dingjie; et al. (2025). "A Survey on Post-training of Large Language Models". arXiv:2503.06072 [cs.CL].
  2. 1 2 3 4 5 6 7 Kumar, Komal; Ashraf, Tajamul; Thawakar, Omkar; et al. (2025). "LLM Post-Training: A Deep Dive into Reasoning Large Language Models". arXiv:2502.21321 [cs.CL].
  3. 1 2 3 4 5 6 Chung, Hyung Won; Hou, Le; Longpre, Shayne; et al. (2024). "Scaling Instruction-Finetuned Language Models". Journal of Machine Learning Research. 25 (70): 1–53. arXiv:2210.11416.
  4. 1 2 3 4 5 6 7 8 9 Kaufmann, Timo; Weng, Paul; Bengs, Viktor; Hüllermeier, Eyke (2025). "A Survey of Reinforcement Learning from Human Feedback". Transactions on Machine Learning Research. arXiv:2312.14925.
  5. 1 2 3 4 Lightman, Hunter; Kosaraju, Vineet; Burda, Yura; et al. (2024). "Let's Verify Step by Step". The Twelfth International Conference on Learning Representations. arXiv:2305.20050.
  6. 1 2 3 4 5 Lambert, Nathan; Morrison, Jacob; Pyatkin, Valentina; et al. (2024). "Tulu 3: Pushing Frontiers in Open Language Model Post-Training". arXiv:2411.15124 [cs.CL].
  7. ↑ Nagel, Markus; Amjad, Rana Ali; van Baalen, Mart; et al. (2020). "Up or Down? Adaptive Rounding for Post-Training Quantization". Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 119. pp. 7197–7206.
  8. ↑ Sanh, Victor; Webson, Albert; Raffel, Colin; et al. (2022). "Multitask Prompted Training Enables Zero-Shot Task Generalization". International Conference on Learning Representations. arXiv:2110.08207.
  9. 1 2 Stiennon, Nisan; Ouyang, Long; Wu, Jeff; et al. (2020). "Learning to Summarize from Human Feedback". Advances in Neural Information Processing Systems 33. arXiv:2009.01325.
  10. 1 2 3 4 5 Ouyang, Long; Wu, Jeff; Jiang, Xu; et al. (2022). "Training Language Models to Follow Instructions with Human Feedback". Advances in Neural Information Processing Systems 35. arXiv:2203.02155.
  11. 1 2 3 4 5 6 Casper, Stephen; Davies, Xander; Shi, Claudia; et al. (2023). "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback". Transactions on Machine Learning Research. arXiv:2307.15217.
  12. 1 2 3 4 Rafailov, Rafael; Sharma, Archit; Mitchell, Eric; et al. (2023). "Direct Preference Optimization: Your Language Model Is Secretly a Reward Model". Advances in Neural Information Processing Systems 36. arXiv:2305.18290.
  13. 1 2 3 Ethayarajh, Kawin; Xu, Winnie; Muennighoff, Niklas; et al. (2024). "KTO: Model Alignment as Prospect Theoretic Optimization". Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 235. arXiv:2402.01306.
  14. 1 2 Wang, Yizhong; Kordi, Yeganeh; Mishra, Swaroop; et al. (2023). "Self-Instruct: Aligning Language Models with Self-Generated Instructions". Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. arXiv:2212.10560.
  15. 1 2 3 Rafailov, Rafael; Chittepu, Yaswanth; Park, Ryan; et al. (2024). "Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms". Advances in Neural Information Processing Systems 37. arXiv:2406.02900.
  16. 1 2 3 Lee, Harrison; Phatale, Samrat; Mansoor, Hassan; et al. (2024). "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback". Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research. Vol. 235. pp. 26874–26901. arXiv:2309.00267.
  17. ↑ Zelikman, Eric; Wu, Yuhuai; Mu, Jesse; Goodman, Noah D. (2022). "STaR: Bootstrapping Reasoning with Reasoning". Advances in Neural Information Processing Systems 35. arXiv:2203.14465.
  18. 1 2 Hu, Edward J.; Shen, Yelong; Wallis, Phillip; et al. (2022). "LoRA: Low-Rank Adaptation of Large Language Models". International Conference on Learning Representations. arXiv:2106.09685.
  19. 1 2 Dettmers, Tim; Pagnoni, Artidoro; Holtzman, Ari; Zettlemoyer, Luke (2023). "QLoRA: Efficient Finetuning of Quantized LLMs". Advances in Neural Information Processing Systems 36. arXiv:2305.14314.
  20. 1 2 Gu, Yuxian; Dong, Li; Wei, Furu; Huang, Minlie (2024). "MiniLLM: On-Policy Distillation of Large Language Models". The Twelfth International Conference on Learning Representations. arXiv:2306.08543.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Post-training_of_large_language_models&oldid=1363846749"