Skip to main content
Why HTML? Back to Abstract Download PDF
License: CC BY 4.0
arXiv:2501.03124v5 [cs.CL] 28 Jun 2025

PRMBench: A Fine-grained and Challenging Benchmark for
Process-Level Reward Models

Mingyang Song Affiliation: Fudan University Affiliation: Shanghai AI Laboratory    Zhaochen Su Affiliation: Soochow University    Xiaoye Qu Affiliation: Shanghai AI Laboratory    Jiawei Zhou ††thanks: Equal senior contribution Affiliation: Stony Brook University    Yu Cheng11footnotemark: 1 Affiliation: The Chinese University of Hong Kongmysong23@m.fudan.edu.cn; suzhaochen0110@gmail.com;quxiaoye@pjlab.org.cn; jzhou@ttic.edu; chengyu@cse.cuhk.edu.hk;Project Page: https://prmbench.github.io
Abstract

Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since large language models (LLMs) suffer from various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs’ performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 25 models, spanning across both open-source PRMs and LLMs prompted as critic models, we uncover significant weaknesses in current PRMs. These findings reveal the challenges inherent in process-level evaluation and highlight key directions for future research, establishing PRMBench as a robust testbed for advancing research on PRM evaluation and development.

Refer to caption
Figure 1: (Left): Given a question QQ, the reasoning step 2 and 5 of OpenAI-o1 model contains errors. (Right): The step-level reward scores generated by ReasonEval-34B Xia et al. (2024b) and MathShepherd-7B Wang et al. (2023). Green scores indicate the PRM prefer labeling this step as correct while red scores indicate the PRM prefer identifying this step as incorrect.
PRM
Benchmarks?
Error Type
Detection?
Fine-grained
classes†
Step
Evaluation
Annotator
Test Case
Size
Average
Steps
MR-GSM8K Zeng et al. (2023) ✗ ✗ 1 ✓ Human 2,999 8.3
RMBench Liu et al. (2024) ✗ ✗ 1 ✗ Synthetic + Human 1,327 -
CriticBench Lin et al. (2024) ✗ ✗ 1 ✗ - - -
MathCheck-GSM Zhou et al. (2024) ✗ ✗ 1 ✓ Synthetic 516 -
MR-Ben Zeng et al. (2024b) ✗ ✗ 1 ✓ Human 5,975 9.5
ProcessBench Zheng et al. (2024) ✓ ✗ 1 ✓ Human 3,400 7.1
PRMBench ✓ ✓ 9 ✓ Synthetic + Human 6,216 13.4
Table 1: Comparison between our proposed PRMBench and other benchmarks or datasets related to reasoning process assessment. †: Fine-grained classes mean the number of evaluation categories according to fine-grained error types of model generation.

1 Introduction

Recent large language models (LLMs) OpenAI (2024a); OpenAI (2024b); Team (2024a), trained on large-scale reinforcement learning, have achieved significant performance in complex reasoning tasks such as mathematics and code generation Yu et al. (2023); Guo et al. (2024); DeepMind (2024); Luo et al. (2023); Qu et al. (2025). A key factor behind their successes is the use of process reward models (PRMs) Wang et al. (2023); Lightman et al. (2023); Uesato et al. (2022), which can help evaluate the correctness of reasoning steps and train LLMs with appropriate rewards. Qin et al. (2024); Zhang et al. (2024b).

However, during the reasoning process of the LLM, it suffers from various types of errors, while recent PRMs are not able to identify all these error types precisely. For instance, as illustrated in Figure 1, given a question QQ, OpenAI o1 model OpenAI (2024b) generates a reasoning procedure containing errors, where step 2 is redundant, step 5 is inconsistent with step 3, and the theory used in step 5 is incorrect due to deception falling. Under these circumstances, ReasonEval-34B Xia et al. (2024b) and Math-Shepherd-7B Wang et al. (2023) fail to identify these errors accurately. Math-Shepherd-7B fails to recognize step 5 as an error, while ReasonEval-34B correctly identifies step 5 but incorrectly classifies step 4 as an error, indicating the unreliability of current PRMs.

To evaluate the diverse error-detection capabilities of PRMs, we present PRMBench, a comprehensive and fine-grained benchmark specifically designed for assessing PRMs. In contrast to existing process-level benchmarks, which can only evaluate the detection of a single error type Zheng et al. (2024); Zeng et al. (2024b), PRMBench offers a more nuanced evaluation. Specifically, PRMBench systematically assesses the performance of PRMs across diverse error categories, including simplicity, soundness, and sensitivity. Our benchmark includes 6,216 fine-grained data instances spreading across three major evaluation categories and nine sub-categories, whose quality is ensured by professional annotators. Additionally, we utilize style-controlled data curation methods to ensure evaluation samples under consistent difficulty levels, mitigating confounding variables.

In our study, we conduct extensive experiments using PRMBench to evaluate 25 models, including dedicated PRMs and SOTA general-purpose or mathematical LLMs, prompted as critic models. We observe that all PRMs partially grasp multi-step process evaluation. Specifically, Gemini-2-Thinking achieves the best performance of 68.8, but still significantly falls behind the human performance of 83.8. Through extensive analysis, we discover a significant inconsistency between step-level and outcome-level evaluations. By evaluating models with PRMBench, we can assess PRMs’ ability to detect step-level errors and false positives, reducing the risk of outcome hacking Gao et al. (2025). To sum up, our contributions are as follows:

Figure 2: An overview of our PRMBench. The left part illustrates our data curation procedure. In the right part of the figure, we showcase demonstrations of our evaluation categories and the relative performance of tested models, with green, yellow, and gray boxes indicating simplicity, soundness, and sensitivity respectively, where red circles represent erroneous steps and green circles indicate correct regular steps.
  • •

    We present PRMBench, the first comprehensive process-level reward model benchmark, comprising 6,216 carefully curated samples and 83,456 step-level labels for a series of evaluations on process-level reward models.

  • •

    PRMBench covers three carefully-crafted evaluation categories and nine sub-categories including simplicity, soundness, and sensitivity. With these fine-grained evaluation axes, we can conduct tailored assessments of models on their specified capabilities and reveal their potential weaknesses during the rewarding procedure.

  • •

    Based on our proposed PRMBench, we conduct in-depth pilot experiments on twenty-five models including PRMs along with SOTA LLMs. Our findings uncover critical weaknesses and provide valuable insights to guide future research to improve the capabilities of PRMs.

  • •

    To facilitate future research, we release the PRM-EVAL toolkit, offering an automated evaluation framework and customizable data generation system. We hope PRMBench will drive progress in step-level reasoning for RLHF and foster further development of more reliable PRMs.

2 Related Work

2.1 Process-level Reward Models

Process-level reward models (PRMs) have shown improvements over traditional outcome-level reward models (ORMs) in enhancing process-level reasoning accuracy and long-process reasoning abilities (Lightman et al., 2023; Uesato et al., 2022). Recently, several PRMs have been proposed for process-level RLHF Wang et al. (2023); Xia et al. (2024b); o1 Team (2024), with Lightman et al. (2023) releasing a large dataset for multi-step reasoning, and Wang et al. (2023) introducing an automatic self-supervised pipeline for process-level labeling. Xia et al. (2024b) uses PRMs as auto-evaluators for multi-step reasoning accuracy. As PRM training and data curation have grown, numerous PRMs o1 Team (2024); Xiong et al. (2024); Team (2024b); Gao et al. (2024) have emerged, along with critic models using LLM-generated feedback McAleese et al. (2024); Zhang et al. (2024a); Gao et al. (2024). However, both PRMs and critic models remain fallible, highlighting the need for comprehensive benchmarks. In this paper, we propose PRMBench, a comprehensive benchmark for evaluating PRMs on fine-grained subjects, establishing a strong foundation for PRM evaluation.

2.2 Reasoning Benchmarks

Evaluating the reasoning capabilities of LLMs is crucial for understanding their potential and limitations. ROSCOE Golovneva et al. (2022) introduces a semantic comparison-based multi-step reasoning accuracy evaluation benchmark. However, recent research suggests that labeled data cannot be assumed to cover all possible solution paths exhaustively Wang et al. (2023); Xia et al. (2024b). To address this, Xia et al. (2024b) uses PRMs or Critic models to evaluate step-level reasoning accuracy. However, PRMs are not always accurate in assessing process-level data, underscoring the need for a comprehensive evaluation benchmark. While other benchmarks Liu et al. (2024); Li et al. (2024a); Lin et al. (2024); Su et al. (2024b) exist, they are not tailored for PRMs and can’t assess step-level reasoning. Some works Zeng et al. (2023); Zeng et al. (2024b); Yan et al. (2024) use LLMs to evaluate reasoning steps, but they often overlook implicit error types. Existing error classification works are not specific to PRMs and lack fine-grained step-level labels Li et al. (2024b). To address these gaps, we propose PRMBench, a solution that offers fine-grained evaluation and detects various error types.

Overall NR. NCL. ES. SC. DC. CI. PS. DR. MS.
Avg. Steps 13.4 15.3 10.3 13.8 14.2 13.3 14.2 12.7 13.4 14.1
Avg. Error Steps 2.1 2.0 2.8 2.8 1.6 1.8 1.7 2.5 2.3 0.0
Avg. First Error Step 7.8 7.8 4.9 8.0 9.1 6.8 11.4 6.2 8.3 N/A
Avg. Question Length 152.7 153.6 152.5 153.5 149.7 152.5 152.7 158.0 153.5 132.2
# of Instances 6216 758 758 757 758 757 757 756 750 165
Table 2: Statistics of PRMBench. NR., NCL., ES., SC., DC., CI., PS., DR., and MS. represent for Non-Redundancy, Non-Circular Logic, Empirical Soundness, Step Consistency, Domain Consistency, Confidence Invariance, Prerequisite Sensitivity, Deception Resistance, and Multi-Solution Consistency respectively.

3 PRMBench

3.1 Evaluation Categories

In this section, we provide a detailed introduction to the evaluation categories of PRMBench, which is organized into three main domains:

  • •

    Simplicity evaluates the ability of PRMs to detect redundancy in reasoning steps. Although redundant steps do not affect correctness, they increase computational costs and reduce efficiency. Additionally, simplifying the reasoning process enhances the clarity of the problem’s core and improves overall understandability.

  • •

    Soundness assesses the accuracy of the rewards produced by PRMs. As discussed in Section 1, errors in reasoning can vary in both causes and manifestations Li et al. (2024b). Therefore, we evaluate not only the correctness of rewards but also the fine-grained performance across different error types and their nuances.

  • •

    Sensitivity measures PRMs’ robustness to details, such as critical conditions or implicit requirements. Sensitivity is vital for ensuring logical completeness and resilience to misleading information Wen et al. (2025), contributing to the overall robustness of PRMs.

Each domain is further divided into detailed sub-categories for a more granular evaluation, which is discussed in detail below. The overall structure of PRMBench along with representative examples of each sub-category are illustrated in Figure 2, and the details of every evaluation category and sub-category are shown in Appendix A.1.

3.1.1 Simplicity

Specifically, the simplicity evaluation category is divided into two sub-categories: Non-Redundancy and Non-Circular Logic, with detailed descriptions provided below:

Non-Redundancy

evaluates the PRMs’ ability to identify redundancy within the reasoning process. Redundancy occurs when the reasoning includes unnecessary steps that do not contribute to the solution, making the process less concise and efficient. These steps can be removed without affecting the correctness of the final solution path.

Non-Circular Logic

assesses the PRMs’ ability to detect circular reasoning within the process. Circular logic is a form of redundancy where the reasoning eventually loops back to a previous step, creating an infinite cycle. This sub-category is treated separately due to the frequent occurrence of circular logic in reasoning processes.

3.1.2 Soundness

We divide the Soundness category into four sub-categories due to its complexity: Empirical Soundness, Step Consistency, Domain Consistency, and Confidence Invariance. The definition of each sub-category is discussed below.

Empirical Soundness

demands PRMs to detect the counterfactual mistakes within the reasoning process. A counterfactual step refers to a statement within a reasoning chain that contradicts established ground truth GG.

Step Consistency

expects PRMs to detect the step-wise contradiction, which means a conflict between a specific step and other steps within a reasoning path. Given a reasoning path P={S1,S2,…,Sn}P=\{S_{1},S_{2},\dots,S_{n}\}, a step contradiction exists if Si⟂SjS_{i}\perp S_{j}, where i,j∈[1,n]i,j\in[1,n] and i≠ji\neq j.

Domain Consistency

requires PRMs to detect domain inconsistency mistakes, which is a special type of counterfactual. It refers to a step within the reasoning chain that uses a statement or theory valid in other domains or cases but is not valid within the current reasoning chain.

Confidence Invariance

demands PRMs to detect over-confident errors, a type of counterfactual where an incorrect statement is made with high confidence, contradicting established ground truth.

3.1.3 Sensitivity

This category includes three sub-categories: Prerequisite Sensitivity, Deception Resistance, and Multi-Solution Consistency, with detailed descriptions provided below.

Prerequisite Sensitivity

requires PRMs to maintain sensitivity to missing conditions or prerequisite mistakes, which means a flaw in the reasoning chain where critical premises, assumptions, or necessary conditions are absent and this omission results in logical gaps, incomplete reasoning, or biased conclusions.

Deception Resistance

demands PRMs to detect the deception or trap within a reasoning process, that is, statements that appear to be correct but are subtly altered to introduce inaccuracies while maintaining the illusion of correctness.

Multi-Solution Consistency

expects PRMs to maintain consistency when faced with different solution paths of the same problem. Concretely, we utilize multiple correct reasoning processes of the same question to test whether the PRM can perform correctly.

3.2 Data Curation

We curate the dataset by extracting metadata and constructing test cases according to our category definitions. Detailed statistics of PRMBench are displayed in Table 2, with the curation procedure outlined below.

Meta Data Extraction

Our metadata is built upon PRM800K Lightman et al. (2023), which provides the questions (QQ), ground truth answers (AA), and ground truth step-level solution processes (SS). We select completely correct solutions from both the training and test sets, filtering out low-quality instances to establish our ground truth answers.

Test Case Construction

Each test case instance is represented as (Q′,A,S′)(Q^{\prime},A,S^{\prime}), where Q′Q^{\prime} denotes the test question and S′S^{\prime} represents the test solution process, which may include errors. With class-specific prompts, as demonstrated in Appendix E.1, we query GPT-4o OpenAI (2024a) to modify the ground-truth reasoning process into versions containing erroneous steps. For the multi-solution, we leverage the newly proposed multi-step reasoning model QwQ11 1 Qwen/QwQ-32B-Preview: https://huggingface.co/Qwen/QwQ-32B-Preview Team (2024a) to generate candidate answers for the given questions. These answers are then filtered to exclude unreasonable or incorrect ones, resulting in multi-solution reasoning processes for a single question.

Model Overall Simplicity Soundness Sensitivity
NR. NCL. Avg. ES SC. DC. CI Avg. PS DR. MS. Avg.
Human Performance 83.8 80.2 81.7 81.0 84.8 85.0 81.3 86.1 84.3 81.6 82.1 96.2 86.0
Open-source Process Level Reward Models
Skywork-PRM-1.5B 61.1 52.0 56.4 54.2 64.8 64.9 63.3 66.5 64.9 57.5 63.3 91.1 70.7
Skywork-PRM-7B 65.1 56.4 62.8 59.6 69.4 67.1 67.7 69.9 68.5 60.9 65.8 93.2 73.3
Llemma-PRM800k-7B 52.0 49.3 53.4 51.4 56.4 47.1 46.7 53.3 50.9 51.0 53.5 93.6 66.0
Llemma-MetaMath-7B 50.5 50.2 50.5 50.3 51.9 47.6 44.4 52.1 49.0 50.5 51.3 96.0 66.0
Llemma-oprm-7B 50.3 48.7 49.3 49.0 54.2 46.8 44.5 53.5 49.8 49.2 51.3 91.8 64.1
MATHMinos-Mistral-7B 54.2 48.8 54.0 51.4 57.0 52.1 50.7 57.8 54.4 52.8 55.8 91.1 66.5
MathShepherd-Mistral-7B 47.0 44.0 50.3 47.1 49.4 44.5 41.3 47.7 45.7 47.2 48.6 86.1 60.7
ReasonEval-7B 60.1 61.0 50.1 55.6 62.1 65.9 61.5 66.0 63.9 55.7 58.0 99.5 71.1
ReasonEval-34B 60.5 54.8 48.1 51.5 66.4 60.3 57.8 67.5 63.0 57.7 64.3 97.2 73.1
RLHFlow-PRM-Mistral-8B 54.4 46.1 47.3 46.7 56.6 55.1 54.4 63.8 57.5 51.5 56.2 97.9 68.5
RLHFlow-PRM-Deepseek-8B 54.2 46.4 48.9 47.6 55.7 55.0 53.2 66.2 57.5 49.0 55.4 99.8 68.1
Qwen2.5-Math-PRM-7B 65.5 49.0 55.1 52.1 71.8 67.3 66.3 78.5 71.0 57.6 69.1 99.7 75.5
Qwen2.5-Math-PRM-72B 68.2 50.4 58.8 54.6 73.7 71.1 72.2 78.6 73.9 60.3 71.2 99.4 77.0
Pure-PRM-7B 65.3 49.2 55.2 52.2 71.1 68.8 64.0 76.9 70.2 60.3 69.2 98.0 75.8
Avg. 57.7 50.5 52.9 51.7 61.5 58.1 56.3 64.2 60.0 54.4 59.5 95.3 69.7
Open LLMs, Prompted as Critic Models
MetaMath-7B 49.7 48.9 46.9 47.9 47.3 48.9 48.4 48.8 48.3 46.5 48.3 98.0 64.2
MetaMath-13B 49.4 50.3 44.4 47.3 47.8 47.4 49.4 48.1 48.2 49.0 48.1 99.5 65.5
Qwen2.5-Math-72B 57.4 55.3 54.9 55.1 55.5 71.6 58.1 59.1 61.1 47.4 53.8 100.0 67.1
QwQ-Preview-32B 63.6 57.2 55.6 56.4 67.4 72.3 66.2 66.9 68.2 57.8 62.7 100.0 73.5
R1-Distill-Llama3.1-70B 57.5 49.5 48.1 48.8 61.4 65.5 65.8 61.1 63.4 48.8 54.1 100.0 67.6
R1-Distill-Qwen-7B 52.6 32.9 37.9 35.4 47.3 54.1 48.4 48.0 49.4 45.6 46.8 100.0 64.1
DeepSeek-R1† 67.8 63.0 62.7 62.9 68.2 68.5 73.5 75.4 71.4 63.3 68.0 100.0 77.1
Avg. 56.8 51.0 50.1 50.5 56.4 61.2 58.5 58.2 58.6 51.2 54.5 99.6 68.5
Proprietary LLMs, Prompted as Critic Models
GPT-4o 66.8 57.0 62.4 59.7 72.0 69.7 70.7 71.1 70.9 62.5 65.7 99.2 75.8
o1-mini† 68.8 65.6 63.7 64.6 74.5 67.7 73.8 72.3 72.1 61.8 64.8 100.0 75.5
Gemini-2.0-flash-exp 66.0 67.2 58.1 62.7 70.4 65.7 66.0 67.3 67.3 61.8 66.2 98.2 75.4
Gemini-2.0-thinking-exp-1219 68.8 68.5 63.8 66.2 72.9 71.3 71.0 71.8 71.8 60.3 65.7 99.8 75.3
Avg. 67.6 64.6 62.0 63.3 72.4 68.6 70.4 70.7 70.5 61.6 65.6 99.3 75.5
Table 3: Performances comparison of popular models on PRMBench. The best performance for each category and task is in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and DeepSeek-R1.

3.3 Quality Control

To ensure a high-quality dataset, we implement a series of steps to filter out unqualified data and maintain data integrity. The specific procedures are outlined below:

Feature Filtering

Our data curation procedure imposes strict structural requirements on the generated responses, where any outputs that do not satisfy these specifications cannot be considered valid for accurately assessing the performance of PRMs. However, even with detailed instructions, LLMs cannot consistently generate outputs that fully adhere to the required structure Asai et al. (2024); Zeng et al. (2024a); Su et al. (2024a). To maintain high data quality, we define stringent filtering rules to exclude instances that fail to meet the necessary structural criteria. Detailed structural requirements are provided in Appendix E.1, and the full description of our data generation process can be found in the supplementary materials.

Human Verification

Furthermore, to further ensure the quality of the data, we manually evaluate 10% of the total instances. We focus on two key qualities for each data instance: ❶ Correctness of modification: Whether the modifications made to the data instance are correct and reasonable. ❷ Difference in the modification: Whether the modified data instance differs from the original. We recruited five volunteers to evaluate our proposed PRMBench and observe over 92% qualification rate on the correctness metric and over 98% qualification rate on the difference metric. The details of human annotation are provided in Appendix A.3.1, and instructions for annotators are provided in Appendix D. This validation ensures the overall quality of our dataset and its suitability for studying process-level language reward models.

4 Experiments

4.1 Models

To provide a comprehensive evaluation of various models on PRMBench, we select a wide range of models, including open-source PRMs like QwenPRM Zhang et al. (2025) and RLHFlowPRMs Xiong et al. (2024), as well as LLMs prompted as critic models, such as o1-mini OpenAI (2024b) and DeepSeek R1 Guo et al. (2025). A complete list of these models can be found in Appendix B.2. Additionally, we present the human evaluation results, with details available in Appendix A.3.2.

All PRMs and LLMs are evaluated on the complete PRMBench dataset, except for o1-mini and DeepSeek-R1, which are evaluated on a subset of PRMBench comprising 394 samples, proportionally selected to reflect the class distribution, in order to reduce evaluation costs.

Considering the complexity of the task, which involves question comprehension, evaluation of the provided processes, and adherence to format constraints, few-shot demonstration setups are employed to help the model adapt to the output format through In-Context Learning (ICL) examples. Specifically, we use two-shot examples when prompting general-purpose LLMs. The impact of few-shot settings is discussed in Section 5.4

4.2 Evaluation Metrics

Given our emphasis on evaluating the error detection capabilities, we use the negative F1 score as a metric for error detection performance. However, this metric may be affected by the inherent biases of models. To mitigate this and provide a unified, normalized score that reflects the overall competency of the evaluated model, following Zheng et al. (2024), we introduce a metric called PRMScore, defined formally in Equation 1.

P​R​M​-​S​c​o​r​e=w1∗F​1n​e​g+w2∗F​1\begin{aligned} PRM\mbox{-}Score=w_{1}*F1_{neg}+w_{2}*F1\end{aligned}

(1)

Where F1 and F1neg refer to F1 scores and negative F1 scores respectively. w1w_{1} and w2w_{2} are weights that are designed to maximize the differentiation between different models. The detailed evaluation procedure is provided in Appendix B.3. Besides, we also provide results of all evaluation categories in fine-grained metrics in Appendix B.4.

4.3 Main Results

The main results are shown in Table 3. Some observations can be summarized as follows:

Model Accuracy PRM Score Sim.
Pos. Neg.
ReasonEval-7B 95.5 21.2 60.0 91.6
ReasonEval-34B 79.1 48.4 60.5 82.8
Skywork-7B 30.1 79.7 36.2 74.3
RLHFlow-DeepSeek-8B 95.0 13.0 54.2 95.0
GPT-4o 82.9 58.2 66.8 76.6
Gemini-2-thinking 89.0 49.8 68.8 82.0
Random 50.0 50.0 50.0 79.4
Table 4: Comparison of model performance on positive and negative test cases, along with their similarities.
The PRMs partially grasp multi-step process evaluation

Our analysis indicates that, although Gemini-2-Thinking achieves the highest performance among all evaluated models, its score is still significantly lower than human performance (68.8 vs. 83.8), highlighting substantial room for improvement in multi-step process evaluation. Some models even perform worse than random guessing, highlighting their limited reliability and potential training biases. Notably, the best open-source PRMs fail to match the performance of general-purpose proprietary LLMs, which suggests that even specifically trained PRMs still lag behind leading general-purpose models. We provide a detailed error analysis in Appendix C.1.

Simplicity is more challenging for PRMs

Our analysis highlights significant variations in model reasoning capabilities across evaluation categories. For instance, in the Sensitivity category, ReasonEval-34B performs relatively well, achieving an average score of 73.1. Especially in the Multi-Solutions sub-category, it excels with a PRMScore of 97.2, approaching near-perfect classification accuracy. This suggests models perform relatively better on correct instance judgment. However, its performance declines markedly in more complex scenarios. In the Simplicity category, ReasonEval-34B’s PRMScore drops to 51.5, suggesting partially reliable performance.

Furthermore, to broaden the domain coverage of PRMBench, we additionally collect STEM-related data and construct PRMBench-STEM, which is designed to evaluate PRM performance across other domains. The data construction methodology and experimental results for PRMBench-STEM are provided in the appendix B.5.

Figure 3: Distribution of error positions, truncated to 16 for better visualization, corresponding to the label field as shown in Figure 2.

5 Detailed Analysis

5.1 Inference Bias within PRMs

Takeaway 1. PRMs show a clear bias during evaluation, often favoring positive rewards.

As shown in Table 3, most open-source PRMs exhibit significant bias during evaluation, with some models performing worse than random guessing. suggesting the potential presence of bias within the inference procedure for our test cases. To validate this assumption, we compare the difference of models’ performance on positive and negative instances. As shown in Table 4, some models exhibit a clear bias during evaluation, often favoring positive rewards. For instance, ReasonEval-7B and RLHFlow-DeepSeek-8B achieve over 95% accuracy on positive-labeled steps but only attain an average of 17% accuracy on negative-labeled steps. Although proprietary LLMs outperform open-source PRMs, they also exhibit bias with a comparatively milder reward tendency.

Additionally, to further investigate inference bias, we evaluate the reward similarity of models’ performance between completely correct reasoning processes and our test cases. The solution-level similarity is defined as S=100−|A​c​cp​o​s−A​c​cn​e​g|S=100-|Acc_{pos}-Acc_{neg}|, where A​c​cAcc denotes the average step accuracy within a solution. The results, shown in Table 4, reveal that certain models, such as ReasonEval-7B and RLHFlow-DeepSeek-8B, exhibit significantly higher similarity than the normal similarity score (79.4), showcasing potential limitations in differentiating positive and negative steps.

Figure 4: The models’ error-detection accuracy across different error steps, where step 1 and steps beyond 11 are truncated for improved visualization.
Model 0-shot 1-shot 2-shot
GPT-4o 68.1 68.2 66.8
Gemini-2-flash 65.3 64.9 66.0
Gemini-2-thinking 67.8 67.8 68.8
Table 5: The impact of ICL few-shot numbers on model performance. The number reported here is PRMScore.

5.2 Performance across Different Steps

Takeaway 2. PRMs show a gradual improvement in performance as the position of the steps increases.

PRMBench includes a wide range of error step positions. The distribution of error positions is illustrated in Figure 3. While differences exist across categories, the overall pattern remains consistent: all categories peak in frequency at step 5 and gradually decrease thereafter. This raises an interesting question: Does the variation in step positions affect model performance? To investigate, we focus on error steps to assess how erroneous step positions influence model accuracy. As depicted in Figure 4, proprietary LLMs maintain stable performance across different error step positions. In contrast, PRMs, including Math-Shepherd-7B and ReasonEval-7B, show a gradual improvement in performance as error step positions increase.

5.3 Impacts of ICL Settings

Takeaway 3. In-context learning has subtle impact on models’ performance on PRMBench.

In this section, we investigate the impact of different ICL few-shot numbers on models’ performance. We vary the number of ICL few-shots to 0, 1, and 2 to examine whether increasing the few-shot number enhances the performance of generative models prompted as critic models. As shown in Table 5, for the Gemini-series models, a subtle improvement in performance is observed with a few-shot setup. However, for GPT-4o, no significant improvement is detected, and in some cases, a larger few-shot number even results in a decline in performance. These findings suggest that a few-shot approach exerts only a subtle impact on model performance on PRMBench.

Method MATH OlymBen Avg. PRMScore
Pass@8 96.2 79.8 88.0 -
Maj@8 71.8 40.3 56.1 -
SkyworkPRM-7B 90.0 60.1 75.1 65.1
LlemmaPRM-7B 87.4 58.3 72.8 52.0
MATHMinos-7B 88.3 59.1 73.7 54.2
MathShepherd-7B 88.6 60.0 74.3 47.0
ReasonEval-7B 87.0 58.4 72.7 60.1
RLHFlowPRM-8B 87.6 58.5 73.0 54.2
Qwen2.5-PRM-7B 88.0 58.7 73.4 65.5
Standard Dev (σ\sigma) 0.91 0.71 0.81 6.40
Somers’ D -0.05 0.05 -0.05 1.00
Table 6: Performance comparison on Best-of-8 using different PRMs. σ\sigma represents the standard deviation of model performances across all benchmarks. Somers’ D refers to the Somers’ D correlation between PRMScore and specific benchmarks.

5.4 Comparison between BoN Evaluation and PRMBench

Takeaway 4. PRMs struggle with detecting false positives, exposing the potential for reward hacking.

We compare the results between our PRMBench and Best-of-N (BoN) evaluation to observe the correlation. Following Zhang et al. (2025); Yang et al. (2024a), we sampled eight responses (i.e., N=8) from Qwen-QwQ across multiple mathematical benchmarks, including GSM8K Cobbe et al. (2021), MATH Hendrycks et al. (2021), Olympiad Bench He et al. (2024a) and MMLU Hendrycks et al. (2020). During evaluation, the PRMs are tasked with assigning a validity score to each step within every candidate response. The overall score for each candidate response is calculated by multiplying the individual step scores, as outlined in Lightman et al. (2023). We also provide majority voting as an baseline and pass@8 as the upper bound. The experiment setting and full BoN evaluation results are shown in Appendix C.2.

Although PRMs excel at selecting correct outcomes, they struggle with step-level reward hacking. As shown in Table 6, the average Somers’ D correlation between PRMBench and BoN is only -0.05, highlighting the inconsistency between step-level and outcome-level evaluation. For instance, Math-Shepherd-7B achieves a PRMScore of 47.0 with 51.3% accuracy in false-positive scenarios within PRMBench, but outperforms most PRMs, including the state-of-the-art QwenPRM-7B, in the BoN evaluation (74.3 vs. 73.4). This inconsistency reveals that PRMs are suboptimal at detecting step-level errors and false positives, exposing potential reward hacking (Gao et al., 2025). Compared to BoN, PRMBench provides a better distinction between models, with a higher standard deviation (6.40 vs. 0.81), indicating its greater sensitivity to fine-grained differences in reasoning steps. Furthermore, based on the discoveries from PRMBench, we provide further discussions including several promising directions for future exploration, which are included in the appendix F.

6 Conclusion

In this paper, we investigate a crucial question: Can existing PRMs detect various types of erroneous reasoning steps and provide reasonable rewards? To address this, we introduce PRMBench, a benchmark characterized by its fine-grained evaluation categories and challenging requirements. We carefully curate 6,216 data samples with 83,456 step-level labels through LLMs and human filtering. PRMBench can be used to evaluate different process-labeling models, ensuring its general applicability. Through a comprehensive evaluation of existing PRMs and generative LLMs prompted as critic models, we can observe that PRMs exhibit partial capability in multi-step process evaluation, showcasing significant room for improvement. Furthermore, we highlight the critical need for detecting detailed error types and conducting comprehensive evaluations of PRMs. Despite these advances, enhancing the reward accuracy of PRMs and improving models’ reasoning abilities remain open research challenges. We encourage future work to leverage and expand upon PRMBench to address these issues.

7 Acknowledgement

We gratefully acknowledge the support and resources provided by the Shanghai Artificial Intelligence Laboratory, which were essential to the successful completion of this research. We are also grateful to Zhiyi Li at The Hong Kong University of Science and Technology for offering valuable examples and inspiration for this study.

8 Limitations

There are still some limitations in our work, which are summarized below:

  • •

    While the PRMBench is large and comprehensive, comprising 6,216 samples and 83,456 step-level labels, a larger dataset could provide more robust evaluation and training opportunities. As the data construction method is flexible and data-agnostic, it can be adapted to more different data sources. We will continuously expand our dataset and explore PRM training in future versions.

  • •

    We evaluate the error-detection capabilities in terms of accuracy. However, a more detailed analysis, including the activation of the model’s neurons and hidden states Zhang et al. (2023), would offer a deeper insight into improvements for PRMs. This limitation is not unique to our study and is common in most evaluations of Large Language Models.

  • •

    PRMBench currently focuses on textual reasoning. In future work, we will further explore its generalization to multimodal reasoning processes, such as those involving the integration of text and images Su et al. (2025); Xia et al. (2024a), to assess error detection capabilities in cross-modal scenarios.

References

  • Asai et al. (2024) Akari Asai, Jacqueline He*, Rulin Shao*, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Tian, D’arcy Mike, David Wadden, Matt Latzke, Minyang, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Dan Weld, Graham Neubig, Doug Downey, Wen-tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. 2024. OpenScholar: Synthesizing scientific literature with retrieval-augmented language models. Arxiv.
  • Cheng et al. (2025) Jie Cheng, Lijun Li, Gang Xiong, Jing Shao, and Yisheng Lv. 2025. Pure: Prm is still effective and compute-efficient for llm math reasoning. https://github.com/CJReinforce/PURE.
  • Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  • DeepMind (2024) DeepMind. 2024. Gemini 2.0 flash experimental. https://deepmind.google/technologies/gemini/flash/. Accessed: 2024-12-25.
  • Gao et al. (2024) Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Wen Xiao, et al. 2024. Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. CoRR.
  • Gao et al. (2025) Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. 2025. On designing effective RL reward at training time for LLM reasoning.
  • Golovneva et al. (2022) Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919.
  • Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  • Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196.
  • Hao et al. (2025) Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444.
  • He et al. (2024a) Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024a. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.
  • He et al. (2024b) Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. 2024b. Opendatalab: Empowering general artificial intelligence with open datasets. arXiv preprint arXiv:2407.13773.
  • Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  • Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  • Li et al. (2024a) Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang, Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. 2024a. Vlrewardbench: A challenging benchmark for vision-language generative reward models. arXiv preprint arXiv:2411.17451.
  • Li et al. (2024b) Xiaoyuan Li, Wenjie Wang, Moxin Li, Junrong Guo, Yang Zhang, and Fuli Feng. 2024b. Evaluating mathematical reasoning of large language models: A focus on error identification and correction. arXiv preprint arXiv:2406.00755.
  • Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  • Lin et al. (2024) Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. Criticbench: Benchmarking llms for critique-correct reasoning. arXiv preprint arXiv:2402.14809.
  • Liu et al. (2024) Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184.
  • Luo et al. (2023) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
  • McAleese et al. (2024) Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215.
  • o1 Team (2024) Skywork o1 Team. 2024. Skywork-o1 open series. https://huggingface.co/Skywork.
  • OpenAI (2024a) OpenAI. 2024a. Gpt-4o system card. https://cdn.openai.com/gpt-4o-system-card.pdf. Accessed: 2024-09-26.
  • OpenAI (2024b) OpenAI. 2024b. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/.
  • Qin et al. (2024) Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. 2024. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982.
  • Qu et al. (2025) Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. 2025. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614.
  • Su et al. (2025) Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. 2025. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617.
  • Su et al. (2024a) Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024a. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in llm. arXiv preprint arXiv:2408.12076.
  • Su et al. (2024b) Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, and Yu Cheng. 2024b. Timo: Towards better temporal reasoning for language models. arXiv preprint arXiv:2406.14192.
  • Team (2024a) Qwen Team. 2024a. Qwq: Reflect deeply on the boundaries of the unknown.
  • Team (2024b) ScalableMath Team. 2024b. Easy-to-hard generalization models. https://huggingface.co/ScalableMath.
  • Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275.
  • Wang et al. (2023) Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abs/2312.08935.
  • Wen et al. (2025) Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, XingYu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. 2025. Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations.
  • Xia et al. (2024a) Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024a. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085.
  • Xia et al. (2024b) Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. 2024b. Evaluating mathematical reasoning beyond accuracy. arXiv preprint arXiv:2404.05692.
  • Xiong et al. (2024) Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang. 2024. An implementation of generative prm. https://github.com/RLHFlow/RLHF-Reward-Modeling.
  • Yan et al. (2024) Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, et al. 2024. Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04509.
  • Yang et al. (2024a) An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024a. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.
  • Yang et al. (2024b) An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024b. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.
  • Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  • Zeng et al. (2024a) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024a. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations.
  • Zeng et al. (2023) Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. 2023. Mr-gsm8k: A meta-reasoning benchmark for large language model evaluation. arXiv preprint arXiv:2312.17080.
  • Zeng et al. (2024b) Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, Linling Shen, Jianqiao Lu, Haochen Tan, Yukang Chen, Hao Zhang, Zhan Shi, Bailin Wang, Zhijiang Guo, and Jiaya Jia. 2024b. Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms. CoRR, abs/2406.13975.
  • Zhang et al. (2024a) Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. 2024a. Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. arXiv preprint arXiv:2411.18203.
  • Zhang et al. (2024b) Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. 2024b. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. arXiv preprint arXiv:2410.02884.
  • Zhang et al. (2025) Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301.
  • Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923.
  • Zheng et al. (2024) Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2024. Processbench: Identifying process errors in mathematical reasoning. arXiv preprint arXiv:2412.06559.
  • Zhou et al. (2024) Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. 2024. Is your model really a good math reasoner? evaluating mathematical reasoning with checklist. arXiv preprint arXiv:2407.08733.

Appendix A Detailed Information for PRMBench

A.1 Evaluation Categories

In this section, We provide detailed information on our evaluation categories. The hierarchical categories, corresponding descriptions, and illustrations are shown in Figure 5. We carefully curated 6,216 data samples and 83,456 step-level labels. The benchmark spreads across three main evaluation categories: simplicity, soundness, and sensitivity. Among them, Simplicity comprises two sub-categories: non-redundancy and non-circular Logic. Soundness includes four main sub-categories: empirical soundness, step consistency, domain consistency, and confidence invariance. Finally, Sensitivity mainly evaluates models in three main parts: prerequisite sensitivity, deception resistance, and multi-solution consistency. The detailed descriptions and illustrations of each sub-category are shown in Figure 5.

Figure 5: Examples of an in-depth evaluation of PRMBench.

A.1.1 Simplicity

Specifically, the Simplicity evaluation category is divided into two sub-categories: Non-Redundancy and Non-Circular Logic, with detailed descriptions provided below:

Non-Redundancy

requires PRM to detect the redundancy within the reasoning procedure. The redundancy situation refers to a process that is not the most concise or efficient, as it includes one or more redundant steps that can be removed without affecting the correctness of the overall solution path. For example, as shown in Figure 5, if A→BA\to B represents a correct inference chain, the redundant reasoning procedure can be displayed as A→C→BA\to C\to B. where CC represents one or more redundant steps C={c|c​ is redundant}C=\{c|c\text{ is redundant}\}.

Non-Circular Logic

In this sub-category, PRMs are required to detect the potential circular logic within the reasoning process. Circular logic is a specific form of redundancy, distinct from general redundancy, in that it finally loops back to a previous reasoning step. For example, as shown in Figure 2, if A→BA\to B represents a correct inference chain, circular logic can be formulated as A→C→A→BA\to C\to A\to B, where the reasoning starts at step AA, progresses through a sequence of steps, and ultimately loops back to AA. We list Non-Circular Logic separately due to its common occurrence in reasoning processes.

Step Descriptions GT ReasonE MathS GPT-4o Gemini
Question
Compute (1+cos⁡π8)​(1+cos⁡3​π8)​(1+cos⁡5​π8)​(1+cos⁡7​π8).\left(1+\cos\frac{\pi}{8}\right)\left(1+\cos\frac{3\pi}{8}\right)\left(1+\cos\frac{5\pi}{8}\right)\left(1+\cos\frac{7\pi}{8}\right).
- - - - -
Step 1
Let’s call the expression we’re trying to compute xx.
✓ 89.1 87.5 100.0 100.0
Step 2
Use the fact that cos⁡(π−a)=−cos⁡a\cos(\pi-a)=-\cos a.
✓ 77.5 82.8 100.0 100.0
Step 3
Then we have x=(1+cos⁡π8)​(1+cos⁡3​π8)​(1+cos⁡5​π8)​(1+cos⁡7​π8)x=\left(1+\cos\frac{\pi}{8}\right)\left(1+\cos\frac{3\pi}{8}\right)\left(1+\cos\frac{5\pi}{8}\right)\left(1+\cos\frac{7\pi}{8}\right).
✓ 71.4 76.7 100.0 100.0
Step 4
This equals (1+cos⁡π8)​(1+cos⁡3​π8)​(1−cos⁡π8)​(1−cos⁡3​π8)\left(1+\cos\frac{\pi}{8}\right)\left(1+\cos\frac{3\pi}{8}\right)\left(1-\cos\frac{\pi}{8}\right)\left(1-\cos\frac{3\pi}{8}\right).
✓ 20.5 76.9 100.0 100.0
Step 5
By the difference of squares, this equals (1−cos2⁡π8)​(1−cos2⁡3​π8)\left(1-\cos^{2}\frac{\pi}{8}\right)\left(1-\cos^{2}\frac{3\pi}{8}\right).
✓ 56.0 83.7 100.0 100.0
Step 6
And by the Pythagorean identity, this equals (sin2⁡π8)​(sin2⁡3​π8)\left(\sin^{2}\frac{\pi}{8}\right)\left(\sin^{2}\frac{3\pi}{8}\right).
✓ 63.5 80.4 100.0 100.0
Step 7
Ok, now we just need to compute the sines.
✓ 84.1 79.1 50.0 50.0
Step 8
We can use the formula for the sum of sines.
✗ 73.6 74.1 0.0 0.0
Step 9
That formula is sin⁡(a)+sin⁡(b)=2​sin⁡(a+b2)​cos⁡(a−b2)\sin(a)+\sin(b)=2\sin\left(\frac{a+b}{2}\right)\cos\left(\frac{a-b}{2}\right).
✗ 79.1 78.5 0.0 100.0
Step 10
That means that sin⁡π8+sin⁡3​π8=2​sin⁡(π/8+3​π/82)​cos⁡(π/8−3​π/82)\sin\frac{\pi}{8}+\sin\frac{3\pi}{8}=2\sin\left(\frac{\pi/8+3\pi/8}{2}\right)\cos\left(\frac{\pi/8-3\pi/8}{2}\right).
✗ 67.4 78.0 0.0 0.0
Step 11
This equals 2​sin⁡π/22​cos⁡(−π/42)2\sin\frac{\pi/2}{2}\cos(-\frac{\pi/4}{2}).
✗ 68.1 72.1 0.0 100.0
Step 12
This equals 2⋅1⋅222\cdot 1\cdot\frac{\sqrt{2}}{2}.
✗ 49.2 73.8 0.0 100.0
Step 13
This equals 2\sqrt{2}.
✗ 35.1 73.8 0.0 100.0
Step 14
But this step doesn’t help us compute the original product of sines.
✗ 72.5 64.3 -50.0 100.0
Step 15
So, (sin2⁡π8)​(sin2⁡3​π8)\left(\sin^{2}\frac{\pi}{8}\right)\left(\sin^{2}\frac{3\pi}{8}\right) remains the same and x=18x=\frac{1}{8}.
✗ 6.3 35.8 0.0 -100.0
Step 16
Therefore, the mistake didn’t change the value of xx.
✓ 22.6 43.5 -100.0 -100.0
Final Acc. - 100 56.2 50.0 93.8 62.5
Reason
A counterfactual step was introduced in steps 8 through 13 by mistakenly using the formula
for the sum of sines instead of the product of sines. This leads to incorrect intermediate calculations.
However, due to fortunate errors, the end result ironically matches the correct answer in step 15.
Table 7: An example of a data instance and error cases from PRMBench. The numbers reported are step-level validity scores generated by models. Scores and labels in red indicate negative samples, while those in green indicate positive samples. “GT” represents ground truth, while “ReasonE,” “MathS,” and “Gemini” correspond to ReasonEval-7B, Math-Shepherd-7B, and Gemini-2.0-flash-thinking-exp, respectively.
Abbr. Full Name Evaluation Category
NR. Non-Redundancy Simplicity
NCL. Non-Circular Logic Simplicity
ES. Empirical Soundness Soundness
SC. Step Consistency Soundness
DC. Domain Consistency Soundness
CI. Confidence Invariance Soundness
PS. Prerequisite Sensitivity Sensitivity
DR. Deception Resistance Sensitivity
MS. Multi-Solution Consistency Sensitivity
Table 8: The impact of ICL few-shot numbers on models’ final performance. The number reported here is PRMScore.

A.1.2 Soundness

We divide the Soundness category into four sub-categories due to its complexity: Empirically Soundness, Step Consistency, Domain Consistency, and Confidence Invariance. The definition of each sub-category is discussed below.

Empirically Soundness

demands PRM to detect the implicit counterfactual mistakes within the reasoning process. A counterfactual step refers to a statement within a reasoning chain that contradicts established ground truth GG. Such contradictions can arise from relying on outdated theories, omitting critical constraints in theory, or incorporating erroneous assumptions.

Step Consistency

expects PRM to detect the implicit step-wise contradiction, which means a conflict between a specific step and other steps within a reasoning path. Given a reasoning path P={S1,S2,…,Sn}P=\{S_{1},S_{2},\dots,S_{n}\}, a step contradiction exists if Si⟂SjS_{i}\perp S_{j}, where i,j∈[1,n]i,j\in[1,n] and i≠ji\neq j.

Domain Consistency

Under this circumstance, PRMs are required to detect potential domain inconsistency mistakes, where domain inconsistency is a special type of counterfactual. It refers to a step within the reasoning chain that uses a statement or theory valid in other domains or cases but is not valid within the current reasoning chain.

Confidence Invariance

demands the PRM to detect over-confident hallucinations, a type of counterfactual where an incorrect statement is made with unwarranted certainty, contradicting established ground truth.

A.1.3 Sensitivity

This category includes three sub-categories: Prerequisite Sensitivity, Deception Resistance, and Multi-Solution Consistency, with detailed descriptions provided below.

Prerequisite Sensitivity

requires the PRM to maintain sensitivity to missing conditions or prerequisite mistakes, which means a flaw in the reasoning chain where critical premises, assumptions, or necessary conditions are absent. This omission results in logical gaps, incomplete reasoning, or biased conclusions. For example, when a missing condition occurs, the model is required to solve the problem through case analysis or further investigation. However, the answer becomes incorrect if the model overlooks the missing condition and proceeds with standard reasoning methods.

Deception Resistancy

demands the PRM to detect the implicit deception or trap within a reasoning process, that is, statements that appear to be correct or align with ground truth but are subtly altered to introduce inaccuracies while maintaining the illusion of correctness.

Multi-Solution Consistency

expects the PRM to maintain consistency when faced with different solution paths of the same problem. Concretely, to evaluate the sensitivity and the generalizability of PRMs, we utilize multiple correct reasoning processes of the same question to test whether the PRM can perform correctly.

A.2 Examples For Different Evaluation Categories

In this section, we provide detailed examples of the various evaluation categories and their corresponding sub-categories. The data instance examples are displayed in Figure 7-18. All datasets used in this work are publicly available and have been released by their original creators, who are responsible for ensuring privacy protection. These datasets are utilized in accordance with their respective licenses and intended purposes, without introducing any harmful or sensitive information.

A.3 Human Annotation Settings

Refer to caption
Figure 6: The data annotation platform.

A.3.1 Quality Control

In the second stage of the quality control process, we recruited five volunteers, each holding a bachelor’s degree or an equivalent qualification, to assess the correctness and validity of PRMBench. To facilitate high-quality labeling, we utilize LabelLLM He et al. (2024b) to help the data annotation procedure, as shown in Figure 6. The annotators’ instructions are shown in Appendix D.

A.3.2 Human Performance Evaluation

For human performance evaluation, we recruited three volunteers, each holding a bachelor’s degree or an equivalent qualification, to assist with data annotation. Following Hao et al. (2025), we randomly selected 50 instances from each sub-category, resulting in a mini-test set of 450 samples. Each annotator was responsible for three subsets, and the results are presented in Table 3.

Appendix B Detailed Experiment Results

B.1 Abbreviation Of Sub-Categories

The full names of abbreviations used in our tables are shown in Table 8.

B.2 Models

To provide a comprehensive evaluation of various models on PRMBench, we select a diverse set of models, including both open-source PRMs and different types of LLMs configured as critic models. Specifically, the open-source PRMs include Skywork-PRM-1.5B/7B o1 Team (2024), LlemmaPRMs Team (2024b), MathMinosPRM Gao et al. (2024), MathShepherd-Mistral-7B Wang et al. (2023), ReasonEval-7B/34B Xia et al. (2024b), Pure-PRM-7B Cheng et al. (2025) and Qwen-PRM-7B/72B Zhang et al. (2025). Additionally, we evaluate state-of-the-art general-purpose LLMs, including the open-source MetaMath-7B/34B Yu et al. (2023), Qwen2.5-Math-72B Yang et al. (2024b), DeepSeek-R1 and its distill series Guo et al. (2025), as well as closed-source LLMs such as GPT-4o OpenAI (2024a), Gemini-2.0-flash DeepMind (2024), and multi-step reasoning-enhanced LLMs like the o1 series models OpenAI (2024b) and Gemini-2-Thinking DeepMind (2024).

All models used in this work are publicly available and have been released by their original authors, who are responsible for ensuring privacy protection. These models are utilized in accordance with their respective licenses and intended purposes, without any modifications.

B.3 Evaluation Procedure

For each annotated question-solution pair, the reward models are tasked with evaluating the correctness and redundancy of each step, assigning a step-level validity score and a step-level redundancy score to each step. We subsequently utilize the specified threshold of each model to obtain the prediction indicating whether the step is correct or redundant. This task is therefore framed as a binary classification problem. Thus we can utilize the evaluation metric defined in Section 4.2 to evaluate the performance of models on PRMBench.

B.4 Detailed Results of PRMBench

In addition to PRMScore displayed in Table 3, we also list the full results with different metrics across different sub-categories here. The detailed evaluation results are shown in Table 15-23.

B.5 Detailed Results of PRMBench-STEM

We extend our benchmark by collecting additional data from various scientific domains, including physics, chemistry, and biology, and construct PRMBench-STEM using a similar data curation methodology as described in the main paper. The statistics of PRMBench-STEM are presented in Table 9. We then evaluate several representative PRMs on PRMBench-STEM, and the results are summarized in Tables 10, 11, and 12 for the Biology, Chemistry, and Physics subsets, respectively. As shown, PRMs exhibit weaker performance on PRMBench-STEM compared to their performance on the original PRMBench. Notably, the Simplicity category remains the most challenging, consistent with our earlier observations.

Domain # of Instances Avg. Step Num Avg. Error Num
Physics 1619 7.32 2.74
Chemistry 1543 7.74 2.78
Biology 2342 6.76 2.49
Table 9: Statistics of PRMBench-STEM.
Model Overall Simplicity Soundness Sensitivity
NR. NCL. Avg. ES SC. DC. CI Avg. PS DR. Avg.
Open-source Process Level Reward Models
Skywork-PRM-1.5B 47.3 46.7 41.6 44.2 45.0 54.1 55.3 49.6 51.0 43.2 41.7 42.4
Skywork-PRM-7B 67.4 54.6 53.6 54.1 71.6 76.2 77.9 74.4 75.0 64.0 65.3 64.6
Llemma-PRM800k-7B 52.7 49.1 52.4 50.7 52.3 49.4 53.9 55.1 52.7 51.1 52.6 51.9
Llemma-MetaMath-7B 41.5 43.7 39.1 41.4 39.3 44.8 44.3 42.0 42.6 38.5 38.2 38.4
Llemma-oprm-7B 50.7 44.4 37.3 40.9 54.5 50.3 53.0 56.8 53.6 51.0 53.6 52.3
MATHMinos-Mistral-7B 47.1 43.0 37.9 40.4 48.0 51.6 47.1 53.6 50.1 45.1 45.3 45.2
MathShepherd-Mistral-7B 53.1 47.2 43.1 45.1 52.3 62.1 63.5 57.5 58.9 50.0 46.4 48.2
ReasonEval-7B 67.5 74.5 62.9 68.7 63.5 74.5 71.8 71.8 70.4 59.9 59.3 59.6
ReasonEval-34B 60.7 68.2 61.9 65.0 61.0 53.8 52.0 59.9 56.7 57.8 62.6 60.2
RLHFlow-PRM-Mistral-8B 45.5 43.0 37.4 40.2 44.5 53.1 49.0 50.3 49.2 42.6 42.2 42.4
RLHFlow-PRM-Deepseek-8B 47.0 42.9 37.4 40.1 44.7 59.0 57.7 52.6 53.5 41.6 41.6 41.6
Qwen2.5-Math-PRM-7B 65.8 42.9 49.0 46.0 69.6 75.1 72.5 75.7 73.2 63.9 65.4 64.7
Qwen2.5-Math-PRM-72B 73.7 42.8 50.6 46.7 82.3 79.6 77.6 82.9 80.6 75.5 76.6 76.1
Avg. 55.4 49.5 46.5 48.0 56.1 60.3 59.7 60.2 59.0 52.6 53.1 52.9
Open LLMs, Prompted as Critic Models
MetaMath-7B 45.1 45.3 35.8 40.6 45.8 48.8 48.8 47.1 47.6 43.6 43.7 43.6
MetaMath-13B 43.6 44.7 35.8 40.2 42.2 46.3 47.1 45.0 45.2 42.4 41.1 41.7
Qwen2.5-Math-72B 42.3 49.5 35.6 42.5 38.0 56.1 45.7 42.8 45.6 37.0 35.6 36.3
QwQ-Preview-32B 54.5 54.8 43.3 49.0 53.6 63.9 64.4 58.5 60.1 51.8 48.3 50.1
R1-Distill-Llama3.1-70B 53.8 49.8 41.7 45.7 53.4 63.6 71.7 57.8 61.6 46.7 44.5 45.6
R1-Distill-Qwen-7B 50.4 39.9 41.8 40.9 46.4 48.7 50.9 47.4 48.3 45.6 43.3 44.5
Avg. 48.3 47.3 39.0 43.2 46.6 54.6 54.8 49.8 51.4 44.5 42.7 43.6
Table 10: Performance comparison of popular models on the Biology subset of PRMBench-STEM. The best performance for each category and task is in bold, while the second-best performance is underlined.
Model Overall Simplicity Soundness Sensitivity
NR. NCL. Avg. ES SC. DC. CI Avg. PS DR. Avg.
Open-source Process Level Reward Models
Skywork-PRM-1.5B 48.1 45.9 43.9 44.9 46.3 52.5 54.0 51.3 51.0 45.3 43.3 44.3
Skywork-PRM-7B 64.3 51.2 51.1 51.1 65.2 70.4 71.6 73.3 70.1 64.2 64.6 64.4
Llemma-PRM800k-7B 52.6 47.1 53.2 50.2 56.5 48.8 47.9 55.7 52.2 52.6 55.2 53.9
Llemma-MetaMath-7B 45.2 45.3 41.5 43.4 44.9 48.2 46.3 47.6 46.7 42.3 42.8 42.5
Llemma-oprm-7B 48.3 44.7 38.1 41.4 53.5 46.6 41.6 56.2 49.5 49.0 52.4 50.7
MATHMinos-Mistral-7B 49.6 44.0 38.8 41.4 50.8 51.9 46.3 58.2 51.8 49.7 50.4 50.0
MathShepherd-Mistral-7B 52.5 47.8 43.4 45.6 55.7 56.3 54.3 58.4 56.2 50.2 49.5 49.8
ReasonEval-7B 62.9 69.4 60.1 64.7 59.2 66.3 64.1 70.3 65.0 57.4 55.5 56.4
ReasonEval-34B 63.2 63.5 55.8 59.6 65.6 58.6 57.8 66.9 62.2 61.9 65.6 63.7
RLHFlow-PRM-Mistral-8B 45.6 43.3 38.5 40.9 44.2 50.9 47.7 51.2 48.5 44.1 42.3 43.2
RLHFlow-PRM-Deepseek-8B 47.3 43.5 38.3 40.9 46.9 51.8 50.5 56.3 51.4 45.4 43.1 44.2
Qwen2.5-Math-PRM-7B 66.3 43.5 49.9 46.7 70.9 66.8 68.8 80.0 71.6 67.0 68.1 67.5
Qwen2.5-Math-PRM-72B 71.4 44.8 53.2 49.0 77.3 71.6 74.6 81.0 76.1 73.1 77.5 75.3
Avg. 55.2 48.8 46.6 47.7 56.7 57.0 55.8 62.0 57.9 54.0 54.6 54.3
Open LLMs, Prompted as Critic Models
MetaMath-7B 45.6 44.2 37.5 40.9 46.0 48.5 48.0 47.3 47.4 45.6 45.5 45.5
MetaMath-13B 44.3 42.9 37.0 40.0 44.7 48.6 47.0 45.3 46.4 45.2 42.2 43.7
Qwen2.5-Math-72B 43.2 50.5 34.8 42.6 37.6 61.7 43.8 43.3 46.6 37.5 36.6 37.0
R1-Distill-Llama3.1-70B 50.6 45.9 41.0 43.5 48.2 65.7 58.1 57.8 57.5 46.7 42.8 44.8
R1-Distill-Qwen-7B 51.8 35.6 41.0 38.3 47.5 51.0 50.1 49.2 49.5 47.1 46.2 46.7
Avg. 47.1 43.8 38.2 41.0 44.8 55.1 49.4 48.6 49.5 44.4 42.7 43.5
Table 11: Performance comparison of popular models on the Chemistry subset of PRMBench-STEM. The best performance for each category and task is in bold, while the second-best performance is underlined.
Model Overall Simplicity Soundness Sensitivity
NR. NCL. Avg. ES SC. DC. CI Avg. PS DR. Avg.
Open-source Process Level Reward Models
Skywork-PRM-1.5B 50.3 44.7 41.8 43.3 51.8 57.7 55.2 56.1 55.2 46.8 46.8 46.8
Skywork-PRM-7B 65.2 51.1 51.9 51.5 68.3 70.3 72.7 76.4 71.9 64.8 65.1 65.0
Llemma-PRM800k-7B 53.8 49.1 52.0 50.6 57.3 49.6 49.4 57.3 53.4 53.8 58.6 56.2
Llemma-MetaMath-7B 46.9 46.4 42.0 44.2 46.5 47.2 44.2 51.7 47.4 47.8 45.4 46.6
Llemma-oprm-7B 47.1 42.8 39.7 41.2 52.2 42.7 40.9 54.3 47.5 48.1 51.6 49.8
MATHMinos-Mistral-7B 46.5 43.6 38.9 41.2 46.9 47.8 44.2 57.8 49.2 45.3 43.3 44.3
MathShepherd-Mistral-7B 53.5 48.0 44.8 46.4 55.9 56.3 54.8 62.8 57.4 48.6 53.5 51.1
ReasonEval-7B 62.5 71.5 57.2 64.4 60.1 66.9 62.2 70.6 65.0 56.9 53.9 55.4
ReasonEval-34B 64.2 68.5 52.0 60.2 68.5 59.3 56.4 69.8 63.5 63.7 66.6 65.2
RLHFlow-PRM-Mistral-8B 46.5 42.7 37.5 40.1 46.0 51.0 50.5 55.7 50.8 44.2 42.2 43.2
Qwen2.5-Math-PRM-7B 66.9 43.7 49.3 46.5 73.6 66.4 70.3 80.0 72.6 69.6 68.8 69.2
Qwen2.5-Math-PRM-72B 71.1 44.2 56.2 50.2 78.9 71.1 71.1 80.3 75.4 72.9 77.6 75.2
Avg. 55.6 45.9 46.3 46.1 58.1 56.8 55.7 64.2 58.7 54.5 55.2 54.8
Open LLMs, Prompted as Critic Models
MetaMath-7B 45.6 43.5 36.9 40.2 47.9 48.9 49.1 48.1 48.5 47.3 43.6 45.4
MetaMath-13B 43.8 42.7 36.6 39.7 44.5 46.7 48.0 46.3 46.4 41.5 41.9 41.7
Qwen2.5-Math-72B 44.8 50.6 37.8 44.2 39.8 64.5 45.5 47.5 49.3 37.9 36.2 37.1
R1-Distill-Llama3.1-70B 53.3 44.5 40.4 42.5 53.0 63.8 59.4 64.6 60.2 49.6 48.3 48.9
R1-Distill-Qwen-7B 51.0 29.7 34.8 32.3 44.1 48.1 45.7 48.9 46.7 44.0 40.7 42.3
Avg. 47.7 42.2 37.3 39.8 45.9 54.4 49.5 51.1 50.2 44.0 42.2 43.1
Table 12: Performance comparison of popular models on the Physics subset of PRMBench-STEM. The best performance for each category and task is in bold, while the second-best performance is underlined.

Appendix C Details for Further Analysis

C.1 Error Analysis

A representative test case and the corresponding model performances are presented in Table 7. This example involves a counterfactual reasoning process, where steps eight through thirteen contain information that contradicts the correct computational principles and should be classified as “negative”. However, most models fail to identify these erroneous reasoning steps and assign relatively positive rewards, except for GPT-4o. While GPT-4o provides a relatively accurate reward, its judgments for key steps are only marginally negative, reflecting low confidence. This highlights a significant room for improvement in PRMs’ detailed error-detection capabilities.

Method GSM8K MATH OlymBen MMLU Avg. PRMScore
Pass@8 98.9 96.2 79.8 96.7 92.9 -
Maj@8 95.4 71.8 40.3 85.8 73.3 -
Skywork-PRM-1.5B 96.7 89.2 58.8 90.2 83.7 61.1
Skywork-PRM-7B 97.1 90.0 60.1 90.3 84.4 65.1
Llemma-PRM800k-7B 96.0 87.4 58.3 90.0 82.9 52.0
Llemma-MetaMath-7B 96.0 88.2 58.6 90.0 83.2 50.5
Llemma-oprm-7B 96.4 86.6 58.0 89.9 82.7 50.3
MATHMinos-Mistral-7B 95.8 88.3 59.1 89.1 83.1 54.2
MathShepherd-Mistral-7B 96.6 88.6 60.0 90.0 83.8 47.0
ReasonEval-7B 96.4 87.0 58.4 90.0 82.9 60.1
RLHFlow-PRM-Mistral-8B 96.5 88.4 59.1 90.4 83.6 54.4
RLHFlow-PRM-Deepseek-8B 96.4 87.6 58.5 90.2 83.2 54.2
ReasonEval-34B 96.4 86.9 56.9 90.0 82.6 60.5
Qwen2.5-Math-PRM-7B 96.7 88.0 58.7 89.8 83.3 65.5
Qwen2.5-Math-PRM-72B 96.7 89.3 60.5 90.0 84.2 68.2
Standard Deviation (σ\sigma) 0.35 0.97 0.93 0.29 0.53 6.41
Somers’ D correlation 0.50 0.26 0.22 0.19 0.28 1.00
Table 13: Performance comparison on Best-of-8 using different PRMs. σ\sigma represents the standard deviation of model performances across all benchmarks. Somers’ D refers to the Somers’ D correlation between PRMScore and specific benchmarks.
Method GSM8K MATH OlymBen MMLU Avg. PRMScore
Pass@8 99.0 96.0 77.0 94.0 91.5 -
Maj@8 96.5 68.0 41.0 86.0 72.9 -
QwQ-Preview-32B 96.5 83.5 56.5 87.5 81.0 63.6
MetaMath-7B 95.5 82.0 58.0 88.0 80.9 49.7
MetaMath-13B 95.0 84.5 57.5 86.5 80.9 49.4
MetaMath-70B 96.5 85.0 58.5 86.0 81.5 45.9
Qwen2.5-Math-7B 96.0 85.5 59.0 86.5 81.8 49.2
Qwen2.5-Math-72B 97.5 82.0 53.5 88.5 80.4 57.4
R1-Distill-Llama3.1-8B 96.5 84.5 53.5 87.5 80.5 52.7
R1-Distill-Llama3.1-70B 95.0 83.0 59.5 86.5 81.0 57.5
R1-Distill-Qwen-7B 97.0 86.5 59.0 86.5 82.2 52.6
R1-Distill-Qwen-32B 97.0 82.0 60.5 84.5 81.0 60.2
WizardMath-7B 96.5 82.5 60.0 87.0 81.5 49.2
Gemini-2.0-flash-exp 97.0 81.5 56.5 86.5 80.4 66.0
Gemini-2.0-thinking-exp-1219 98.0 87.5 60.5 89.5 83.9 68.8
Standard Deviation (σ\sigma) 0.87 1.83 2.25 1.19 0.91 7.02
Somers’ D correlation 0.36 -0.21 0.03 0.24 -0.15 1.00
Table 14: Performance comparison on Best-of-8 using different LLMs as a Judge. σ\sigma represents the standard deviation of model performances across all benchmarks. Somers’ D refers to the Somers’ D correlation between PRMScore and specific benchmarks.

C.2 Details for BoN Evaluation

Following Zhang et al. (2025), we sample eight responses (i.e., N=8) from Qwen-QwQ-Preview 32B Team (2024a). During evaluation, the PRMs are tasked with assigning a validity score to each step within every candidate response. The overall score for each candidate response is calculated by multiplying the individual step scores, as outlined in Lightman et al. (2023). We then select the highest-ranked candidate response, compare it with the correct answer, and calculate the accuracy, which we refer to as prm@8. Additionally, we report the result of majority voting among the eight sampled responses (maj@8) as the baseline, and we define pass@8 as the proportion of test samples where any of the eight samplings lead to the correct final answer, which serves as the upper bound.

We conduct BoN evaluation across all models on GSM8K Cobbe et al. (2021), MATH Hendrycks et al. (2021), Olympiad Bench He et al. (2024a) and MMLU Hendrycks et al. (2020). Due to the limit of the space, We omit some results in Table 6. Thus we provide the full results of all PRMs in Table 13. Moreover, due to cost constraints, we select a subset of 200 samples for each benchmark and obtain a total of 800 samples to evaluate generative LLMs of their BoN performance. The results are shown in Table 14.

Appendix D Instructions for Human Annotators

D.1 Backgrounds

With the emergence of multi-step reasoning enhanced language models such as OpenAI o1, and Deepmind Gemini-thinking, these models demonstrate the ability to decompose complex problems and solve them step by step. However, while their solutions often appear correct, they may contain errors in understanding, calculation, or reasoning logic, which is also known as false positive situations. A popular way to evaluate the results generated by these models is by utilizing process-level reward models (PRMs). Nevertheless, PRMs are fallible and not always correct. Existing benchmarks are not adequate for evaluating PRMs on different error types. Therefore, we are building a comprehensive evaluation benchmark for PRMs that can have a fine-grained detection of PRMs.

D.2 Task Definition

We begin by collecting completely correct multi-solution data and leveraging state-of-the-art LLMs to introduce various types of errors into these correct solutions, thereby generating our test cases. The detailed error types are described in Section 3. All synthesized data instances undergo an initial filtering process based on specific features.

Your task is to identify whether the modification taken is reasonable and whether the modified data instance is different from the original data instance.

Sub-task 1

The first sub-task is a binary classification task whose options include yes and no. Your task is to decide whether the modified step-by-step solution generated by LLMs is reasonable. The word “reasonable” has two aspects for evaluation.

  • •

    The modified process generated by LLMs seems like a possible solution path that could happen.

  • •

    The modified process generated by LLMs is exactly wrong and the type of error is suitable for the current “classification”.

Please assign a “yes” for this sub-task if both of the answers to the above two questions are “yes”. Otherwise, assign a “no” for this sub-task.

D.2.1 Sub-task 2

The second sub-task is a binary classification task whose options include yes and no. Your task is to decide whether the modified step-by-step solution generated by LLMs is different from the original solution process. The word “different” means the modified solution process is logically different from the original one, or there exist different statements compared to the original process.

Please assign a “yes” to this sub-task if your answer to the above question is “yes”. Otherwise, assign a “no” for this sub-task.

D.2.2 Error Types

Redundancy

refers to a process that is not the most concise or efficient, as it includes one or more redundant steps that can be removed without affecting the correctness of the overall solution path. For example, if A→BA\to B represents a correct inference chain, your task is to introduce one or more redundant steps C={c|c​ is redundent}C=\{c|c\text{ is redundent}\} and reformulate the solution chain as A→C→BA\to C\to B.

Circular logic

is a specific form of redundancy, characterized by a reasoning chain that starts at a step SS, progresses through a sequence of steps, and ultimately loops back to SS. Symbolically, this can be expressed as S→A→B→SS\to A\to B\to S, where SS, AA, and BB represent individual reasoning steps. Your task is to modify the reasoning process to introduce such circular logic.

Counterfactual

A counterfactual step refers to a statement within a reasoning chain that contradicts ground truth or established theories. Such contradictions can arise from relying on outdated theories, omitting critical constraints in a theory, or incorporating erroneous assumptions. Your task is to modify the reasoning process to introduce such counterfactual steps.

Step contradiction

refers to a conflict between a specific step and other steps within a reasoning path. Given a reasoning path P=S1,S2,…,SnP={S_{1},S_{2},\dots,S_{n}}, a step contradiction exists if Si⟂SjS_{i}\perp S_{j}, where i,j∈[1,n]i,j\in[1,n] and i≠ji\neq j. Your task is to modify the reasoning process to introduce such step contradiction steps.

Domain inconsistency

is a special type of counterfactual. It refers to a step or a few steps within the reasoning chain that uses a statement or theory valid in other domains or cases but is not valid within the current reasoning chain. Your task is to modify the reasoning process to introduce steps for such domain inconsistency.

Confident hallucination

is a special type of counterfactual. It refers to a statement within the reasoning chain that contradicts established ground truth and is presented with an overly confident tone. In other words, it involves stating an incorrect statement with unwarranted certainty. Your task is to modify the reasoning process to introduce such confident hallucination steps.

Missing condition or prerequisite

refers to a flaw in the reasoning chain where critical premises, assumptions, or necessary conditions are absent. This omission results in logical gaps, incomplete reasoning, or biased conclusions. For example, when a missing condition occurs, the model must solve the problem through case analysis or further investigation. However, the answer becomes incorrect if the model overlooks the missing condition and proceeds with standard reasoning methods. Your task is to modify the reasoning process to introduce such missing condition errors.

Deception or traps

refer to statements that appear to be correct or align with ground truth but are subtly altered to introduce inaccuracies while maintaining the illusion of correctness. Your task is to modify the reasoning process to introduce such deception or trap error steps.

Appendix E Prompts

E.1 Prompts For Generating Data

As introduced in Section 3.2, we query GPT-4o OpenAI (2024a) to synthesize the metadata at the very first step of our test case construction procedure. To better prompt LLMs to generate high-quality data instances, we carefully designed our prompts, which are displayed in Figure 19-22. We display only one example here due to limitations in space. And the prompts can be found in our supplementary materials.

E.2 Prompts For Evaluating Generative LLMs

As introduced in Section 4.1, we prompt some state-of-the-art generative LLMs as critic models to evaluate their rewarding capabilities on PRMBench. To make a fair comparison between different models, we carefully design the prompts and utilize a unified prompt to query them. The prompt used is displayed in Figure 23 and 24.

Appendix F Further Discussion

Inspired by the results and discoveries on PRMBench, we further propose several promising directions for future research, which we hope can offer valuable insights and contribute to the advancement of the research community.

Anti-redundancy training:

As stated in Section 4.3, our work highlights a specific weakness of current PRMs in identifying redundant reasoning steps. To mitigate this, one possible approach is to modify the label distribution during training. PRM training data is typically labeled as Correct, Neutral, or Incorrect, where the Neutral label often corresponds to redundant steps. By reducing the proportion of Neutral samples in the training data, we can train PRMs with stronger anti-redundancy capabilities.

Contrastive training:

A high-quality data curation pipeline is introduced in Section 3.1, which can also be adapted to curate training samples labeled with fine-grained error types. By leveraging contrastive learning or preference alignment with the curated data, the error sensitivity and detection capabilities of PRMs can be further improved.

Step-level evaluation for LLMs:

As introduced in Section 5.4, the inconsistency between PRMBench and BoN evaluation reveals the false positive situation and the risk of reward hacking within LM post-training. Therefore, traditional outcome-level label-based evaluation is not enough, highlighting the need for a comprehensive step-level evaluation of LLM’s reasoning procedure.

Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 61.1 88.6 33.6 80.6 88.8 33.2 44.7 91.7
Skywork-PRM-7B 65.1 89.2 40.9 81.7 88.5 42.7 56.6 90.1
Llemma-PRM800k-7B 52.0 75.7 28.3 63.7 66.4 48.5 22.2 79.5
Llemma-MetaMath-7B 50.5 80.4 20.7 68.5 75.6 27.7 15.1 84.2
Llemma-oprm-7B 50.3 77.3 23.3 64.9 69.9 36.1 16.6 83.7
MATHMinos-Mistral-7B 54.2 79.2 29.1 67.9 72.8 41.7 38.0 84.6
MathShepherd-Mistral-7B 47.0 64.9 29.2 53.0 51.5 61.1 54.6 83.5
ReasonEval-7B 60.1 90.8 29.3 83.8 95.5 21.2 30.3 91.6
ReasonEval-34B 60.5 83.8 37.2 74.2 79.1 48.4 50.8 82.8
RLHFlow-PRM-Mistral-8B 54.4 87.7 21.1 78.8 90.2 17.9 22.1 92.8
RLHFlow-PRM-Deepseek-8B 54.2 89.9 18.6 82.0 95.0 13.0 17.0 95.0
Qwen2.5-Math-PRM-7B 65.5 91.5 39.4 85.1 95.4 30.6 37.8 89.0
Qwen2.5-Math-PRM-72B 68.2 91.4 45.1 85.1 93.8 38.7 48.5 86.8
Pure-PRM-7B 65.3 90.1 40.5 83.0 91.8 36.6 41.8 86.6
Avg. 57.7 84.3 31.2 75.2 82.4 35.5 35.4 87.3
Open LLMs, Prompted as Critic Models
MetaMath-7B 49.7 88.2 11.2 79.1 90.9 9.1 7.6 91.6
MetaMath-13B 49.4 89.6 9.1 81.4 94.4 6.3 4.8 94.8
Qwen2.5-Math-72B 57.4 90.3 24.4 82.9 96.9 15.9 19.8 92.4
QwQ-Preview-32B 63.6 87.6 39.6 79.4 89.2 36.4 40.2 83.1
R1-Distill-Llama3.1-70B 57.5 91.4 23.5 84.6 97.4 15.2 19.0 93.8
R1-Distill-Qwen-7B 52.6 83.2 22.0 72.3 80.5 26.0 23.9 79.3
DeepSeek-R1† 67.8 87.2 48.4 79.5 83.1 60.5 63.7 77.9
Avg. 56.8 88.2 25.4 79.9 90.3 24.2 25.6 87.5
Proprietary LLMs, Prompted as Critic Models
GPT-4o 66.8 86.9 46.7 79.0 82.9 58.2 64.4 76.6
o1-mini† 68.8 89.2 48.3 82.1 86.9 55.4 56.4 80.4
Gemini-2.0-flash-exp 66.0 86.5 45.5 78.4 82.3 57.2 64.3 80.0
Gemini-2.0-thinking-exp-1219 68.8 89.7 47.8 82.8 89.0 49.8 57.0 82.0
Avg. 67.6 88.1 47.1 80.6 85.3 55.2 60.5 79.7
Table 15: A performance comparison of popular models across detailed metrics in ALL categories of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 52.0 88.4 15.6 79.6 88.8 15.1 18.7 95.0
Skywork-PRM-7B 56.4 89.0 23.9 80.7 88.8 24.2 31.3 94.5
Llemma-PRM800k-7B 49.3 77.7 20.9 65.2 69.2 36.9 20.5 86.4
Llemma-MetaMath-7B 50.2 85.3 15.0 75.0 83.2 17.6 15.3 94.4
Llemma-oprm-7B 48.7 80.3 17.2 68.1 74.1 26.5 16.0 92.0
MATHMinos-Mistral-7B 48.8 82.7 14.9 71.2 79.2 19.0 20.8 95.7
MathShepherd-Mistral-7B 44.0 67.7 20.2 54.0 55.6 43.7 39.7 93.8
ReasonEval-7B 61.0 91.5 30.5 84.9 94.0 25.1 32.8 89.8
ReasonEval-34B 54.8 86.7 22.9 77.4 85.3 25.3 31.6 83.8
RLHFlow-PRM-Mistral-8B 46.1 89.1 3.2 80.3 92.3 2.4 3.4 98.5
RLHFlow-PRM-Deepseek-8B 46.4 91.0 1.9 83.5 96.1 1.2 1.5 99.0
Qwen2.5-Math-PRM-7B 49.0 92.1 5.9 85.4 98.0 3.5 3.3 98.2
Qwen2.5-Math-PRM-72B 50.4 91.9 8.8 85.2 97.4 5.4 6.2 98.2
Pure-PRM-7B 49.2 90.8 7.6 83.3 95.2 5.2 5.4 96.2
Avg. 50.5 86.0 14.9 76.7 85.5 17.9 17.6 94.0
Open LLMs, Prompted as Critic Models
MetaMath-7B 48.9 80.7 17.0 68.7 74.9 25.5 23.2 76.3
MetaMath-13B 50.3 86.5 14.0 76.7 85.7 14.9 11.8 85.4
Qwen2.5-Math-72B 55.3 90.3 20.3 82.7 93.9 15.4 17.0 90.1
QwQ-Preview-32B 57.2 85.8 28.7 76.3 84.3 31.4 33.9 81.9
R1-Distill-Llama3.1-70B 49.5 92.5 6.5 86.1 98.8 3.6 5.4 98.0
R1-Distill-Qwen-7B 32.9 43.9 21.9 34.7 29.4 70.1 68.6 31.5
DeepSeek-R1† 63.0 86.9 39.2 78.4 82.2 53.3 62.5 76.5
Avg. 51.0 80.9 21.1 71.9 78.5 30.6 31.8 77.1
Proprietary LLMs, Prompted as Critic Models
GPT-4o 57.0 77.8 36.3 67.0 66.5 70.4 77.1 68.9
o1-mini† 65.6 90.8 40.4 84.1 90.4 41.5 45.8 84.1
Gemini-2.0-flash-exp 67.2 91.5 42.9 85.1 91.8 41.7 49.7 82.4
Gemini-2.0-thinking-exp-1219 68.5 91.4 45.6 85.1 90.9 47.1 56.5 84.7
Avg. 64.6 87.9 41.3 80.4 84.9 50.2 57.3 80.0
Table 16: A performance comparison of popular models across detailed metrics in NR. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 35.8 34.7 36.9 35.8 22.7 75.2 71.4 72.9
Skywork-PRM-7B 41.2 44.2 38.1 41.3 30.9 72.6 71.6 73.5
Llemma-PRM800k-7B 53.4 65.5 41.3 56.6 54.9 61.4 15.7 65.6
Llemma-MetaMath-7B 50.5 70.8 30.3 58.8 66.4 36.0 15.7 74.8
Llemma-oprm-7B 49.3 66.3 32.2 55.0 59.0 42.9 10.2 73.3
MATHMinos-Mistral-7B 54.0 70.8 37.2 60.2 66.6 43.2 40.6 79.1
MathShepherd-Mistral-7B 50.3 60.4 40.2 52.3 49.9 58.6 50.3 79.0
ReasonEval-7B 50.1 80.8 19.4 69.0 89.8 13.6 19.0 88.1
ReasonEval-34B 48.1 76.7 19.5 63.9 81.9 16.0 19.9 83.4
RLHFlow-PRM-Mistral-8B 47.3 79.8 14.9 67.3 88.7 10.5 18.9 93.7
RLHFlow-PRM-Deepseek-8B 48.9 82.3 15.4 70.7 93.6 9.8 15.6 94.6
Qwen2.5-Math-PRM-7B 55.1 84.5 25.7 74.3 96.1 16.3 22.7 91.6
Qwen2.5-Math-PRM-72B 58.8 84.5 33.1 74.8 94.3 22.8 33.5 89.4
Pure-PRM-7B 55.2 82.6 27.8 72.0 91.6 19.7 24.3 88.0
Avg. 49.9 70.3 29.4 60.9 70.5 35.6 30.7 81.9
Open LLMs, Prompted as Critic Models
MetaMath-7B 46.9 74.9 19.0 61.7 75.2 18.7 13.6 77.8
MetaMath-13B 44.4 76.4 12.4 62.8 81.0 10.2 7.3 83.8
Qwen2.5-Math-72B 54.9 81.2 28.6 70.2 89.5 21.1 27.7 81.5
QwQ-Preview-32B 55.6 76.1 35.2 65.0 77.9 33.1 37.2 75.8
R1-Distill-Llama3.1-70B 48.1 84.4 11.8 73.5 98.1 6.6 8.5 95.6
R1-Distill-Qwen-7B 37.9 38.2 37.7 37.9 25.8 73.4 73.3 27.2
DeepSeek-R1† 62.7 76.4 49.0 67.7 71.6 57.3 61.7 -
Avg. 50.1 72.5 27.7 62.7 74.2 31.5 32.8 73.6
Proprietary LLMs, Prompted as Critic Models
GPT-4o 62.4 73.5 51.3 65.6 65.4 66.2 80.6 59.9
o1-mini† 63.7 80.4 47.0 71.4 80.6 46.6 47.9 -
Gemini-2.0-flash-exp 58.1 81.8 34.5 71.5 88.1 27.4 34.7 79.4
Gemini-2.0-thinking-exp-1219 63.8 81.2 46.4 72.2 82.8 44.0 54.8 74.6
Avg. 62.0 79.2 44.8 70.2 79.2 46.1 54.5 71.3
Table 17: A performance comparison of popular models across detailed metrics in NCL. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 32.4 32.9 32.0 32.4 20.3 85.4 76.4 72.2
Skywork-PRM-7B 36.7 39.1 34.4 36.8 24.9 88.9 82.5 67.6
Llemma-PRM800k-7B 56.4 76.8 36.1 65.9 69.2 51.7 20.0 81.2
Llemma-MetaMath-7B 51.9 78.0 25.8 66.0 73.9 31.7 15.4 83.3
Llemma-oprm-7B 54.2 77.3 31.1 65.9 71.4 41.6 15.6 83.5
MATHMinos-Mistral-7B 57.0 77.3 36.7 66.6 71.1 48.6 39.7 82.0
MathShepherd-Mistral-7B 49.4 62.4 36.4 52.7 48.9 68.0 58.0 81.6
ReasonEval-7B 62.1 89.7 34.6 82.2 96.6 23.7 37.4 91.2
ReasonEval-34B 66.4 83.3 49.4 74.9 78.1 61.9 61.1 81.5
RLHFlow-PRM-Mistral-8B 56.6 85.8 27.4 76.2 89.6 22.5 24.5 90.5
RLHFlow-PRM-Deepseek-8B 55.7 87.8 23.5 79.0 94.6 16.2 21.4 93.5
Qwen2.5-Math-PRM-7B 71.8 90.8 52.8 84.6 94.8 43.3 56.2 83.7
Qwen2.5-Math-PRM-72B 73.7 90.7 56.8 84.7 93.2 50.6 66.9 81.4
Pure-PRM-7B 71.1 89.5 52.6 82.9 91.5 48.0 55.1 82.6
Avg. 56.8 75.8 37.8 67.9 72.7 48.7 45.0 82.6
Open LLMs, Prompted as Critic Models
MetaMath-7B 47.3 89.8 4.7 81.6 97.3 2.7 1.3 97.2
MetaMath-13B 47.8 90.0 5.6 81.9 98.4 3.1 3.1 98.6
Qwen2.5-Math-72B 55.5 88.6 22.4 80.2 99.1 13.1 16.2 94.7
QwQ-Preview-32B 67.4 87.6 47.2 79.9 92.8 38.1 42.5 83.5
R1-Distill-Llama3.1-70B 61.4 89.8 33.0 82.3 97.1 22.1 26.8 90.9
R1-Distill-Qwen-7B 47.3 90.2 4.4 82.3 99.1 2.4 3.3 98.2
DeepSeek-R1† 68.2 84.3 52.0 76.3 78.0 69.0 77.1 -
Avg. 56.4 88.6 24.2 80.6 94.5 21.5 24.3 93.9
Proprietary LLMs, Prompted as Critic Models
GPT-4o 72.0 88.9 55.2 82.2 88.8 55.4 63.2 78.7
o1-mini† 74.5 88.9 60.0 82.7 85.6 69.8 75.0 -
Gemini-2.0-flash-exp 70.4 85.3 55.4 77.9 80.0 69.4 76.3 77.1
Gemini-2.0-thinking-exp-1219 72.9 89.4 56.4 83.0 89.8 55.5 64.8 79.5
Avg. 72.4 88.1 56.7 81.4 86.0 62.5 69.8 78.4
Table 18: A performance comparison of popular models across detailed metrics in ES. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 64.9 91.1 38.7 84.4 88.7 47.1 58.3 91.2
Skywork-PRM-7B 67.1 91.0 43.2 84.4 87.7 56.6 69.6 88.9
Llemma-PRM800k-7B 47.1 75.2 18.9 62.1 64.3 42.5 24.1 80.5
Llemma-MetaMath-7B 47.6 81.3 13.9 69.2 74.5 23.9 15.5 85.0
Llemma-oprm-7B 46.8 77.6 16.1 64.6 68.3 32.6 16.8 83.4
MATHMinos-Mistral-7B 52.1 80.7 23.5 69.2 72.6 42.4 39.4 85.5
MathShepherd-Mistral-7B 44.5 64.9 24.2 52.0 49.9 68.6 63.0 81.9
ReasonEval-7B 65.9 94.1 37.6 89.2 96.6 29.4 40.1 92.0
ReasonEval-34B 60.3 83.7 36.9 74.1 74.8 68.2 70.6 81.5
RLHFlow-PRM-Mistral-8B 55.1 90.0 20.2 82.2 90.0 20.1 22.9 92.3
RLHFlow-PRM-Deepseek-8B 55.0 92.4 17.7 86.1 95.2 13.4 16.6 95.1
Qwen2.5-Math-PRM-7B 67.3 93.4 41.2 88.1 94.4 37.6 43.8 87.9
Qwen2.5-Math-PRM-72B 71.1 92.9 49.2 87.5 91.6 54.5 61.6 83.1
Pure-PRM-7B 68.8 91.7 45.8 85.7 89.6 54.5 61.0 83.5
Avg. 58.1 85.7 30.5 77.1 81.3 42.2 43.1 86.6
Open LLMs, Prompted as Critic Models
MetaMath-7B 48.9 92.6 5.1 86.3 94.7 4.0 4.1 96.3
MetaMath-13B 47.4 93.6 1.2 88.1 97.8 0.7 0.9 99.6
Qwen2.5-Math-72B 71.6 93.5 49.6 88.5 95.1 43.8 49.5 86.9
QwQ-Preview-32B 72.3 91.3 53.3 85.3 88.8 62.4 65.0 79.7
R1-Distill-Llama3.1-70B 65.5 93.4 37.6 88.1 95.3 31.6 37.2 89.6
R1-Distill-Qwen-7B 54.1 94.0 14.1 88.8 98.9 8.2 9.2 97.0
DeepSeek-R1† 68.5 90.5 46.5 83.8 86.0 65.7 63.2 75.0
Avg. 61.2 92.7 29.6 87.0 93.8 30.9 32.7 89.2
Proprietary LLMs, Prompted as Critic Models
GPT-4o 69.7 89.9 49.6 83.1 84.3 74.3 76.9 76.3
o1-mini† 67.7 89.7 45.7 82.7 84.4 68.6 70.8 74.2
Gemini-2.0-flash-exp 65.7 86.0 45.4 77.7 77.0 83.1 85.6 77.3
Gemini-2.0-thinking-exp-1219 71.3 91.4 51.2 85.4 87.6 68.5 72.8 81.0
Avg. 68.6 89.3 48.0 82.3 83.3 73.6 76.5 77.2
Table 19: A performance comparison of popular models across detailed metrics in SC. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 63.3 89.6 37.0 82.2 87.9 42.1 49.9 90.5
Skywork-PRM-7B 67.7 90.2 45.2 83.3 87.3 55.4 65.4 88.3
Llemma-PRM800k-7B 46.7 71.6 21.9 58.3 59.9 47.1 20.5 74.3
Llemma-MetaMath-7B 44.4 76.0 12.9 62.3 68.0 22.5 12.9 79.1
Llemma-oprm-7B 44.5 73.4 15.6 59.6 63.8 30.2 13.6 79.4
MATHMinos-Mistral-7B 50.7 76.1 25.3 63.8 66.6 45.8 39.0 80.6
MathShepherd-Mistral-7B 41.3 59.2 23.3 46.7 44.6 60.5 52.2 79.8
ReasonEval-7B 61.5 92.1 31.0 85.8 95.4 23.8 30.5 91.2
ReasonEval-34B 57.8 80.3 35.3 69.7 71.0 61.9 57.9 77.5
RLHFlow-PRM-Mistral-8B 54.4 87.8 21.0 78.9 87.8 20.9 18.8 90.4
RLHFlow-PRM-Deepseek-8B 53.2 90.4 15.9 82.8 93.7 12.2 11.6 94.2
Qwen2.5-Math-PRM-7B 66.3 91.5 41.2 85.1 92.2 39.1 42.0 85.0
Qwen2.5-Math-PRM-72B 72.2 91.7 52.7 85.9 90.1 58.7 65.5 80.5
Pure-PRM-7B 64.0 89.8 38.3 82.5 89.0 40.7 42.5 84.2
Avg. 56.3 82.8 29.8 73.4 78.4 40.1 37.3 83.9
Open LLMs, Prompted as Critic Models
MetaMath-7B 48.4 90.9 5.8 83.5 96.5 3.6 3.8 96.7
MetaMath-13B 49.4 92.1 6.7 85.4 98.1 3.9 3.8 98.8
Qwen2.5-Math-72B 58.1 91.8 24.3 85.3 97.0 16.4 15.0 92.6
QwQ-Preview-32B 66.2 87.0 45.5 78.9 82.9 57.0 51.6 74.4
R1-Distill-Llama3.1-70B 65.8 91.9 39.7 85.7 93.4 35.4 35.5 86.1
R1-Distill-Qwen-7B 48.4 92.1 4.7 85.4 98.4 2.7 2.8 98.4
DeepSeek-R1† 73.5 91.3 55.6 85.5 85.7 84.1 77.1 75.4
Avg. 58.5 91.0 26.0 84.2 93.1 29.0 27.1 88.9
Proprietary LLMs, Prompted as Critic Models
GPT-4o 70.7 88.2 53.3 81.2 81.3 80.4 81.4 71.4
o1-mini† 73.8 92.2 55.5 86.7 87.7 77.8 74.5 80.0
Gemini-2.0-flash-exp 66.0 83.6 48.4 75.1 73.2 87.5 87.3 70.8
Gemini-2.0-thinking-exp-1219 71.0 88.8 53.2 81.9 82.6 77.2 79.2 73.7
Avg. 70.4 88.2 52.6 81.2 81.2 80.7 80.6 74.0
Table 20: A performance comparison of popular models across detailed metrics in DC. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for the o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 66.5 91.3 41.7 84.8 89.7 47.3 62.2 92.5
Skywork-PRM-7B 69.9 91.5 48.2 85.4 88.8 59.3 71.5 90.8
Llemma-PRM800k-7B 53.3 79.9 26.6 68.5 70.9 50.0 29.4 86.2
Llemma-MetaMath-7B 52.1 83.7 20.4 73.0 78.5 30.3 15.6 88.9
Llemma-oprm-7B 53.5 81.7 25.4 70.6 74.0 43.8 27.2 89.0
MATHMinos-Mistral-7B 57.8 82.5 33.1 72.3 74.5 56.3 53.8 87.3
MathShepherd-Mistral-7B 47.7 66.6 28.7 54.5 51.7 75.1 72.2 87.3
ReasonEval-7B 66.0 93.7 38.3 88.6 96.9 29.0 40.6 92.8
ReasonEval-34B 67.5 87.7 47.2 80.1 81.0 73.3 73.3 86.9
RLHFlow-PRM-Mistral-8B 63.8 91.0 36.7 84.2 90.7 37.5 45.0 90.9
RLHFlow-PRM-Deepseek-8B 66.2 93.0 39.5 87.5 94.9 33.5 43.3 92.1
Qwen2.5-Math-PRM-7B 78.5 94.9 62.0 91.0 95.2 60.5 70.4 85.4
Qwen2.5-Math-PRM-72B 78.6 94.7 62.5 90.7 94.3 64.1 73.7 84.8
Pure-PRM-7B 76.9 93.9 59.9 89.5 92.9 64.7 70.8 86.3
Avg. 64.2 87.6 40.7 80.0 83.9 51.8 53.5 88.7
Open LLMs, Prompted as Critic Models
MetaMath-7B 48.8 93.9 3.7 88.5 97.1 2.4 2.1 97.7
MetaMath-13B 48.1 94.7 1.5 89.9 99.1 0.8 0.6 99.0
Qwen2.5-Math-72B 59.1 93.1 25.1 87.4 99.6 14.7 19.8 96.2
QwQ-Preview-32B 66.9 91.7 42.2 85.4 94.8 34.2 43.1 89.1
R1-Distill-Llama3.1-70B 61.1 93.9 28.2 88.8 98.3 18.5 22.0 94.8
R1-Distill-Qwen-7B 48.0 94.6 1.3 89.8 99.8 0.7 0.7 99.5
DeepSeek-R1† 75.4 92.7 58.1 87.6 88.8 78.1 82.4 82.7
Avg. 58.2 93.5 22.9 88.2 96.8 21.4 24.4 94.2
Proprietary LLMs, Prompted as Critic Models
GPT-4o 71.1 92.1 50.2 86.3 90.4 56.5 60.4 85.2
o1-mini† 72.3 91.6 53.1 85.7 87.3 72.7 75.0 80.1
Gemini-2.0-flash-exp 67.3 88.3 46.4 80.8 82.4 68.8 76.0 84.9
Gemini-2.0-thinking-exp-1219 71.8 92.6 51.1 87.1 91.4 55.8 60.5 86.1
Avg. 70.7 91.1 50.2 85.0 87.9 63.5 68.0 84.1
Table 21: A performance comparison of popular models across detailed metrics in CI. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 33.1 36.2 30.1 33.3 23.2 78.4 73.9 70.2
Skywork-PRM-7B 36.8 42.9 30.8 37.4 28.8 76.0 72.8 70.2
Llemma-PRM800k-7B 51.0 71.7 30.3 59.7 62.3 47.9 24.4 75.0
Llemma-MetaMath-7B 50.5 78.7 22.4 66.5 75.5 26.5 15.6 82.8
Llemma-oprm-7B 49.2 74.4 24.1 61.7 68.0 33.4 16.1 81.2
MATHMinos-Mistral-7B 52.8 77.5 28.0 65.7 73.3 34.2 29.5 83.1
MathShepherd-Mistral-7B 47.2 63.6 30.9 52.3 51.8 54.5 45.0 81.4
ReasonEval-7B 55.6 88.4 22.9 79.8 95.4 15.4 17.3 92.8
ReasonEval-34B 57.7 80.5 35.0 70.0 76.8 41.6 35.2 82.6
RLHFlow-PRM-Mistral-8B 51.5 85.3 17.6 75.0 89.9 13.7 16.2 93.1
RLHFlow-PRM-Deepseek-8B 49.0 87.0 10.9 77.4 94.4 7.1 6.7 95.9
Qwen2.5-Math-PRM-7B 57.6 88.8 26.5 80.5 95.6 18.1 17.7 91.7
Qwen2.5-Math-PRM-72B 60.3 88.5 32.2 80.3 93.8 24.1 24.9 89.4
Pure-PRM-7B 60.3 87.4 33.1 78.8 91.4 27.0 24.0 87.3
Avg. 50.9 75.0 26.8 65.6 72.9 35.6 29.9 84.0
Open LLMs, Prompted as Critic Models
MetaMath-7B 46.5 88.0 5.0 78.7 96.3 3.0 2.3 96.2
MetaMath-13B 49.0 88.6 9.4 79.7 97.6 5.4 3.1 96.7
Qwen2.5-Math-72B 47.4 88.5 6.3 79.5 99.9 3.3 3.1 98.8
QwQ-Preview-32B 57.8 86.4 29.1 77.2 92.8 21.5 18.3 88.3
R1-Distill-Llama3.1-70B 48.8 88.9 8.7 80.3 98.7 4.8 4.8 97.2
R1-Distill-Qwen-7B 45.6 89.5 1.6 81.0 99.3 0.8 1.1 99.1
DeepSeek-R1† 63.1 82.7 43.5 73.5 82.6 43.7 34.8 -
Avg. 51.2 87.5 14.8 78.6 95.3 11.8 9.6 96.1
Proprietary LLMs, Prompted as Critic Models
GPT-4o 62.5 86.8 38.3 78.2 88.8 34.7 33.3 83.8
o1-mini† 61.8 84.9 38.7 75.7 88.6 33.1 18.8 -
Gemini-2.0-flash-exp 61.8 83.2 40.4 73.8 80.5 45.7 44.2 81.4
Gemini-2.0-thinking-exp-1219 60.3 87.1 33.5 78.4 90.5 28.0 27.0 87.0
Avg. 61.6 85.5 37.7 76.5 87.1 35.4 30.8 84.0
Table 22: A performance comparison of popular models across detailed metrics in PS. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Model Name PRMScore F1 Negative F1 Acc Positive Acc Negative Acc First similarity
Open-source Process Level Reward Models
Skywork-PRM-1.5B 32.3 36.3 28.2 32.5 22.8 84.0 79.1 74.5
Skywork-PRM-7B 37.4 44.9 29.9 38.3 29.8 83.3 77.8 73.3
Llemma-PRM800k-7B 53.5 77.8 29.1 66.2 70.4 43.8 23.1 85.1
Llemma-MetaMath-7B 51.3 80.6 22.1 68.9 76.6 27.9 15.1 85.6
Llemma-oprm-7B 51.3 78.4 24.1 66.4 72.5 33.8 17.2 86.6
MATHMinos-Mistral-7B 55.8 79.1 32.4 68.1 72.8 45.2 41.4 83.7
MathShepherd-Mistral-7B 48.6 65.6 31.7 54.2 52.5 62.8 57.1 84.0
ReasonEval-7B 58.0 90.6 25.4 83.2 96.7 16.9 24.6 93.8
ReasonEval-34B 64.3 84.4 44.3 75.6 79.4 57.3 56.8 83.6
RLHFlow-PRM-Mistral-8B 56.2 87.5 24.9 78.6 90.3 21.0 27.1 92.2
RLHFlow-PRM-Deepseek-8B 55.4 89.5 21.4 81.5 95.1 14.9 19.4 95.0
Qwen2.5-Math-PRM-7B 69.1 91.7 46.6 85.6 95.4 37.3 46.9 87.0
Qwen2.5-Math-PRM-72B 71.2 91.5 50.9 85.5 93.9 44.3 55.9 85.2
Pure-PRM-7B 69.2 90.2 48.3 83.6 91.3 45.3 51.7 84.5
Avg. 55.3 77.7 32.8 69.2 74.3 44.1 42.4 85.3
Open LLMs, Prompted as Critic Models
MetaMath-7B 48.3 90.4 6.2 82.5 95.3 4.1 3.5 96.6
MetaMath-13B 48.1 91.4 4.8 84.3 98.7 2.6 4.0 99.6
Qwen2.5-Math-72B 53.8 90.5 17.1 82.9 99.3 9.6 11.5 96.3
QwQ-Preview-32B 62.7 89.2 36.1 81.6 94.9 26.6 29.8 88.8
R1-Distill-Llama3.1-70B 54.1 91.0 17.2 83.7 99.0 9.9 12.0 96.1
R1-Distill-Qwen-7B 46.8 91.8 1.9 84.9 99.5 1.0 1.2 99.3
DeepSeek-R1† 69.2 89.0 49.5 81.9 87.3 54.1 59.5 -
Avg. 54.7 90.5 19.0 83.1 96.3 15.4 17.4 96.1
Proprietary LLMs, Prompted as Critic Models
GPT-4o 65.7 89.2 42.2 81.8 90.5 39.3 41.3 84.8
o1-mini† 64.8 86.7 42.9 78.4 84.5 48.2 43.8 -
Gemini-2.0-flash-exp 66.2 86.3 46.1 78.1 82.7 55.5 60.2 83.4
Gemini-2.0-thinking-exp-1219 65.7 89.7 41.8 82.5 91.8 37.0 40.2 86.4
Avg. 65.6 88.0 43.2 80.2 87.4 45.0 46.4 84.9
Table 23: A performance comparison of popular models across detailed metrics in DR. sub-category of PRMBench. The best performance for each metric is highlighted in bold, while the second-best performance is underlined. †: To reduce costs, we evaluated only a subset of 394 samples for o1-mini and Deepseek-R1.
Figure 7: An example data instance for NR. category.
Figure 8: An example data instance for NCL. category.
Figure 9: An example data instance for ES. category
Figure 10: An example data instance for SC. category.
Figure 11: An example data instance for DC. category.
Figure 12: An example data instance for CI. category.
Figure 13: An example data instance for PS. category.
Figure 14: An example data instance for DR. category (1).
Figure 15: An example data instance for DR. category (2).
Figure 16: An example data instance for DR. category (3).
Figure 17: An example data instance for MS. category (1).
Figure 18: An example data instance for MS. category (2).
Figure 19: Prompts used for test case construction during data generation (1).
Figure 20: Prompts used for test case construction during data generation (2).
Figure 21: Prompts used for test case construction during data generation (3).
Figure 22: Prompts used for test case construction during data generation (4).
Figure 23: Prompts used for evaluating generative LLMs (1).
Figure 24: Prompts used for evaluating generative LLMs (2).