I Built a Benchmark That Grades AI Code Reviewers on Planted Defects

This is a submission for the Kaggle Benchmarking Challenge.*

Why I built it

I spent the last month running fourteen independent audit rounds on a clean-room
codebase. Outside reviewers — human and AI — kept catching defects that survived
my own builder, my own test suite, and my own verifier. The experience left me
with a question I couldn't shake: can an AI model do what those auditors did?

Not fix the bug. Find it. Name the lines. Say what's wrong and why.

Existing benchmarks don't isolate this. SWE-bench measures repair — a model can
stumble into a passing patch without ever identifying the defect. Devign does
binary "vulnerable / not vulnerable" classification — no localization, no
explanation, no severity. A model that says "vulnerable" for the wrong reason
scores the same as one that found the actual flaw. I haven't found a benchmark
that combines all three of: planted defects with controlled difficulty tiers,
mandatory line localization and explanation, and false-positive measurement on
paired clean variants.

So I built Red-Pen Bench. https://www.kaggle.com/benchmarks/robertgilbertusa/red-pen-bench

What it tests

44 planted defects drawn from the mutation-testing operator taxonomy. The
approach follows the selective-mutation tradition: Offutt, Lee, Rothermel,
Untch, and Zapf ("An Experimental Determination of Sufficient Mutant
Operators," ACM TOSEM 5(2):99–118, 1996) showed that five operators (ABS, UOI,
LCR, AOR, ROR) capture 99.5% of the mutation score of the full operator set.
That 99.5% figure belongs to the complete five-operator set — this corpus uses
four of the five (AOR, ROR, UOI, LCR; no ABS) plus statement-deletion defects
and higher-order mutants (two interacting faults) for the hard tier, which that
study doesn't cover. The operator family is the method; the exact coverage
number is theirs, not mine.

Every defect ships in four variants:

• Original — the defect in its natural setting
• Disguised — the same mechanism transplanted to a different domain, different names (reduces the chance that a model can rely on surface familiarity or template recognition)
• Fixed — the repaired code (false-positive check)
• Clean — correct code written from scratch, independent of the defective version (control)

Three difficulty tiers:

• Easy (18 items): single-operator defects — off-by-one loops, wrong arithmetic operators, inverted guards
• Medium (18 items): wrong logical connectors, missing edge-case checks, ceiling-vs-floor errors, the mutable-default-argument trap
• Hard (8 items): higher-order mutants — an auth check that's both inverted and miswired, a withdrawal that's both overdraft-permissive and audit-evasive

Every item passed a killability gate before entering the corpus: an automated
test that fails on the defect variants and passes on the fixed/clean ones.
And the corpus survived its own red pen — my first build had 44 defects that
an independent verifier proved were really 10 templates with renamed variables.
I threw it out, regenerated 44 genuinely distinct defects, and the verifier
signed off on the second build: 44/44 structurally distinct, 44/44 killability,
176-snippet integrity. The benchmark grades reviewers; it had to survive review
itself.

One defect, three of its four variants (easy tier, off-by-one):

# Original — the defect
def sum_prices(items):
    total = 0
    for i in range(len(items) - 1):  # skips the last element
        total += items[i]
    return total

# Disguised — same mechanism, different domain and names
def tally_receipts(entries):
    aggregate = 0
    n = len(entries)
    for j in range(n - 1):
        aggregate = aggregate + entries[j]
    return aggregate

# Fixed — the repair (false-positive check)
def sum_prices(items):
    total = 0
    for i in range(len(items)):
        total += items[i]
    return total

A model must output DEFECT: yes, LINES: 3, and an explanation naming the
off-by-one that skips the last element.

Scoring (no LLM judge)

Models must output three things: DEFECT: yes/no, LINES: ,
EXPLANATION: . Scored automatically, with no model
grading a model:

• Detection rate — fraction of the 2n defective variants (original + disguised) where the verdict matched ground truth
• Line-level F1 — mean over all 4n variants of line-level F1 against the planted fault lines (on fixed/clean variants: 1 if the model named no lines, 0 otherwise)
• Keyword recall — mean over the 2n defective variants of planted-keyword hits in the explanation
• False-positive rate — fraction of the 2n non-defective control variants (fixed + independently clean) flagged DEFECT: yes

Tier score = (detection + line-F1 + keyword-recall) / 3 − false-positive rate.

The denominators scale exactly to the variant counts. Easy and medium tiers
(n=18) average the four terms over 36, 72, 36, and 36 measurements
respectively; the hard tier (n=8) averages over 16, 32, 16, and 16. The
"Average" column is the unweighted mean of
the three tier scores, recomputed from the displayed tier values — each tier
weighted equally despite the different item counts. (Kaggle displays
two-decimal roundings of the same underlying values — 0.80, 0.64, 0.53, 0.12 —
consistent with half-even rounding, which turns 0.125 into 0.12.)

For calibration, two trivial baselines scored by the same code: always-no
(DEFECT: no on everything — 16.7% on every tier; the arithmetic is detection
0, defect-line F1 0, clean-line F1 1, keyword recall 0, false-positive rate 0,
so (0 + 0.5 + 0)/3 − 0 = 16.7%) and
always-yes (DEFECT: yes with no lines named and a keyword-free explanation
— −50.0% on every tier; a flag-everything model that names lines would score
differently). Note what the baselines expose about the metric itself: the
composite's zero point is arbitrary. A model that attempts detection and errs
can score below one that abstains — the meaningful threshold is the always-no
baseline (16.7%), not zero. Anything below 16.7% is losing to doing nothing at
all.

The models

Four models, four labs, all run live on Kaggle Benchmarks on October 10, 2026,
against the v1 benchmark (private at the time of the runs; public since
October 11, 2026):

• Gemini 3.7 Flash (Google)
• Claude Haiku 5.5 (Anthropic)
• GPT-5.4 mini (OpenAI)
• DeepSeek-R1 (DeepSeek)

The model names above are the exact identifiers Kaggle displays — confirmed
against the live leaderboard. All four ran the same three tasks (v1, locked)
against the same dataset version. The benchmark page records "last updated
October 10, 2026" and "evaluated on October 10, 2026"; its citation reads:
Robert Gilbert USA, Red-Pen Bench,
https://www.kaggle.com/benchmarks/robertgilbertusa/red-pen-bench/versions/1,
2026.

What I found

Executive summary: Gemini 3.7 Flash led overall, Claude Haiku 5.5 was the
most consistent across tiers, GPT-5.4 mini showed the largest observed
hard-versus-easy gap (76.0% on hard against ~41% on easy and medium), and
DeepSeek-R1 finished below the always-no baseline on average — all from a
single run per model.

The takeaway: a reviewer that flags everything scores below one that says
nothing. If you're grading AI code review, measure false positives first — or
your leaderboard is an enthusiasm contest.

A cautious reading of the table: on the hard tier, the detection and false-positive
rates move in 1/16 ≈ 6.3-point steps — but each counts only one-third in the
composite, so a single detection flip moves the tier score about 2 points,
and the line-F1 and keyword-recall terms are effectively continuous. Treat
small hard-tier gaps as noise until replicated. And with one run per model,
nothing below is a definitive ranking — it's one observation per cell.

Finding 1: two of four models scored higher on hard than on easy.
Gemini scored 80.6% on hard vs 70.4% on easy. GPT-5.4 mini scored 76.0% on hard
vs 41.6% on easy. I designed the tiers assuming difficulty increases
monotonically. It doesn't — not for these models. I call these difficulty
tiers by construction; the results show they are not difficulty-calibrated
across models. Two confounds I can't
separate from the leaderboard data: the false-positive term is computed on
tier-specific clean variants, and tier correlates with domain — measured in the
corpus, hard items cluster in business-logic domains (banking, security,
retail, analytics, ETL, scheduling) while easy items are mostly small
pedagogical domains (school, math, geometry). So "harder" and
"different-looking" are entangled here. My hypothesis, not a measured
mechanism: the easy defects look like correct code at a glance, while the hard
defects — two things wrong in one function — look conspicuous enough to trigger
closer reading. That's an interpretation to investigate, not a proven cause.

Finding 2: GPT-5.4 mini has a split personality. 41% on easy and medium,
76% on hard — a 34.7-point spread, the second-widest in the table. It reasons
through tangled multi-fault logic far better than it handles the easy tier's
single-operator defects. Whether that's a capability gap or a carefulness gap,
this benchmark can't say — it doesn't measure internal attention, and the
leaderboard doesn't give a per-operator breakdown. That pattern could be especially relevant to production review,
where subtle defects in ordinary code are easy to overlook.

Finding 3: Claude Haiku 5.5 has the narrowest spread. 71.3% / 60.7% /
61.4% — a 10.6-point spread, the smallest in the table, with no tier collapsing
and none spiking. That's a descriptive fact about this run, not evidence of
production reliability: the smallest observed spread across three tiers in a
single run is a preliminary signal, not an operational recommendation. And
note the tension in the table — Gemini's worst tier (70.4%) is about Haiku's
best (71.3%), and Gemini leads the average by 16 points. "Steadiest" and
"best" are different claims; this run supports only the first.

Finding 4: DeepSeek-R1 finished below the always-no baseline. 0.125
average, 9.7% on medium, −6.7% on hard — all below the 16.7% that doing nothing
achieves. A negative tier score means exactly one thing under this formula: the
false-positive rate exceeded the mean of detection, line-F1, and keyword-recall.
Two confounds before drawing conclusions:

• Output format. Without per-model parse-failure counts and raw outputs, I can't determine how much output-format noncompliance affected the result. The parser takes the first "DEFECT: yes/no" match in the response, so a reasoning trace that mentions "DEFECT: yes" before its final verdict could override the intended answer. Depending on the variant and the model's final answer, this could affect both detection and false-positive measurements — I can't establish the size or direction of that effect from the leaderboard alone.
• Model age. R1 is a January 2025 model measured against 2026 releases, so model age is conflated with model type.

The observed result is real; its causes are not yet established.

A personal postscript. None of the above is the whole story. The table says
DeepSeek-R1 is the worst reviewer I measured; my own experience says DeepSeek
is the sharpest auditor I've worked with — with one clarification that matters:
the audits below were done in the DeepSeek app, file by file, with DeepThink
and Search enabled (model version not recorded) — a different setup from the
benchmark's single-snippet Kaggle runs. On the MINI-CRAY build — a clean-room
vector-machine emulator that went through fourteen independent audit rounds —
DeepSeek did four consecutive rounds (9 through 12) and found defects that the
eight earlier rounds had missed: a chain scoreboard that was never
evicted, silently inflating chain-miss statistics; a 2-D array shape check that
passed validation and then died inside NumPy with an opaque broadcast error; a
hazard check that could forget a reader still in flight; and a cluster of numpy
type-aliasing holes across six functions that got unified into a single rule.
Those are real bugs in shipped releases, each with a regression test now. Two
honest caveats: those were whole-codebase audits, file by file — a different
task from the single-snippet review the benchmark scores — and this is a
hits-only recollection. The record isn't one-sided either: a CRAY-1
floating-point format module it drafted failed four of six behavioral checks
against the hardware manual. This
is anecdote, not measurement — one builder's experience, not a controlled
comparison. But it's why I don't read the 0.125 as the last word on DeepSeek.

What I'd measure next: repeat runs (this is one run per model — no variance
estimate yet), per-component and per-operator breakdowns, parse-failure counts
per model, cross-language disguised variants (Python→Rust), repo-level audits
with full codebase context, spec-dependent
defects (the ~35% of real bugs that need requirements knowledge), and a human
baseline — even 10 items reviewed by a competent engineer would calibrate what
"good" means.

Limitations

One run per model, so every number above is a point estimate with no measured
variance, and no definitive ranking is supportable. The hard tier is 8 items —
component rates move in ~6-point steps (about 2 points on the composite per
detection flip) and confidence intervals are wide. Keyword recall is a lexical
proxy for explanation quality, not a judgment of it. Line-F1 rewards naming no
lines on clean code, which is correct behavior but a different skill from
localizing a real fault. The "Average" column weights all three tiers equally.
Kaggle-side prompt templates, sampling parameters, and reasoning-mode settings
were set by the harness — I neither controlled nor recorded them. No human
baseline exists yet. The benchmark was private when these runs completed and
has been public since October 11, 2026.

Why this matters

Code review is where AI assistance is being deployed fastest and evaluated
least rigorously. If we can't measure whether a model finds defects — with
location and reason, not just a vibe — we're flying blind on the highest-stakes
use case.

Red-Pen Bench is designed to survive its own publication: the four-variant
structure means new defects can be planted quarterly from the operator taxonomy,
retired items published with solutions (turning contamination into training
data), and live items staying blind. Static benchmarks decay; this one has a
refresh protocol.

Try it

https://www.kaggle.com/benchmarks/robertgilbertusa/red-pen-bench — run your own model, or plant a defect and submit it.

Built for the Kaggle Benchmarking Challenge. 44 defects, 176 code variants,
zero LLM judges, one red pen.