My Benchmark's First Leaderboard Measured My Grader, Not the Models

#devchallenge #machinelearning #ai #kagglechallenge

Kaggle Benchmarking Challenge Submission

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Scores and costs in this post are from my Kaggle benchmark page as of 11:10 AM PT on October 11, 2026.

Numeric grounding: can a model answer using only the numbers a short context states, and abstain when the number is missing? In a financial or operations summary, a made-up number is worse than "I can't tell". Every company, figure and unit in the benchmark is invented ("Vendo Bin A", "brens per lune"), so no model can answer from memory.

I treated two things as suspects, not just the models. The first was the grader: I replayed raw replies before believing any score. The second was the answer key: every key in the hardest task had to survive a blind vote by three models that are not being scored.

The benchmark has two layers. Each row is scored 0 or 1 on the final ANSWER line.

Sanity gate (five tasks, 6 to 9 rows each). Each one isolates a single classic grounding failure:

lookup: read a stated value
abstain: say CANNOT BE DETERMINED when the value is not stated
distractor: pick the right value out of several similar ones
multistep: combine stated values with arithmetic
units: convert using a made-up unit defined in the context

Stress test: ng-hard (10 rows). Each row stacks several traps in one paragraph:

a figure corrected twice, where only the last correction counts
a correction that applies only to a neighbouring entity or year
near-miss missing data ("fiscal 2024" when the question asks about calendar 2024)
two-step unit chains with three or more distractor numbers

Models Tested

Gemini 3.7 Flash, GLM-5, Qwen 3 235B A22B Instruct and gpt-oss-120b, all run on Kaggle. They come from four different model makers (Google, Z.ai, Alibaba and OpenAI), so a parsing quirk tied to one model family would show up as a gap between families. They also span a wide price range, which mattered more than I expected (Finding 4).

Findings

1. The first leaderboard measured my grader, not the models. The first run produced a clear spread: 0.98, 0.94, 0.78 and 0.62. Before publishing, I replayed the actual replies, and most of the "wrong" answers were right. The grader only accepted a line that started with ANSWER:, so it scored these correct replies as 0:

27 ANSWER: 27 (number and marker on one line): Qwen scored 0% on distractor because of this
**ANSWER: 72** (markdown bold): gpt-oss scored 16.7% on multistep
\text{ANSWER: } 135 (LaTeX)

One abstain row was also keyed wrongly. The context said "No side reactions occurred" and the question asked how much side product formed. Zero is a correct answer, but my key required "cannot be determined".

2. With a fair grader, every model passed the sanity gate. The fixed grader reads the last ANSWER marker anywhere in the reply. For a number, it takes the first number after that marker. For an abstention, it needs the full phrase CANNOT BE DETERMINED. A wrong number, a missing answer, or abstaining when a value is stated still scores 0. Before the grader went live, I checked it against a table of 21 reply formats modelled on the replies I saw, including 8 that must still score 0.

Parsing and that one bad key together had moved a model's score on the five core tasks by as much as 0.38 (gpt-oss: 0.62 to 1.00). All four models then scored 100% on all five gate tasks. So on short, clean contexts the gate is a pass/fail floor, not a ranking: a model you trust with numbers should clear it, and all four did.

3. The stress test is where models start to differ. Before running ng-hard, I had all 12 draft answer keys checked blind by three models that are not in the benchmark. I kept a row only if at least two of them agreed with its key, so 2 rows were dropped. On one dropped row, all three checkers answered 6.40 for "North Ridge" when only "North Ridge Annex" was stated. That near-miss fooled strong models, but a row I cannot defend is a coin flip with extra steps.

Gemini 3.7 Flash, GLM-5 and Qwen 3 235B scored 100% on ng-hard. gpt-oss-120b scored 80% (8 of 10 rows), and its two misses were different kinds of failure:

An arithmetic slip. The context gave a rate that was corrected twice, ending at 11.60 brens per lune, plus a run of 3.0 lunes and a dock amount of 1.20 brens. The key is 11.60 x 3.0 + 1.20 = 36.00. gpt-oss replied ANSWER: 4.20, which is exactly 3.0 + 1.20: the run length plus the dock amount, with the rate unused.
An incomplete abstention. The question asked for actual 2025 capacity, but the context only stated planned 2025 and actual 2024 figures. gpt-oss's entire visible reply was ANSWER: CANNOT BE. That was the right call, ending one word short of the required phrase. Kaggle's run log shows 164 output tokens for that reply. The full-phrase rule was fixed before this run, so the row scored 0, but it could fairly be argued either way.

So gpt-oss's overall 0.97, against 1.00 for the other three, rests on one clear error and one incomplete answer.

4. Same score, very different bill. Here is what Kaggle's Score vs. Total Cost chart shows for the full six-task run:

Qwen 3 235B: 100 for $0.006
Gemini 3.7 Flash: 100 for $0.07
GLM-5: 100 for $0.19
gpt-oss-120b: 96.67 for $0.005

Among the three perfect scorers, GLM-5 cost about 30 times as much as Qwen. Once accuracy ties, cost per flawless run is the useful tiebreaker.

What I would measure next: the same traps inside long documents and tables, and repeated runs per model to see whether gpt-oss's incomplete abstention happens again or was a one-off.

Indicative only: each task has 6 to 10 rows, so a single row is worth 10 to 17 points, and each model ran once.

What I take from it:

Replay raw replies before you trust a leaderboard. Most of the spread I first measured came from parsing, not from the models.
Validate answer keys independently. A blind vote by models you are not scoring caught keys I could not defend.
On short, clean contexts, single-trap tests are now a floor for these four models. Stacked traps are where differences start.
When accuracy ties, cost decides. Among the three perfect scorers it varied about 30-fold.