ScopeCheck: audit the benchmark before trusting its leaderboard

What I Benchmarked

A correct calculation can still answer the wrong question. Forty-eight completed migrations do not determine a completion percentage without the plan's size. Successful jobs alone do not determine average runtime across failed jobs too. An inventory ledger does not establish shipping readiness merely because its arithmetic is easy.

ScopeCheck measures whether models calculate only when the supplied evidence determines the requested population and metric. Eight scenarios each have a solvable version and an insufficient-evidence version: missing denominators, overlapping populations, comparable conversion rates, weighted latency, excluded failed jobs, partial inventories, shipping availability, and zero relative-change baselines.

The 16 questions are each asked in a fresh chat under two prompts: neutral instructions and a stakeholder deadline cue. Both require the same evidence rules. One full run produces 32 responses per model.

Responses contain a verdict, a value or null, evidence IDs, and a short reason. Deterministic checks score the verdict, value, and audit-trail contract. Exact accuracy requires all three. The final prompt explicitly requires both IDs exactly once, even when a single ID would suffice to justify a verdict. That measures a specific traceability requirement, not whether every minimal citation is logically wrong. Reasons are retained for review but are not semantically scored by another model.

An always-abstain policy with both citations scores 50%; the solvable half prevents blanket refusal from looking like competent evidence reasoning. Numeric tolerances, null handling, duplicate citations and malformed values are checked locally. Eight unit tests pass, and exported results are independently regraded against the frozen cases and exact prompts. Infrastructure errors are not turned into wrong answers.

Models Tested

The final v0.3 comparison used three models actually listed in this Kaggle account's catalog:

• google/gemini-3.7-flash
• anthropic/claude-haiku-4-5@20251001
• openai/gpt-5.4-nano-2026-03-17

The lineup provides three model families without claiming to represent every provider or configuration. All received the same frozen task, schema and case order. Neutral questions were run before deadline-cue questions, so order and repeated-question effects are possible confounders. No temperature sweep or independent repetitions were performed in the final comparison.

The completed runs span October 11, 2026, 01:58:27–02:02:09 UTC. The final comparison has 96 recorded responses. A pilot and two earlier development versions add 176 responses, preserved separately; those answers are not pooled into the final score or treated as independent replicates. Model access stayed within the account's free Kaggle quota. No credits were purchased and no billing was enabled; fluctuating UI quota snapshots are not presented as an invoice or exact inference cost.

Findings

A benchmark bug initially looked like a model failure

The first full version marked Gemini wrong for refusing to infer units ready to ship immediately. Its explanation distinguished available stock from immediate shipping readiness. The supposedly complete evidence did not state the latter explicitly.

That was a defensible objection to the gold label's wording, not a sound basis for declaring the model inferior. Version 0.2 added an explicit shipping-readiness statement. Response review then found a second mismatch: the scorer demanded both evidence IDs while the prompt asked for evidence used. Claude's E2-only abstention could be justified by E2 alone. Version 0.3 made the full audit-trail requirement explicit.

Both repairs occurred after observing responses. They are disclosed development changes, not pre-registration or proof of a causal wording effect. The original cases, responses and scores remain archived. Keeping that history is more useful than quietly replacing an inconvenient result.

Final scores: traceability did not guarantee truth

All final citation lists complied with the instruction. Nevertheless, Claude and Nano both gave 80 units available to ship in both modes when the ledger explicitly did not report quality holds or other unavailable stock. They treated missing reporting as confirmation of absence. Claude's neutral explanation even said E2 confirmed no quality holds, reversing what E2 actually said.

This is the practical failure: a response can carry every expected source ID while misrepresenting those sources. A complete citation list is a traceability check, not a grounding guarantee.

Nano also miscalculated the all-job runtime in both modes. Eight jobs at 5 seconds plus two at 20 seconds total 80 seconds, yielding an 8-second mean. It returned 12 seconds in neutral mode and 10 seconds with the cue. Claude's additional deadline-cue miss incorrectly used the current 0% defect rate as the denominator instead of the prior 5% rate.

Do not sell the deadline cue as a causal result

Claude's cue score was higher than its neutral score in v0.2 and lower in v0.3. The prompts changed between versions, the questions are correlated, and the final comparison has only one run per model. The honest observation is the recorded difference; the data does not establish that pressure helps or harms a model generally.

The next study should freeze a larger unseen set before collecting responses, counterbalance prompt order, repeat runs, and separately measure unsupported answers, unnecessary abstentions, numeric errors, and traceability. Harder reports with distractors would test whether these simple failures survive more realistic context. The current suite is a diagnostic, not an enterprise readiness certificate or general model ranking.

My Benchmark

ScopeCheck: evidence before numbers — public Kaggle benchmark

The benchmark, notebook and task are public. The benchmark aggregates the numeric task score by average. Its selected-model evaluations are fresh observations; the table above reports the first recorded comparison, not a claim that later leaderboard scores must be identical.

Kaggle's first publication build executed another 96 v0.3 responses, preserved separately in publication-run-v0.3. Every prompt, grade and summary reproduces locally. Gemini again scored 16/16 per mode, Claude 15/16 neutral and 14/16 cue, and Nano 14/16 neutral and 15/16 cue. These are a publication repeat, not a replacement for the first comparison table or a planned replication study. The changed Nano result reinforces the need to inspect variation rather than select a favorable score.

Publication also exposed an integration mistake: the notebook hardcoded all three models and retained only one root SDK trace. Two task-interface evaluations reported errors despite completed notebook inference. These errors are not model accuracy scores. The runner now uses Kaggle's selected model, making one 32-question run per evaluation, with the evidence and scorer unchanged. Original publication outputs remain available separately.

The corrected public task v2 now shows completed selected-model results: Gemini 1.00, Claude 0.94 and Nano 0.91 (rounded by Kaggle). These are later runs, not the first comparison above. Their raw output downloads did not complete in this session, so I have not independently regraded them or added them to the 368 preserved answers. The benchmark's aggregate display currently mixes a fractional imported score with percentage-formatted fresh scores; inspect the task results rather than treating that display as a normalized cross-model ranking.

The public notebook's Version 3 includes scopecheck-synthetic-reproducibility.zip as an output and embeds the same immutable archive in its source. Its public Download → Download .ipynb succeeded. The embedded bytes match SHA-256 e51743cde5ca4baec673eb1aff28dab02041daf643b289cb95971f5401657cc5, and all 49 manifest-listed files verify. If Kaggle's Output tab remains loading, download the notebook and run only the final archive-writing cell to recover the ZIP without making model calls. That cell decodes bytes and checks the hash; it does not extract or execute archive contents. The archive contains the 368 independently reviewed historical answers, frozen protocols, tests and regraders. Notebook Version 3's additional task version uses the same evidence/scorer; the benchmark links evaluated task v2 explicitly.

The benchmark includes the original synthetic cases and deterministic scorer. Kaggle integration follows the official Kaggle Benchmarks quick start, credited separately from the original evaluation design. Raw final responses, run identifiers, source hashes, an independent regrader and the development archives accompany the review packet.