DEV Community

Grain64
Grain64

Posted on

What a 12-row model test can and cannot tell you (with the interval code)

A twelve-example test is enough to catch a cheaper model that breaks on your data. It is not enough to prove it is as good as the one you run. The gap is easy to put a number on.

1. Put an interval on the pass rate. A model that gets 12 of 12 right has a 95% Wilson interval of 75.7% to 100%. 11 of 12 gives 64.6% to 98.5%. 10 of 12 gives 55.2% to 95.3%. For comparison, 290 right out of 300 gives 94.0% to 98.2%. Hold the lower edge, not the point estimate, against your bar.

function wilson(k, n, z = 1.96) {
  const p = k / n, d = 1 + z * z / n;
  const c = p + z * z / (2 * n);
  const m = z * Math.sqrt(p * (1 - p) / n + z * z / (4 * n * n));
  return [(c - m) / d, (c + m) / d];
}
Enter fullscreen mode Exit fullscreen mode

2. Use a small sample to rule models out, not in. If a candidate misses rows your current model passes, read those rows first. They show what the candidate does with your inputs, and that is worth the minute. A clean 12 of 12 only says it is worth a bigger run.

3. Pick the rows that hurt. A random twelve mostly measures the easy cases. Take rows from your logs where the answer was ambiguous, the label is rare, or a model has failed before.

4. Count every attempt in the price. Cost per successful task is the money spent on all attempts, retries and failures included, divided by the tasks that passed. A model that sometimes answers with nothing gets retried, and the retries are on your invoice. On one public run of 300 function-calling tasks, Mistral Small (2603) returned an empty reply 58 times even after retries and passed 70.7%.

5. Repeat on 100 or more rows before you switch. The small test chooses which models earn the bigger run. The bigger run, with the interval from check 1, makes the decision.

If you want checks 2 to 4 without writing the loop, the verdict pages on Clearbar have a box for it: paste 5 to 20 input => expected pairs, pick up to three candidates next to your current model, and the run happens in your browser with your own OpenRouter key. The key goes to OpenRouter only and is not stored, and a run is capped at $0.50. You get pass rate and cost per successful task for each model: https://clearbar.grain64.com/verdicts/sms-spam/?utm_source=devto&utm_campaign=devto-articles&utm_content=article

What is the smallest sample you have trusted before switching a model, and what did it miss?

From Clearbar, by Grain64.

Top comments (0)