DEV Community

Cover image for Your agent's success rate is hiding a reliability problem: measure pass^k, not just pass@1
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Your agent's success rate is hiding a reliability problem: measure pass^k, not just pass@1

Monday AI Insight. You run your agent on 50 test tasks, it solves 35, and the dashboard says 70% success. Ship it?

Here's the lesson: a single-run success rate tells you how often your agent can do a task, not how often it will. Real users hit the same task again and again. If your agent solves a task on Monday and fumbles the identical request on Tuesday, a one-shot eval never sees it. The metric that does is pass^k.

pass@k vs pass^k: opposite questions

Two metrics with nearly the same name answer opposite questions:

  • pass@k asks: if I try k times, does at least one attempt succeed? It was popularized for code generation by the Codex paper (Chen et al., 2021), where you can sample many programs and keep one that passes the unit tests. It rewards discovery.
  • pass^k ("pass hat k") asks: if I run the same task k times, do all k succeed? It was introduced in the τ-bench paper (Yao et al., 2024, Sierra) for agents that talk to users and call tools, like customer service, where every run matters. It rewards consistency.

As k grows, pass@k goes up and pass^k goes down. For a coding assistant that shows you several options, pass@k is reasonable. For an agent that issues refunds or rebooks flights, a failure on any one run is a real customer problem, so pass^k is the honest number.

What τ-bench found

τ-bench simulates conversations between an LLM-played user and an agent that has domain API tools plus a policy document (two domains: τ-retail and τ-airline). Success is checked by comparing the final database state with the annotated goal state, so a chatty, wrong answer doesn't pass.

From the paper:

  • gpt-4o with function calling reached about 61% pass^1 on τ-retail and about 35% on τ-airline.
  • With more trials, consistency fell fast: pass^8 dropped to about 25% on τ-retail for the same model.
  • The authors' failure analysis points at complex reasoning over databases, following ad-hoc policies, and handling compound requests (more than one thing in a single conversation).

So the same agent that looks like "61% success" solves only about a quarter of tasks reliably across 8 tries.

Why the gap is so big

Even a strong per-run success rate compounds badly. Simple arithmetic, assuming runs are independent: if a task succeeds 90% of the time, the chance that 8 runs in a row all succeed is 0.9^8 ≈ 43%. And at production volume, a task type that comes up thousands of times will hit the bad branch regularly.

Sources of run-to-run variation in agents:

  • Sampling randomness in the model
  • The user phrasing the same request differently
  • Different tool-call order or a timeout midway through
  • Long conversations where a policy detail gets dropped

How to compute it (it's cheap)

Run each task n times (n ≥ k), count successes c, and use the unbiased estimators from the τ-bench paper:

  • pass^k = average over tasks of C(c, k) / C(n, k)
  • pass@k = 1 − average over tasks of C(n − c, k) / C(n, k)
from math import comb

def pass_hat_k(results: dict[str, list[bool]], k: int) -> float:
    """results: task_id -> list of n trial outcomes (True = success). Needs n >= k."""
    scores = []
    for trials in results.values():
        n, c = len(trials), sum(trials)
        scores.append(comb(c, k) / comb(n, k))
    return sum(scores) / len(scores)

def pass_at_k(results: dict[str, list[bool]], k: int) -> float:
    scores = []
    for trials in results.values():
        n, c = len(trials), sum(trials)
        scores.append(1 - comb(n - c, k) / comb(n, k))
    return sum(scores) / len(scores)

# Example: 3 tasks x 4 trials
results = {
    "refund_simple":   [True, True, True, True],
    "exchange_two":    [True, False, True, True],
    "policy_reject":   [False, True, False, True],
}
for k in (1, 2, 4):
    print(k, round(pass_hat_k(results, k), 2), round(pass_at_k(results, k), 2))
Enter fullscreen mode Exit fullscreen mode

Note that pass^1 equals the plain average success rate, so you're not throwing away your old metric. You're adding the curve that shows how it decays.

How I'd use it in practice

  1. Pick a small, high-quality task set. τ-bench deliberately trades quantity for quality; a few dozen realistic tasks with clear goal states beat hundreds of vague ones.
  2. Check outcomes, not transcripts. Assert on the end state (the order row, the refund record, the ticket status), like τ-bench does with the database.
  3. Run every task several times (for example n = 8) and report pass^1 and pass^k side by side.
  4. Look at the flaky tasks first. Tasks that pass sometimes are where your prompt, tool descriptions, or policy handling are ambiguous. Tasks that always fail are a different problem (missing capability or tool).
  5. Vary the user. Paraphrase requests, split them across turns, and combine two requests in one message, since compound requests are a known failure point.
  6. Gate releases on pass^k for high-stakes flows, not only on pass^1.

The checklist

  • Report pass^1 and pass^k (k = 4 or 8) for agent evals
  • Use pass@k only when a human or a verifier picks the best of several attempts
  • Score by final state, not by how good the conversation sounds
  • Re-run each task n ≥ k times with varied user phrasing
  • Triage "sometimes passes" tasks before "never passes" tasks
  • Track the pass^k curve across prompt and model changes

A 70% success rate is a starting point. The question your users care about is whether the agent gets it right every time they ask.

Do you re-run your agent evals multiple times, or is each task scored once?

References

Top comments (2)

Collapse
 
mateo_ruiz_6992b1fce47843 profile image
Mateo Ruiz •

The distinction between pass@k and pass^k is important because discovering one successful execution is very different from delivering a dependable workflow every time. I'd add one more dimension: track failure consistency by category, not just the overall pass^k curve. A task that intermittently fails because of tool timeouts needs a different fix from one that fails when the agent interprets a policy or handles a compound request.

At IT Path Solutions, this is an important part of evaluating production agentic systems. I'd also keep the evaluation environment and task state controlled while varying user phrasing, tool responses, and execution conditions. That helps distinguish model variability from infrastructure-related failures.

Ultimately, a reliable agent isn't one that succeeds occasionally or even performs well on average. It's one whose failures are measurable, explainable, and increasingly predictable before it reaches real users.

Collapse
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

Splitting failures by category is a good addition. I'd also track pass^k per task and rerun each task many times, since a task that fails 1 in 4 looks fine on pass@1 but shows up clearly at k=8.