A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.
That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.
The number that surprises people
Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.
- One attempt ≈ 51k tokens.
- With
retries: 3and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.
That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.
retry_policy:
naive:
attempts: 3
on_failure: "replay full context to same model"
hidden_cost_multiplier: 4 # 1 attempt + 3 full-context replays
shows_up_in_dashboard: false
cost_aware:
attempts: 3
on_failure: "route to next equivalent model"
carries_context: true
hidden_cost_multiplier: 1 # context travels, never replayed
shows_up_in_dashboard: true
Why the bill stays invisible
Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:
- Latency triples → someone notices.
- Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.
So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.
The fix is routing, not retrying
You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."
With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.
flowchart LR
subgraph Naive[Naive retry: context replayed]
N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
N2 -->|fail| N3[Replay 50k in]
N3 -->|fail| N4[Replay 50k in]
N4 -->|fail| N5[Replay 50k in]
N1 -.paid 4x.-.-> N5
end
subgraph Smart[Cost-aware: context travels]
S1[Task · 50k ctx] --> S2[Model A call]
S2 -->|unavailable| S3[Model B · same ctx]
S3 -->|unavailable| S4[Model C · same ctx]
S1 -.ctx paid once.-.-> S4
enddef step(task, model_registry):
# cost-aware: context is paid ONCE, then travels with the task
for model in model_registry.ordered_for(task.capability):
try:
return model.call(task.handoff_context) # no replay
except Unavailable:
continue # next equivalent, same context
raise NoCapableModel
# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
for _ in range(attempts):
try:
return model.call(task.full_context) # full context RE-SENT each time
except (Timeout, Unavailable):
continue
raise RetryExhausted
The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.
The question I'll leave you with
Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?
No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.
Top comments (62)
The 4× tax number is the one I'll quote. One addition from the ops side: the retry path also needs idempotency on tool-call results — replaying the full context against the same model isn't just expensive, it can double-apply a tool call if the first attempt actually succeeded server-side and only the response was lost. So it's the router you describe plus an idempotency check; either one alone still leaves a nasty failure mode.
Great addition — the silent-success case is exactly the one people miss when they treat "retry" as just re-running the loop. If the tool call already executed server-side but only the response was dropped, a naive context replay re-derives the same decision and fires it again. Double-apply.
One refinement I'd push: idempotency keys only help when the tool can take one. The cleaner split is to treat the tool layer as its own retry boundary, separate from the LLM context replay. You don't actually need to replay the whole context to retry a failed tool call — re-issue the call with the same idempotency key, and the context-replay loop (the expensive 4× part) only triggers when the model's decision failed, not the tool transport. Two different retry loops, two different cost profiles.
Which raises the messy case: tools with no native idempotency (a bare POST that won't accept a key). Do you wrap them behind an idempotency proxy, or just accept the double-apply risk on the low-stakes ones and only guard the destructive ones?
"Two different retry loops, two different cost profiles" is the cleanest way to frame this, and it's the part most agent frameworks get wrong.
They treat "tool call failed" and "model call failed" as the same thing, so they replay the whole context for both. But a tool call doesn't need the model's context — it just needs the same idempotency key.
Curious how you handle the "tool call succeeded but the result was lost" case. If the model never sees the result, it might decide to call the tool again with different arguments. Does the retry loop re-inject the original result, or does it let the model re-decide?
Good question — and it's the exact failure mode that breaks naive retries.
The fix is ownership: the retry controller, not the model, owns the tool result. On a silent-success (timeout, but the call executed server-side), you don't let the model re-decide. You key the tool call by signature — tool name + args hash — and cache the result; on retry you short-circuit and hand the model the cached result as if it came back clean. The model never gets a second pass at picking different arguments, because it's not in the loop for transport failures.
The trap is when retries live inside the model's own loop — then you can't stop the re-decision. So it's less "retry the tool" and more "the orchestrator replays the cached outcome." That only holds if the tool is idempotent-confirmed; otherwise you're replaying a side effect.
Which exposes the messier case: partial success. If the tool wrote something but you're unsure how much — a batch insert that committed half — the cached-result shortcut is wrong. You need either true idempotency or a fresh failure. How do you signal "ambiguous success" back to the orchestrator without forcing a full re-run?
"Ambiguous success" is the exact state that most retry systems don't have a word for.
You're right that the cached-result shortcut is wrong here, because replaying a partial write is worse than not replaying at all. The only honest answer is to treat "ambiguous success" as a first-class state, not as a failure or a success.
The orchestrator needs a way to say: "I don't know if this fully committed. Here's what I do know: the call was sent, the connection dropped, the tool reports partial completion." The model can then decide whether to retry, compensate, or escalate — but it decides with the actual state, not with a lie.
Treating "ambiguous success" as a first-class state is the part most retry code skips — and you're right that the cached-result shortcut is actively dangerous here, because a partial write replayed is worse than no replay. The line I'd draw: the retry controller owns the tool result, so "ambiguous success" is exactly the one state where it must refuse to auto-retry and instead surface the observed facts (sent / dropped / partial) the way you describe. That keeps the model deciding with real state instead of a guessed one.
The open question for me is where "ambiguous success" lives in the state machine — its own explicit node, or folded into a terminal "needs-human / escalate" bucket? The former gives finer recovery (compensate vs retry), the latter ships simpler. Which way did you land?
I landed on the explicit node (the former).
Folding it straight into a terminal "needs-human" bucket is tempting for simplicity, but it throws away the opportunity for automated compensation. If a tool call returns "ambiguous success" (e.g., a timeout on a write operation), the system shouldn't immediately wake up a human.
Instead, the explicit node lets the agent run a read-only "state check" or "idempotency check" tool to figure out if the write actually succeeded. Only if that check also fails do we route to the terminal escalation bucket. It's a bit more work on the state machine, but it saves a massive amount of human interruption!
This is the right call — the "needs-human" default is what quietly burns the most expensive resource on problems a machine could settle. The part I like is that the state check is itself a tool the agent calls, so the human stays out of the loop until there's a real judgment to make. The risk I'd watch is the check becoming its own retry loop: if the read is flaky, "ambiguous success" can bounce between the check and escalation forever. How do you bound the number of state-check attempts before it's allowed to wake a human?
The risk of the check becoming a retry loop is real. In my experience, the best way to bound it is a strict state machine with a retry counter.
If the agent hits the 'ambiguous' state, it increments a counter. Once that counter hits a hard limit (e.g., 3), the agent is physically forbidden from calling the check tool again and must yield to the human. If the read is flaky, no amount of agent retries will fix the underlying system — escalation is the only sane path.
The hard-limit-and-yield-to-human part is the piece most teams skip, and it's the part that actually stops the loop. One thing we learned the hard way: the counter has to live somewhere the agent process can't reset. We kept it in agent local state at first, and a restart silently zeroed it — suddenly the bound was gone. Once we moved it out, the 'ambiguous' state actually meant something.
Curious whether you persist the counter in the ledger too, or keep it in the agent's own state?
This is the difference between 'retry with backoff' and 'retry with a leash.' The leash only works if the agent can't slip it off by restarting.
The observation that dashboards show cost per successful request is the uncomfortable one. Failed attempts disappear into the average, so the feature looks cheap right up until the invoice arrives.
SkepAgent's point deserves as much attention as the token math. A retry after a tool call that timed out but actually succeeded does not only cost tokens. It can apply the same change twice. So the order matters: make side effects idempotent first, then decide how many retries you can afford.
challan116's suggestion to budget retries as their own line item matches what we do on client builds. When a feature is scoped, it gets an expected cost per run and a separate allowance for failures, written down next to the acceptance criteria. If the failure line starts growing, that is a signal about the prompt or the model choice, and it shows up within days.
When you route to the next model and carry context forward, how do you stop the second model from repeating a side effect the first one already triggered?
One distinction I think matters here is between a failed response and a failed operation. A timeout doesn't prove the side effect didn't happen.
For state-changing operations, I'd separate retrying the transport from retrying the operation itself. If the outcome is ambiguous, preserve the original operation reference and reconcile its status before deciding whether another execution is safe. Idempotency helps, but it doesn't automatically resolve partial completion or providers that don't support idempotency keys.
Good distinction, and it's the one our R4 sketch glossed: retrying the transport (the call didn't arrive) is safe to repeat; retrying the operation (did the side effect land?) needs the reconcile-first step you describe. The trap is treating a timeout as a transport failure when it might be an operation that already committed — re-firing then double-applies. The piece I'd add: the ledger you mention has to be readable by any model in the chain, not just the one that started it, or the handoff to model B reintroduces the same ambiguity you just closed. Question: when the status is still ambiguous after reconcile, do you fail the task or escalate to a human/queue?
The ordering point is the crux — idempotency has to be a precondition, not a retry setting. You can't retry your way out of a non-idempotent side effect; the retry just multiplies the damage.
On the cross-model case: the same "retry controller owns the tool result" boundary from earlier helps here too. If model B's attempt is a transport failure (not a decision failure), the cached result from model A's already-successful call short-circuits — so B never re-fires the side effect, it just receives the result as if it came back clean. The nasty case is partial: A triggered the side effect but the result was lost, and B is a different model that can't see A's cache. There you need a side-effect ledger keyed by task id, not by model — so any model in the chain can check "did this task's effect already land?" before acting.
Question: do you run that side-effect ledger as a separate service, or bake it into the orchestrator? Separate is cleaner to reason about but adds a hop on every call.
I'd lean toward keeping the ledger logically separate from the orchestrator, but not necessarily deploying it as a separate service from day one.
For state-changing financial operations, I'd initially persist the operation record, provider reference, and execution status alongside the transaction state in a durable database. The orchestrator can consult that record before deciding whether an operation is safe to execute again.
The important distinction for me is between a transport failure and an ambiguous operation outcome. If a provider may have completed the side effect but the response was lost, I'd reconcile the original operation before considering another execution.
I'm exploring this in Agentic Switcher, where the focus is deterministic orchestration, ownership fencing, and safe recovery across provider integrations. A separate ledger service could make sense later, but I'd first establish the consistency and recovery semantics.
One question I'd be interested in exploring is how you'd handle a side effect that succeeded externally but whose ledger update failed. That seems to be the hardest boundary to get right.
This matches what I see — logical separation, physical colocation early. The trap is making the ledger a synchronous dependency of the retry path; then a ledger hiccup stalls exactly the operation it's supposed to protect. Keeping it as a side-write (operation record + provider ref + status) that the retry reads but never blocks on is the version that actually ships. The real job of the ledger isn't audit, it's decision — "have I already committed this signature?" has to be answerable in microseconds or the retry just fires anyway. How are you gating the retry on the ledger read — optimistic fire-then-check, or a hard gate?
I'd favour a durable gate before the external side effect, not optimistic fire-then-check.The important distinction is that the gate shouldn't become a remote synchronous dependency. I'd keep the ledger logically separate but co-located initially, with an atomic state transition or idempotency constraint protecting the operation before the provider call.If the provider outcome is uncertain, the orchestrator should reconcile using the provider reference or idempotency key rather than blindly retrying. A ledger outage should fail closed for potentially duplicate financial operations, while allowing safe reads and reconciliation to continue where possible.
The goal is to make retry safety a property of the state machine and its invariants, not just a fast ledger lookup.
How do you handle the ambiguous-outcome case when the provider has accepted a request but the orchestrator times out before recording the result?
The fail-closed-on-ledger-outage distinction is the right one — safe reads keep flowing, duplicate financial writes stop cold. On the ambiguous case you raised: we treat "provider accepted, orchestrator timed out before recording" as a third state, not success and not failure, and we reconcile by idempotency key before any reissue. That third state is where most "duplicate charge" bugs actually live. Do you keep that pending state inside the orchestrator, or do you push it down to the ledger as a row the reconciler owns?
I'd push the durable pending state into the ledger and make the reconciler responsible for resolving it. The orchestrator owns execution; the ledger owns the authoritative operation state; the reconciler owns closing the gap between intent and confirmed outcome.Before the provider call, we persist the operation identity and establish an atomic execution claim. If the call times out, we transition to an outcome-unknown state rather than treating it as either success or failure. The reconciler then checks the provider using the idempotency key or provider reference before any reissue.The orchestrator may hold transient execution context, but recovery must not depend on that process surviving. One caveat: where the provider supports neither idempotency nor reliable status lookup, we cannot guarantee safe automatic retries; that case needs an explicit uncertainty and escalation path.For me, that separation is what makes reconciliation a recovery mechanism rather than just another retry loop.
The outcome-unknown transition is the key move and most teams miss it — they jump straight to success/failure and then can't tell a timeout from a dropped call. Persisting the operation identity before the provider call is what makes the reconciler safe to run at all.
The one place it got hairy for us was idempotency-key reuse across a flaky gateway — same key, different semantically-meaningful attempt. For providers with no status lookup, do you escalate every time, or keep a manual 'confirmed-lost' bucket someone drains on a weekly pass?
Routing-the-task-not-the-model is the right instinct, and the token math is brutal once you see it. One thing I'd put before the retry-budget discussion: idempotency.
A timeout doesn't tell you the call failed — it tells you that you didn't hear back. If the side effect already ran (the payment went, the email sent, the row was written), "carry context forward and advance to the next model" can apply it twice. The cheapest retry policy in the world still double-charges someone if the thing it retried had a side effect.
So I'd gate it: reads retry freely; writes retry only behind an idempotency key, or re-grade the outcome (did the action actually happen?) before re-running. Otherwise the retry tax you measure in tokens comes back later as a reconciliation bill — same lesson as your dashboard point: the real cost is on the path you're not looking at.
Glad this one traveled over from your local-LLM piece — and yes, idempotency first is the right ordering. The retry-budget discussion is moot if a retry can double-apply a side effect; you've capped the cost of the call but doubled the damage. The split I landed on: the retry controller owns the tool result keyed by signature, so a silent-success (heard nothing, but it ran) returns the cached outcome instead of re-firing. That's what makes a cost cap safe to actually enforce. You put it better than I did: a timeout is a communication failure, not an operation failure. How are you surfacing that distinction to the caller — same error type, or a distinct "unknown" status?
"You don't make retries cheaper by retrying harder" is the core insight here, and it's the opposite of what most teams do.
The default response to "the model is down" is "retry with the same model, hope it comes back". But if the model is down, retrying it doesn't help — routing to a different model does.
Curious how you handle the "no equivalent model available" case. If all the models in the registry are down, does the task fail gracefully, or does it queue and wait?
"Retrying harder" is just re-paying the 4× tax with more determination — the insight holds, and it's the opposite of instinct.
On the no-equivalent case, it depends entirely on the task's latency contract, and the trap is treating both the same:
The mistake is a global "retry N times then give up" that applies the synchronous logic to async work (wasted waits) or the async logic to sync work (timeout blowups).
Question: do you split failure modes by task class (sync vs async, read vs write), or by a single global timeout? Class-based is more code but each path actually fits its caller.
"Class-based is more code but each path actually fits its caller" is the right tradeoff, and it's the one most teams avoid because it's harder to explain in a design doc.
The split isn't just sync vs async. It's also read vs write, and inside writes, idempotent vs non-idempotent. A sync read can fail fast and serve a cached result. A sync write needs to know whether it committed. An async read can queue. An async write can queue, but only if it's idempotent — otherwise you're queuing a duplicate.
Four classes, four different retry policies. It's more code, but each one is simpler than a global policy trying to handle all four.
The read-vs-write and idempotent-vs-not split is the part most "one global retry policy" designs quietly collapse. Four small policies that each fit their caller beat one clever policy that tries to branch on all four — and the branching logic is usually where the bugs hide. The async-write-only-if-idempotent rule is the one people skip and then discover via duplicate side effects. When your async-write path hits a flaky queue, does it ever downgrade to sync, or does the queue failure just become a different retry class?
Definitely a different retry class. Downgrading to sync is a trap — it breaks the whole point of having an async boundary.
If the queue is flaky, the async-write path should have its own Circuit Breaker. After X failed publishes, the circuit trips, the async path fails fast (returning a 202/503 to the user), and the system just retries the queue publish with exponential backoff. Keeping the failure isolated to the queue is much safer than letting a sync fallback take down your primary database.
Exactly — downgrading to sync is the trap because it quietly moves the queue's failure into your primary path. The 'keep the failure isolated to the queue' instinct is the right one. We made the opposite call once and let the sync fallback touch the database; one flaky publish turned into a cascade we felt everywhere.
Did you put a separate budget on the circuit-breaker's own retries, or do they still count against the same per-task cost you're tracking?
The trap of downgrading to sync is that it feels like a resilience measure, but it's actually a blast-radius amplifier. You're not 'handling' the failure; you're just relocating it to a place where it can do more damage.
One correction and one number.
The correction: "context paid once" doesn't hold, and stefan's question above is the right one. Model B has no memory of model A. It receives the context and pays for it. What routing changes is the rate on that send, cache read or full ingestion, never the number of sends. Every call re-sends the transcript, so the unit that matters is turns, not attempts.
The number: on 4,451 token-bearing turns across 53 sessions, the median turn pushed 135,267 tokens. Across the corpus 92.9% of the tokens were cache reads, 6.7% fresh input, and the model generated 0.41% of what it read. Priced at 0.1 for a cache read against 1 for fresh input, losing the cache on a turn multiplies that turn's cost by a median of 8.1x (IQR 6.5 to 9.0).
The part that changes the advice is where that cache is lost. Appending the error text to the end of the prompt is cheap, because the prefix still matches and only the new suffix gets billed fresh. What's expensive is changing anything before the end. Editing the system prompt, re-summarizing history, or moving to a model with its own cache namespace reprices everything from the first changed token. We measured the same shape in context compaction: after a rebuild, 69.4% of first turns were more than half fresh input.
So a same-model retry against a warm cache costs roughly what the failed attempt cost, not four times it. The tax arrives when the fix rewrites the front of the prompt.
The instrument I'd add to the billing join above is cache reads divided by total input, per turn. A retry that appends barely moves it. A rewrite drops it, and the drop is the bill.
The correction lands harder than mine did — and the number reframes the whole thing. "Turns, not attempts" is the right unit; the retry that re-sends the transcript isn't free even at a better rate, because the cost that moves is the cache, not the count. Your 92.9% cache-read corpus is the part I under-weighted: a slim handoff helps for exactly the reason your data shows — it keeps the prefix stable, and a stable prefix is what keeps the cache warm. Re-attaching the full original on retry is what blows it, and at your median that's an 8.1× turn. So "slim handoff" and "where the cache is lost" are the same lever seen from two sides; I'd just add that the retry budget should be counted in cache-lost turns, not attempt counts, because the bill isn't the retry — it's the re-warm. The part I'm still unsure on: when you hand off to model B that needs different framing, how do you keep the prefix from breaking — carry a frozen prefix, or accept the cache loss as the price of the switch?
You're right that the bill is the re-warm, and to your question I'd answer the boring way: we accept the loss.
A frozen prefix doesn't survive a switch. The receiving model has never seen that text, so whatever the sender cached can't be spent on it, and the first turn pays full input on however much you carry. What stays warm is the second turn onward, and only if nothing before the end of the handoff changes.
The number that made me stop treating a rebuild as free came out of compaction, which is the same event from a different direction. Across 62 rebuilds in two long sessions, 69.4% had a first turn that was more than half fresh input. Those rebuilds did pay themselves back in about 14 turns, but only because the rebuild also halves what gets re-sent afterwards. A model switch has no shrink, so it's the cost without the refund. It has to earn back on rate alone.
So the rule I'd give you: switch late, and switch on a turn where the prefix was going to break anyway. If the retry was already re-summarizing the history, the switch is close to free because you were paying for the rebuild regardless.
The "switch on a turn where the prefix was going to break anyway" rule is the cleanest framing I've seen — the rebuild cost is already in the budget, so the model switch just rides it instead of adding a new one. And the 69.4% number is the gut-check that keeps "rebuild is free" from becoming a habit. One thing I'd push on: across those 62 rebuilds, did any carry a partial prefix — only the stable middle — into the new model, or is it always all-or-nothing re-warm? A partial carry could save part of the hit without the full refund — have you ever measured whether the stable middle of the prefix survives the switch and actually reduces the re-warm?
Good push, and the honest answer splits in two.
The switch case I can't measure. My database holds exactly one model-switched message in its entire history, so the scenario you are asking about, where a rebuild turn is also the moment the model changes, has n=1 here. Anything I said about it would be a story about one turn.
The carry case I can measure, and it isn't all-or-nothing. Across 62 compactions with a readable turn on each side, I compared the cache-read tokens on the first turn after the rebuild against that turn's total context. Median carry is 33.5%, range 0 to 96.8%. 21 of 58 events came back over 50% cached, 12 came back under 5%, and 3 came back at exactly zero. So a partial prefix does survive a rebuild, and the share is not stable at all.
What varies most is the session, not the event. Two sessions with 26 and 10 rebuilds each have median carries of 47.7% and 67.1%. A third with 14 rebuilds sits at 11.1%. One average over all of them would've hidden exactly the thing you are asking about.
Here is the limit, though. A cache-read count tells you how many tokens the provider served from cache. It doesn't tell you which part of the prompt they came from. So "the stable middle survives" is not something my record can show. I can show how much survived on each rebuild. I can't show that it was the middle, and I would be guessing if I said the summary stayed cold and the tail stayed warm. To answer that you need a provider that reports cache hits per block, or a controlled test where you change one region at a time and watch the number.
the 4x holds for a timeout, but where the call dies changes the bill a lot. a provider that errors before the first token usually costs you nothing, one that dies mid stream already charged you for the full prefill. we log failure stage next to attempt count, otherwise the dashboard averages the two together.
one gap in the failover version: switching models drops the prompt cache, so the next model pays full price on that 50k context anyway. we retry the same provider first while the cache is warm and only fail over on hard errors.
did you compare cache hit rate on retries versus failovers?
Logging failure stage next to attempt count is the fix for the averaged-dashboard problem — a pre-token error is free and a mid-stream death is full prefill, and a mean hides that gap completely. Retry-the-same-provider-while-warm before failing over is exactly the cache-aware order most people get backwards. When you do fall over, do you re-order the prefix so the new model sees the cacheable part first, or do you just eat the full re-warm on that 50k context? That choice is most of the remaining bill on a 50k context — do you re-order the prefix on failover, or just accept the full re-warm?
we reorder on failover. the cacheable prefix (system prompt plus tool defs) goes first and the volatile turns last, so the new provider at least writes a cache entry that pays off on the next retry. the first call still eats the full cold read though, so we only fail over once the retry budget on the warm provider is gone.
are cache write surcharges eating the savings for you on shorter contexts?
One thing I'd push on in the cost_aware block: hidden_cost_multiplier 1 only holds if "context travels" means the new model doesn't have to read it again. It does. Routing to the next model still bills the full 50k input tokens on that model, so you've swapped a replay on model A for a first read on model B, and you've thrown away any prompt cache you'd built on A. With caching, a retry against the same provider can actually be the cheaper path, since the replayed prefix is billed at the cache-read rate rather than full price, as long as the retry lands inside the cache window. So the real fix probably depends on why it failed. A timeout or 5xx from an overloaded endpoint is where routing helps, while a malformed output is better retried in place with the cache warm. Did your team separate those two failure types in the budget, or was it one retry counter?
This is the nuance I wish I'd put in the article. We defaulted to 'route on any failure' and only later separated the cache-warm in-place retry from the route-away retry — and they have completely different cost shapes, like you said. Routing a malformed-output case away just throws away the prompt cache you'd built and re-bills the full input on the next model.
We ended up tracking cache-hit-rate per step as its own line item. Once you split the two failure types, did it change your actual cost numbers much, or mostly just change where the spikes showed up?
I don't have a measured before/after to give you, my comment was reasoning from how the billing works rather than from a measured split. I'd expect the total to actually drop, but only by the malformed-output share, since the timeout side doesn't change. Timeouts and 5xx still pay a full first read on the next model, so that part of the bill doesn't really change. The malformed-output retries are where it shifts, because a retry that lands inside the cache window bills the replayed prefix at the cache-read rate, which is a fraction of the full input price on the providers that offer caching. The regenerated output is still full price though, so there's a ceiling on it. How much that saves overall depends on how many of your failures were malformed output in the first place. Tracking cache-hit-rate per step is a good call, it's the number that tells you whether the in-place retries are actually landing warm. What share of your retries turned out to be malformed output vs timeouts?
I'm not following the "context paid once" part.
model.call(task.handoff_context)still sends input to the next model. Is your gateway absorbing that cost, or is the handoff context smaller? Switching providers alone doesn't explain the drop from 4x to 1x.Fair challenge — "paid once" was sloppy framing on my part. Model B absolutely pays for what it receives; routing doesn't make tokens free. What it actually changes is three things: (1) the rate — cheap steps hit cheap models; (2) the handoff carries the slimmed context (results + deltas), not the full original replay; (3) a retry that doesn't re-attach the full original. The 4×→1× lives mostly in (2) and (3), not in "paying once." You're right that switching providers alone barely moves the number — the win needs the smaller handoff and a retry that doesn't re-bloat. How are you scoping what travels in the handoff today — full context, or a trimmed carry?
The audit trick that worked for us: log
cache_read_input_tokensnext to each attempt id, then join retries by task id — the replay tax falls out of the billing data as input-tokens-minus-cache-reads per attempt, no app instrumentation needed.One counterpoint to the routing fix: "equivalent" drifts. A fallback model with a different tool-calling format silently degrades the handoff context, so you trade a 4x bill for a quiet quality drop. Worth logging which tier actually served each attempt.
Does your handoff context carry tool definitions too? If model B doesn't expose the same tools, the task may need a re-plan rather than a rerun.
The audit trick is exactly right, and the part I like is that it needs zero app instrumentation — the replay tax just falls out of billing data once you join by task id.
On the format-drift counterpoint: yes, that's the silent killer. The tool schema isn't "data" you carry, it's part of the context the model reasons over. When model B speaks a different tool dialect, replaying the same tool_calls doesn't just degrade — it breaks, and the failure looks like a model problem, not a routing one. The split that's worked for us: the routing layer validates tool compatibility before handoff. If B can't express the same tools, you don't rerun — you either re-plan against B's toolset or you translate the schema. Rerunning the identical call is the worst of both.
Question for you: do you pin tool schemas per-model in a registry up front, or infer compatibility at call time? The up-front pinning catches the drift before it ships, but it rots the moment a provider changes a format.
This matches what I see running LLM traffic behind an API gateway. Retries replay the full context and the bill lands on the shared budget, not the caller. Making the task travel with its context instead of replaying it is the right call.
The shared-budget sleeper is the part I'd put on a poster: the caller sees a cheap call, the retry bills the pool, and nobody owns the delta. Making the task carry its context instead of replaying it is the real fix, not a smarter backoff. When you carry context, do you carry the raw transcript or a compacted artifact? That choice changes what the retry can actually recover from — a compacted task might lose the exact state a retry needs to resume cleanly — do you carry the transcript verbatim, or has the compacted version ever dropped something a retry actually needed?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.