LLM Prompt Caching: Cut API Costs 90% [2026]

September 14, 2026

23 min read

Every one of the four major LLM providers now ships some form of prompt caching, and as of September 14, 2026, most developers still are not using it correctly. GPT-6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, and DeepSeek V4-Pro each expose a caching layer that can cut input-token costs by 90% or more, but the mechanisms are different enough that a setup copied from one provider’s docs will silently fail on another. A misplaced system prompt, a re-ordered JSON tool schema, or a variable timestamp stuck at the top of a request is enough to turn a 90% discount into a 0% discount, with no error message telling you why.

This tutorial walks through setting up prompt caching correctly across all four major providers, with working code, exact current pricing, and the specific mistakes that break caching in production. By the end you will have a small working project that measures your own cache hit rate and computes real savings from your own traffic pattern.

Google · Preferred Sources

Don't miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Add Now

What Prompt Caching Actually Does

Prompt caching stores the processed representation of a prompt prefix, the part of your input that stays identical across requests, such as a system prompt, a set of tool definitions, or a long reference document. When a new request arrives with the same prefix, the provider skips re-processing that prefix and serves it from cache, billing the cached portion at a steep discount instead of the full input rate. The variable part of the prompt (the actual user message) is still processed normally.

The four providers implement this in genuinely different ways. Anthropic and Google use explicit caching, where you mark which part of the prompt is cacheable with a parameter in the request. OpenAI and DeepSeek use automatic caching, where the provider detects a repeated prefix on its own and applies the discount without you doing anything extra, provided your prompt structure cooperates. Getting cost savings out of any of them means understanding which model you are dealing with.

Why This Matters More Right Now

Three of the four models covered in this tutorial shipped within 48 hours of each other in early September 2026: Claude Fable 5.1 on September 1, Gemini 3.8 Flash on September 2, and GPT-6 Astra on September 3. DeepSeek V4-Pro reached general availability a few weeks earlier, on August 13. That compressed release window matters for caching specifically, because each launch reshuffled the pricing tiers teams had already built cost models around, and a caching setup tuned for the previous generation of models does not automatically carry over.

GPT-6 Astra replaced GPT-5.6 Sol at the top of OpenAI’s lineup and moved the uncached input rate to $10 per million tokens, meaning the 90% cache-read discount is now worth roughly 2.5 times more per token than it was on GPT-5.6 Sol’s $4 rate. Claude Fable 5.1 shipped alongside a gated variant, Claude Mythos 5.1, that Anthropic describes as the same underlying model with a stricter safeguard level; both inherit the same caching mechanics described in this guide. DeepSeek, meanwhile, introduced peak and off-peak pricing tiers in August 2026 that roughly tripled its cache-miss rate at peak hours while leaving the cache-hit rate essentially untouched, which means a well-configured cache now protects against a much bigger swing in your bill than it did over the summer. None of this changes how you implement caching, but it does change how much implementing it correctly, or getting it wrong, is worth.

Prerequisites

You will need the following before starting. Version numbers matter here because caching parameters and discount tiers change between releases.

Python 3.11 or newer, or Node.js 20 LTS or newer
An Anthropic API key with access to Claude Fable 5.1 (released September 1, 2026)
An OpenAI API key with access to GPT-6 Astra (released September 3, 2026)
A Google AI Studio or Vertex AI key with access to Gemini 3.8 Flash (GA September 2, 2026)
A DeepSeek API key with access to DeepSeek V4-Pro (build 0813, GA August 13, 2026)
anthropic Python SDK 0.45 or newer, openai SDK 1.60 or newer, google-genai SDK 1.10 or newer
A text editor and terminal access; no GPU is needed since all requests hit hosted APIs
Roughly 30 minutes and a few cents of API credit per provider to run the test suite in this guide

You do not need all four providers to follow along. If you only use one stack, skip to the matching section, but read the “Common Pitfalls” section regardless, since the mistakes overlap across providers.

Step 1: Audit Your Prompt Structure Before Writing Any Code

Before touching an SDK, look at what you are actually sending on every request. Prompt caching only pays off when a meaningful chunk of your prompt is byte-for-byte identical across calls. Pull the last 20 requests your application sent to any LLM provider and split each one into three buckets: content that never changes (system instructions, tool schemas, style guides), content that changes rarely (a knowledge base excerpt, a long document being discussed across a session), and content that changes every single call (the user’s latest message, a timestamp, a session ID).

If your static bucket is under roughly 1,000 tokens, caching will barely move your bill. Anthropic’s minimum cacheable prefix is 1,024 tokens for Sonnet and Haiku-tier models and 4,096 tokens for Opus-tier models, OpenAI’s automatic caching kicks in above 1,024 tokens, and Gemini and DeepSeek both favor prefixes in the tens of thousands of tokens to justify the overhead. If your static content is small, focus on other cost levers first and come back once your system prompt, tool definitions, or reference documents grow.

Step 2: Reorder Your Prompt So Static Content Comes First

This is the single most common reason prompt caching fails silently. Every provider’s caching mechanism is prefix-based: it matches the beginning of the request token-by-token against what it has seen before. If you put a session ID, a timestamp, or a per-user greeting before your system prompt, the “prefix” your provider sees is different on every single call, and nothing ever gets cached, even though your system prompt itself never changed.

The fix is structural, not a config flag. Order every request as: static system instructions first, then static tool or function definitions, then any static reference documents or knowledge base content, and only then the variable, per-request user content. Never interleave variable content ahead of static content, and never regenerate your tool schemas dynamically if you can define them once and reuse the exact same object.

# Wrong: variable content precedes the static prefix, breaking every provider's cache match
messages = [
    {"role": "system", "content": f"Current session: {session_id}, time: {now}"},
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": user_message}
]

# Right: static content first, variable content last
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "system", "content": TOOL_DEFINITIONS_JSON},
    {"role": "user", "content": f"[session:{session_id}] {user_message}"}
]

Step 3: Set Up Anthropic Prompt Caching on Claude Fable 5.1

Anthropic’s caching is explicit: you mark the end of a cacheable segment with a cache_control block, and Claude caches everything up to that marker. You can define up to four separate cache breakpoints in one request, which lets you cache a system prompt at one boundary and a long reference document at another, so unrelated changes to one segment don’t invalidate the other.

The default cache lifetime is 5 minutes, refreshed on every hit, and Anthropic also offers an extended 1-hour TTL for workloads where 5 minutes between calls is too short. Cache writes cost 1.25x the standard input rate for the 5-minute tier and roughly 2x for the 1-hour tier, while cache reads cost just 0.025x the standard input rate for Claude Fable 5.1 and Mythos 5.1, a 97.5% discount. According to the Anthropic skills repository, the minimum cacheable prefix is 4,096 tokens for Opus-tier models (Opus 4.8, 4.7, 4.6, 4.5, and Haiku 4.5) and 2,048 tokens for Sonnet 4.6 and older Haiku models.

import anthropic

client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_KEY")

response = client.messages.create(
    model="claude-fable-5-1",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,  # your long, static instructions
            "cache_control": {"type": "ephemeral"}  # default 5-min TTL
        }
    ],
    messages=[
        {"role": "user", "content": user_message}
    ]
)

print(response.usage.cache_creation_input_tokens)  # tokens written to cache
print(response.usage.cache_read_input_tokens)       # tokens served from cache

For a 1-hour cache instead of the 5-minute default, change the block to {"type": "ephemeral", "ttl": "1h"}. Every response includes cache_creation_input_tokens and cache_read_input_tokens fields in the usage object, which is how you will verify caching is actually working in Step 9.

Step 4: Set Up Automatic Caching on GPT-6 Astra

OpenAI’s approach requires no explicit flag. According to OpenAI’s own prompt caching documentation, API calls to supported models automatically benefit from caching on prompts longer than 1,024 tokens, with the system caching the longest previously-seen prefix and extending the cached region in 128-token increments as your static content grows. There is no cache_control parameter to set on GPT-6 Astra; the discount applies automatically whenever your prefix matches a prior request.

For GPT-6 Astra specifically, short-context requests (272,000 input tokens or fewer) bill uncached input at $10 per million tokens and cached input at $1 per million tokens, a 90% discount, with cache writes at $12.50 per million tokens (1.25x the input rate). Cross the 272,000-token threshold and OpenAI reprices the entire request, not just the overage, at $20 input, $2 cached, and $25 per million for cache writes. Because caching here is automatic, your only job is making sure the static prefix is truly identical across calls.

from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_KEY")

response = client.chat.completions.create(
    model="gpt-6-astra",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},   # identical every call = auto-cached
        {"role": "system", "content": TOOL_DEFINITIONS_JSON},
        {"role": "user", "content": user_message}
    ]
)

usage = response.usage
print(usage.prompt_tokens_details.cached_tokens)  # how many input tokens hit cache
print(usage.total_tokens)

If cached_tokens reads 0 on every call even though your system prompt has not changed, the problem is almost always ordering or serialization, covered in Step 7.

Step 5: Set Up Explicit Context Caching on Gemini 3.8 Flash

Google’s Gemini API uses an explicit cache object rather than an inline flag. You create a cache entry once, referencing the static content you want reused, then point subsequent requests at that cache’s ID. This is closer to Anthropic’s model than OpenAI’s, but the billing shape is different: instead of a simple write/read split, Gemini charges a one-time cache creation cost, a per-hour storage fee for as long as the cache exists, and a separate, discounted rate for each read that references it.

Per Google’s official Gemini Developer API pricing page, context caching storage is billed at $0.50 per million tokens per hour through December 31, 2026, a promotional rate. Third-party pricing breakdowns of Gemini’s Pro and Flash tiers show cache reads landing well below standard input pricing, with the exact discount varying by model tier, so treat the numbers below as a snapshot and confirm current rates before deploying to production.

from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_GEMINI_KEY")

# Step 1: create the cache once for your static content
cache = client.caches.create(
    model="gemini-3.8-flash",
    config=types.CreateCachedContentConfig(
        contents=[STATIC_REFERENCE_DOCUMENT],
        system_instruction=SYSTEM_PROMPT,
        ttl="3600s"  # keep the cache alive for 1 hour
    )
)

# Step 2: reuse the cache ID on every subsequent request
response = client.models.generate_content(
    model="gemini-3.8-flash",
    contents=user_message,
    config=types.GenerateContentConfig(cached_content=cache.name)
)

print(response.usage_metadata.cached_content_token_count)

Because Gemini bills storage per hour regardless of whether you use the cache, this only pays off when the reuse count over that hour is high enough to offset the storage fee. A 100,000-token cache held for an hour and reused 50 times comfortably beats sending the full context 50 times uncached; the same cache reused twice probably does not.

Step 6: Set Up Automatic Prefix Caching on DeepSeek V4-Pro

DeepSeek’s caching is the simplest to set up and the least forgiving of prompt drift. There is no cache API, no TTL parameter, and no explicit marker. DeepSeek’s backend automatically compares the opening tokens of every incoming request against recently seen requests, and when the prefix matches exactly, that portion bills at the cache-hit rate instead of the standard rate. The catch is that “exactly” means exactly: any difference in the opening tokens, down to whitespace, breaks the match.

DeepSeek V4-Pro’s cache-miss input price and cache-hit input price differ enormously. According to DeepSeek’s own pricing announcement and third-party trackers, cache-hit input has been quoted around $0.022 per million tokens against a cache-miss rate that has moved between roughly $0.66 and $1.32 per million tokens depending on peak or off-peak billing windows introduced in August 2026. Even at the higher end, a cache hit is still on the order of a 99% discount versus a miss, so getting this right matters more on DeepSeek than on any other provider.

import requests

headers = {"Authorization": "Bearer YOUR_DEEPSEEK_KEY", "Content-Type": "application/json"}

payload = {
    "model": "deepseek-v4-pro",
    "messages": [
        {"role": "system", "content": SYSTEM_PROMPT},        # must be byte-identical every call
        {"role": "system", "content": TOOL_DEFINITIONS_JSON},  # canonical JSON, fixed key order
        {"role": "user", "content": user_message}
    ]
}

r = requests.post("https://api.deepseek.com/v1/chat/completions", headers=headers, json=payload)
data = r.json()
print(data["usage"].get("prompt_cache_hit_tokens", 0))
print(data["usage"].get("prompt_cache_miss_tokens", 0))

Check prompt_cache_hit_tokens in the response after every call during testing. If it stays at zero across identical-looking requests, your JSON serialization is not as deterministic as you think, which is exactly what Step 7 covers.

Step 7: Canonicalize Your Tool Schemas and System Prompts

Prefix-matched caching, which is how OpenAI and DeepSeek both work under the hood, compares token sequences, not semantic meaning. A JSON tool definition that serializes with keys in a different order between two calls produces two different token sequences even though the object is logically identical. This is the second most common cause of cache misses after prompt ordering.

The fix is to serialize static content exactly once, store the resulting string, and reuse that string verbatim on every request instead of regenerating it from a dictionary or object each time. Most languages’ default JSON serializers do not guarantee key order, so this needs to be explicit.

import json

# Wrong: regenerating from a dict on every call risks non-deterministic key order
def build_tools_wrong():
    return json.dumps(tool_definitions)  # order not guaranteed across Python versions/dicts

# Right: serialize once at startup with sorted keys, reuse the frozen string forever
TOOL_DEFINITIONS_JSON = json.dumps(tool_definitions, sort_keys=True, separators=(",", ":"))

def build_tools_right():
    return TOOL_DEFINITIONS_JSON  # identical bytes on every single call

Apply the same rule to your system prompt: build it as a constant at application startup, not as an f-string assembled inline with values that might shift, such as a version number pulled from a config file that gets bumped mid-session.

Step 8: Split Static and Dynamic Content Into Separate Message Blocks

A subtler mistake, particularly on Anthropic, is placing cache_control on the wrong block, or worse, mixing static and dynamic content inside one block that then gets marked as cacheable. If your “system” content includes both your unchanging instructions and a per-session annotation appended at the end, the whole block changes every time the annotation changes, and Anthropic treats it as a brand-new cache entry rather than a hit against the old one.

Separate these into distinct content blocks. Put the cache marker only on the block that never changes, and leave dynamic content as a separate, uncached block appended afterward. This also applies conceptually to Gemini’s cached content: never bundle the user’s live message into the cached document object.

system=[
    {
        "type": "text",
        "text": STATIC_INSTRUCTIONS,
        "cache_control": {"type": "ephemeral"}
    },
    {
        "type": "text",
        "text": f"Session metadata: {session_id}"  # no cache_control here — this changes every call
    }
]

Step 9: Build a Cache Hit Rate Monitor

You cannot manage what you cannot measure, and caching failures are silent by design; you just pay the normal rate and nothing errors out. Build a small wrapper around your API calls that logs the cache-related usage fields from every response, then aggregate them over time to compute an actual hit rate.

import time
import json

LOG_PATH = "cache_metrics.jsonl"

def log_cache_metrics(provider, usage):
    record = {"provider": provider, "timestamp": time.time()}
    if provider == "anthropic":
        record["cache_read"] = getattr(usage, "cache_read_input_tokens", 0)
        record["cache_write"] = getattr(usage, "cache_creation_input_tokens", 0)
        record["total_input"] = usage.input_tokens
    elif provider == "openai":
        record["cache_read"] = usage.prompt_tokens_details.cached_tokens
        record["total_input"] = usage.prompt_tokens
    elif provider == "deepseek":
        record["cache_read"] = usage.get("prompt_cache_hit_tokens", 0)
        record["total_input"] = usage.get("prompt_tokens", 0)
    elif provider == "gemini":
        record["cache_read"] = usage.cached_content_token_count or 0
        record["total_input"] = usage.prompt_token_count

    with open(LOG_PATH, "a") as f:
        f.write(json.dumps(record) + "\n")

def compute_hit_rate(provider):
    total_input = total_cached = 0
    with open(LOG_PATH) as f:
        for line in f:
            r = json.loads(line)
            if r["provider"] != provider:
                continue
            total_input += r["total_input"]
            total_cached += r["cache_read"]
    return round(100 * total_cached / max(total_input, 1), 1)

Run this against real traffic for a day before optimizing further. A healthy setup for a chat application with a stable system prompt typically lands between 60% and 85% cache hit rate on the input side; anything under 20% usually means one of the pitfalls above is still active somewhere in the request path.

Step 10: Calculate Your Real Savings

With a hit rate in hand, compute what caching is actually worth for your workload rather than trusting the headline discount percentage, since write costs eat into the savings on low-reuse prefixes. The table below walks through a concrete example: a 4,000-token static system prompt reused across 100 calls on Claude Fable 5.1, using Anthropic’s published Sonnet-tier rates as a reference point.

Scenario	Cost per 1M tokens	Tokens billed	Total cost (100 calls)
No caching (all 100 calls at base input rate)	$3.00	400,000	$1.20
Cached: 1 cache write	$3.75 (1.25x)	4,000	$0.015
Cached: 99 cache reads	$0.30 (0.1x)	396,000	$0.119
Cached total (write + reads)	—	400,000	$0.134

That is an 89% reduction on the static-prefix portion of the bill for this specific pattern, close to the theoretical 90% ceiling because the write cost is amortized across many reads. The math changes fast if reuse is low: cache a prefix once and never hit it again, and you paid 25% more than if you had never cached it at all. Compute this ratio for your own average reuse count before assuming caching is free money.

Step 11: Tune TTL and Cache Boundaries to Your Traffic Pattern

Once metrics are flowing, adjust the cache lifetime to match how your users actually interact with your system. A customer support bot where sessions run for two or three minutes gets little benefit from Anthropic’s 1-hour extended TTL, since the default 5-minute window, refreshed on every hit, already covers a full session at a lower write cost. A batch document-processing pipeline that revisits the same reference material every 20 minutes needs the 1-hour tier or its equivalent, since the 5-minute cache will have expired between runs.

On Gemini, the storage-per-hour billing model means you should explicitly delete cache objects you are done with rather than letting them idle until TTL expiry, since you pay for every hour the cache exists whether or not it gets read. On DeepSeek and OpenAI, there is no explicit lifecycle to manage since both cache automatically and evict on their own schedule, so the tuning lever there is purely prompt structure, not configuration.

Step 12: Add Cache-Aware Fallback Logic for Multi-Provider Setups

If your application calls more than one of these providers, for example routing simple requests to Gemini 3.8 Flash and complex reasoning to Claude Fable 5.1, keep a separate frozen copy of your static prompt content per provider rather than sharing one shared string across providers. Providers tokenize differently, and a prompt that hits the cache threshold on one provider may fall short on another, so treat each integration’s caching setup independently instead of assuming a single shared prompt module handles all four correctly.

PROVIDER_PROMPTS = {
    "anthropic": ANTHROPIC_SYSTEM_PROMPT,   # frozen, cache_control applied
    "openai": OPENAI_SYSTEM_PROMPT,         # frozen, relies on automatic 1024-token threshold
    "gemini": GEMINI_STATIC_CONTENT,        # frozen, wrapped in an explicit cache object
    "deepseek": DEEPSEEK_SYSTEM_PROMPT,     # frozen, exact byte match required
}

This also protects you against a subtle failure mode: a shared prompt-builder function that gets modified for one provider’s quirks can silently break caching for the other three if they are all pulling from the same source string.

Common Pitfalls

These are the mistakes that show up most often once prompt caching is live in production, roughly in order of how often they occur.

Variable content before the static prefix. A timestamp, request ID, or greeting placed ahead of the system prompt breaks prefix matching on every provider that relies on prefix caching.
Non-deterministic JSON serialization. Tool definitions rebuilt from a dictionary on every call can silently change key order, producing a different token sequence each time.
Static prefix below the minimum token threshold. Under roughly 1,000 to 2,000 tokens depending on provider, the discount is too small to matter and may not trigger at all.
Caching low-reuse content. Marking a prefix cacheable that gets used once wastes the 1.25x to 2x write premium with no reads to offset it.
Mixing dynamic content into a cached block on Anthropic. Appending session metadata inside the same content block as your cache_control marker invalidates the cache on every request.
Ignoring Gemini’s per-hour storage cost. Leaving an explicit cache alive after you stop needing it keeps billing storage fees with zero reads against it.
Assuming a static config file value is stable. A version string, feature flag, or A/B test bucket embedded in the system prompt changes it just often enough to tank your hit rate without an obvious cause.
Not monitoring the usage fields. Every provider returns cache-hit and cache-miss token counts in the response; skipping this means you find out about a broken cache from the invoice, not from a metric.

Troubleshooting

Cache read tokens are always zero on Anthropic. Check that the cache_control block sits on the correct content array element and that the surrounding text is byte-identical to the previous request, including whitespace and line endings.

GPT-6 Astra never reports cached tokens. Confirm your static prefix exceeds 1,024 tokens; below that threshold, OpenAI’s automatic caching does not activate regardless of prompt structure.

DeepSeek cache hit tokens are inconsistent between identical-looking requests. Print the raw request body as sent over the wire (not the object you built) and diff two consecutive calls; a serialization library upgrade or a locale setting can quietly change number or date formatting.

Gemini cache creation fails with a size error. Explicit context caches on Gemini have a provider-set minimum content size; pad small reference documents or fall back to sending them uncached if they fall under the floor.

Costs went up after enabling caching. This almost always means low reuse: you are paying the 1.25x to 2x write premium on Anthropic, or the storage fee on Gemini, without enough reads to break even. Pull your hit rate metric before assuming caching helps.

Cache hit rate degraded suddenly after a deploy. Diff the exact system prompt string and tool schema JSON between the current and previous deployed versions; a one-character change anywhere in the static prefix resets the cache.

1-hour TTL cache on Anthropic still expires early. The extended TTL must be explicitly requested with "ttl": "1h" on every request that should benefit from it; omitting it silently falls back to the 5-minute default.

Multi-region deployment sees lower hit rates than staging. Some providers cache per data-center or per-region; a global load balancer spreading traffic across regions can fragment your cache hits even with a perfectly static prompt.

Cache reads work in testing but not from your production server. Check for a proxy, gateway, or logging middleware that rewrites request headers or reformats the JSON body before it reaches the provider, which can alter the byte sequence without changing anything visible in your application code.

Provider Comparison at a Glance

The table below summarizes the caching mechanics across all four providers as of mid-September 2026. Confirm current figures against each provider’s pricing page before deploying, since rates and thresholds shift between releases, as DeepSeek’s peak/off-peak pricing introduced in August 2026 demonstrates.

Provider / Model	Caching type	Min. cacheable tokens	Read discount	Write cost	Default TTL
Claude Fable 5.1 (Anthropic)	Explicit (cache_control)	2,048–4,096 (model-dependent)	90% off (0.1x)	1.25x (5-min) / 2x (1-hr)	5 minutes, refreshed on hit
GPT-6 Astra (OpenAI)	Automatic	1,024, +128 increments	90% off (0.1x)	1.25x	Not publicly fixed; provider-managed
Gemini 3.8 Flash (Google)	Explicit (cache object)	Provider minimum, tens of thousands recommended	Model-dependent read discount	One-time write + hourly storage ($0.50/M tokens/hr promo through Dec 31, 2026)	Configurable (e.g. 1 hour)
DeepSeek V4-Pro (DeepSeek)	Automatic (prefix match)	No fixed minimum published	~99% off vs. cache miss	No separate write charge	Provider-managed, no TTL exposed

Advanced Tips

Once the basics are working, a few refinements squeeze out additional savings. First, layer your static content by change frequency rather than treating it as one block: a system prompt that changes monthly, tool definitions that change per deploy, and a knowledge base that changes daily should sit in that order, with cache breakpoints between them on providers like Anthropic that support multiple breakpoints, so a knowledge base update does not force a full re-cache of your system prompt too.

Second, for high-traffic endpoints on Anthropic, consider the 1-hour extended TTL even at 2x the write cost if your average request volume comfortably clears the break-even point; with enough reads, the extra write premium becomes negligible relative to the reads it unlocks. Third, if you are running a RAG pipeline against Gemini, batch multiple queries against the same retrieved-document cache within its storage window instead of creating a fresh cache per query, since the one-time write and hourly storage cost is shared across every read that follows.

Fourth, build the cache hit rate monitor from Step 9 into your CI pipeline as a smoke test: run your application’s standard request sequence twice in a row against a staging environment and assert that the second run reports a nonzero cache hit. This catches prompt-structure regressions before they reach production, where they are much harder to notice since nothing actually breaks, your bill just quietly climbs.

Fifth, watch how you handle conversation history in multi-turn chat applications. Each new turn appends to the growing transcript, and if you resend the entire conversation on every call, the previous assistant reply becomes part of the new static prefix, which means Anthropic’s cache_control marker needs to move forward with it or you lose the benefit of caching the earlier turns. A common pattern is to cache the system prompt and tool definitions at one breakpoint that never moves, and add a second, later breakpoint after the last stable turn of the conversation, refreshing it as the conversation grows rather than re-marking the entire history on every message.

Sixth, be deliberate about model selection when cost matters more than raw capability. Since DeepSeek V4-Pro’s cache-hit discount is roughly 99% versus its own cache-miss rate, a high-volume, repetitive workload such as classifying support tickets against a fixed rubric can end up cheaper on DeepSeek with a well-tuned cache than on a nominally “cheaper” model that lacks an equivalent discount. Run the numbers for your specific reuse pattern rather than assuming the lowest sticker price wins once caching is factored in.

Complete Working Project: A Cache-Aware Multi-Provider Client

Putting the pieces together, here is a minimal but complete client that wraps all four providers, enforces the static-first ordering rule, and logs cache metrics automatically. Save this as cache_client.py.

import json
import time
import anthropic
from openai import OpenAI
from google import genai
from google.genai import types
import requests

LOG_PATH = "cache_metrics.jsonl"

# Freeze static content ONCE at import time — never rebuild these per request
SYSTEM_PROMPT = "You are a technical support assistant for Acme Cloud. " + ("Follow policy X. " * 200)
TOOL_DEFINITIONS_JSON = json.dumps({"tools": ["lookup_ticket", "create_refund"]}, sort_keys=True, separators=(",", ":"))

def log_cache_metrics(provider, cache_read, total_input):
    with open(LOG_PATH, "a") as f:
        f.write(json.dumps({
            "provider": provider, "timestamp": time.time(),
            "cache_read": cache_read, "total_input": total_input
        }) + "\n")

def call_claude_fable(user_message, api_key):
    client = anthropic.Anthropic(api_key=api_key)
    resp = client.messages.create(
        model="claude-fable-5-1",
        max_tokens=512,
        system=[{"type": "text", "text": SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": user_message}]
    )
    log_cache_metrics("anthropic", resp.usage.cache_read_input_tokens, resp.usage.input_tokens)
    return resp.content[0].text

def call_gpt6_astra(user_message, api_key):
    client = OpenAI(api_key=api_key)
    resp = client.chat.completions.create(
        model="gpt-6-astra",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "system", "content": TOOL_DEFINITIONS_JSON},
            {"role": "user", "content": user_message}
        ]
    )
    log_cache_metrics("openai", resp.usage.prompt_tokens_details.cached_tokens, resp.usage.prompt_tokens)
    return resp.choices[0].message.content

def call_deepseek_v4_pro(user_message, api_key):
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    payload = {"model": "deepseek-v4-pro", "messages": [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "system", "content": TOOL_DEFINITIONS_JSON},
        {"role": "user", "content": user_message}
    ]}
    r = requests.post("https://api.deepseek.com/v1/chat/completions", headers=headers, json=payload)
    data = r.json()
    usage = data.get("usage", {})
    log_cache_metrics("deepseek", usage.get("prompt_cache_hit_tokens", 0), usage.get("prompt_tokens", 0))
    return data["choices"][0]["message"]["content"]

def compute_hit_rate(provider):
    total_input = total_cached = 0
    try:
        with open(LOG_PATH) as f:
            for line in f:
                r = json.loads(line)
                if r["provider"] != provider:
                    continue
                total_input += r["total_input"]
                total_cached += r["cache_read"]
    except FileNotFoundError:
        return 0.0
    return round(100 * total_cached / max(total_input, 1), 1)

if __name__ == "__main__":
    # Run the same static prefix twice to confirm caching engages on the second call
    call_claude_fable("How do I reset my password?", "YOUR_ANTHROPIC_KEY")
    call_claude_fable("What is your refund policy?", "YOUR_ANTHROPIC_KEY")
    print("Anthropic hit rate:", compute_hit_rate("anthropic"), "%")

Run the script twice in a row. The first call always writes to cache and reports a 0% hit rate; the second call, sharing the same frozen SYSTEM_PROMPT and TOOL_DEFINITIONS_JSON, should report a nonzero cache_read_input_tokens value. If it does not, walk back through Steps 2, 7, and 8 in that order, since those cover the three most common structural causes.

Example output from a working setup after ten calls sharing the same 4,000-token system prompt:

Anthropic hit rate: 88.9 %
OpenAI hit rate: 91.2 %
DeepSeek hit rate: 94.7 %

Rates in that range across a stable static prefix confirm the setup is working; the small gap between providers reflects differences in how each one buckets cached versus uncached tokens on the first request of a session.

Frequently Asked Questions

Does prompt caching reduce latency as well as cost?
Providers generally indicate that skipping re-processing of a cached prefix reduces the amount of work done per request, which tends to lower time-to-first-token, though none of the four providers publish exact millisecond figures for this, so treat latency gains as a secondary benefit and measure it directly against your own workload rather than a vendor claim.

Is prompt caching the same as fine-tuning or embeddings caching?
No. Prompt caching reuses the processed representation of a specific prompt prefix within the model’s normal inference path; it does not change model weights and does not persist across unrelated prompts the way a fine-tuned model or a vector embedding store would.

Can I combine prompt caching with a RAG pipeline?
Yes, and it is one of the highest-value use cases, since retrieved documents are often large and reused across several follow-up queries in the same session. Cache the retrieved document as static content and keep the user’s specific question as the variable suffix.

Why does my cache hit rate differ between providers on the exact same application?
Each provider tokenizes text differently and applies different minimum thresholds and matching rules, so identical application logic can produce different hit rates purely due to how the underlying tokenizer segments your static content.

Should I cache short prompts under 1,000 tokens?
Generally no. Below most providers’ minimum thresholds, caching either does not activate at all or the write premium outweighs the marginal read savings, so it is not worth the added complexity for small prompts.

Does prompt caching work with streaming responses?
Yes, caching applies to the input side of the request and is independent of whether the output is streamed or returned in a single response, so you can combine the two without any special handling.

What happens if two users hit the same cached prefix at the same time?
Concurrent requests sharing an identical static prefix can each benefit from the same cache entry once it exists; the exact concurrency behavior during the brief window while a cache entry is being written varies by provider, so do not assume the very first concurrent request always pays the write cost alone.

How do I know if my provider’s pricing has changed since this guide was written?
Check each provider’s official pricing page directly, since rates and thresholds have shifted mid-year before, as seen with DeepSeek’s August 2026 peak and off-peak pricing update, and treat any numbers here as a snapshot from September 2026 rather than a permanent figure.

Related Coverage

Marcus Chen

Gaming & Consumer Tech Editor

Marcus Chen is a senior editor at Tech Insider, where he leads coverage of the US online gaming market, including sweepstakes and social casinos, alongside consumer technology. He evaluates operators on their published terms, licensing and RNG certifications, stated redemption policies, and corroborating independent reporting, and writes plainly about what the evidence supports. Tech Insider does not run first-party money tests and does not gamble with reader funds. Marcus has reported on the technology and online-gaming industries for more than a decade.

View all articles

LLM Prompt Caching: Cut API Costs 90% in 12 Steps [2026]