Most teams reach for a smaller model when the language-model bill climbs. Often there is a cheaper move that costs no quality at all: prompt caching. If your requests repeat a long, stable chunk of input — a system prompt, a knowledge document, a big tool schema — caching lets you pay full price for that chunk once and a steep discount every time after.
Here is what it actually does, when it helps, and when it does not.
What prompt caching does
When a provider processes a prompt, it converts the input tokens into an internal representation before generating a response. Prompt caching stores that representation for a prefix of your prompt. On the next request that begins with the exact same prefix, the provider reuses the cached work instead of reprocessing it, and bills those cached input tokens at a fraction of the normal rate.
The key word is prefix. Caching works on a stable beginning of the prompt. Everything up to the first point where requests differ can be cached; everything after that point is processed fresh each time.
Why the prefix rule shapes your prompts
Because only a shared prefix caches, the order of your prompt matters. Put the stable, reusable content first — system instructions, reference documents, tool definitions — and the variable content, like the user's specific question, last. A prompt organized this way maximizes the cacheable prefix; the same content in the reverse order caches almost nothing.
This is a small change with a large effect. Many teams leave money on the table simply by interleaving stable and variable content instead of front-loading the stable part.
When caching pays off
Caching rewards repetition of a large, stable prefix. Three patterns benefit most: a long system prompt reused across many calls; a retrieval or document-grounded workload where the same source text is queried repeatedly; and agents, where the system prompt and early conversation recur on every step of a task.
The size of the win depends on two numbers: how much of your input is the shared prefix, and how often it repeats. You can put your own figures against it — cached share and the discounted rate — with the LLM API cost calculator. Workloads with a long stable prefix and short variable tails see the biggest reduction.
When it does not help
Caching does little when every request is unique, when the shared prefix is short, or when requests arrive too far apart — cached prefixes expire after a provider-specific window, so sporadic traffic keeps missing the cache. There can also be a small surcharge to write to the cache, so caching a prefix used only once or twice can cost slightly more, not less.
The takeaway
Before downgrading your model to save money, check whether your prompts share a long stable prefix — and if they do, front-load it and turn on caching. It is the rare optimization that cuts cost with no effect on output quality, which makes it the first place to look, not the last.