Does prompt caching actually cut LLM cost? A cost-model experiment
A first-principles cost model showing how much prompt caching saves as a function of two variables you control: the share of input that is a stable prefix, and how often it repeats.
Teams reach for a smaller model when the LLM bill climbs. How much would prompt caching save instead, and what determines whether it is worth turning on?
Method
This is a computational experiment: rather than measure one provider, I model the cost algebra that every provider shares, so the conclusion holds regardless of the exact prices.
Cost per request is input tokens times input price plus output tokens times output price. With caching, the cached share of input is billed at a discounted multiplier and the rest at full price. I vary two inputs — the fraction of input served from cache and the discount multiplier — and hold everything else fixed.
You can reproduce every figure by entering the same variables into the LLM API cost calculator and reading cost per request as the cached share moves from 0% to 100%.
Observations
Savings are linear in the cached share of input. If cached tokens are billed at one tenth of full price, moving from 0% to 80% cached cuts the input bill by roughly 72% — the saving is the cached fraction times one minus the discount multiplier.
The output side is untouched by caching. In output-heavy workloads the total saving is therefore much smaller than the input saving, which is why the input/output split matters as much as the cache rate.
The break-even is a prefix that repeats. A long stable system prompt reused across many calls is almost pure upside; a workload where every request is unique sees essentially no benefit no matter how generous the discount.
Reproduce this with the related tool: /tools/llm-api-cost-calculator
Limitations
This models the steady-state read discount only. Real caching has minimum cacheable lengths, expiry windows, and sometimes a write surcharge, all of which reduce the idealized saving.
It is a cost-model demonstration, not a measurement of any specific provider or model. Plug in current prices to get numbers that apply to your case.
It says nothing about latency or quality — both of which caching can also affect, and neither of which this experiment measures.
Follow the experiments
I share methods, results, and what did not work. Get new experiments by email — occasional, no spam, unsubscribe anytime.