How I Cut Our LLM API Bills by 73% With Prompt Caching: A Production Engineer's Guide (2026)

Last quarter, my Anthropic console showed $612 in API costs across our six in-house AI products. After a focused two-week refactor centered on prompt caching, the same workload dropped to $167 — a 73% cut without changing models, downgrading reasoning quality, or hitting a single rate limit. The same techniques applied to OpenAI's Responses API brought another $94/month off our GPT-5 spend.

If your team is running LLMs in production in 2026 and your monthly bill keeps creeping up, prompt caching is the single highest-leverage thing you can do this week. Not next quarter. Not after the migration. This week. I've now implemented it across all six of our shipped AI products — SmartExam AI Generator, DiabeCheck Food Scanner, BizChat Revenue Assistant, DocSumm AI Summarizer, ServiceBot AI Helpdesk, and ContentForge AI Studio — and the pattern is consistent enough that I want to write down exactly what worked, what didn't, and the mistakes that cost me real money before I figured them out.

This guide is written for engineers who already ship LLM features and want concrete numbers, not for executives who need a 30,000-foot view. Skip ahead to the implementation walkthrough if you want code-level detail.

Why prompt caching matters more in 2026 than it did in 2025

Three things changed in the last twelve months that make caching the dominant cost lever, not a nice-to-have:

• Context windows ballooned, and so did the cost of filling them. Claude Opus 4.7 ships with a 1M-context window. GPT-5.4 followed. Gemini 2.5 Pro stayed at 2M. Every customer interaction that benefits from long context — RAG, codebase analysis, document review — now stuffs 30K-200K tokens into the request. At Sonnet 4.6 input pricing of $3 per million tokens, a single 100K-token request costs $0.30 in input alone. Run that 5,000 times a day and you are spending $1,500/day on tokens your model has already seen.
• Pricing finally rewards caching. Anthropic's prompt caching pricing settled at 10% of base input cost for cache reads and 1.25x base for cache writes, with optional 1-hour TTL extensions. OpenAI launched semantic caching on the Responses API in late 2025 with similar economics. Gemini introduced explicit cached content in early 2026. The bet you are making is no longer "this might work" — it is "I am paying 10x what I should because I never turned it on."
• Tool-calling traces dominate token spend in agentic apps. When ServiceBot AI Helpdesk handles a customer ticket, the conversation history plus tool definitions plus example trajectories add up to ~28K tokens before the user even types a question. Without caching, every turn re-pays for that prefix. With caching, the prefix is amortized across the whole conversation.

The third reason is the one most teams underestimate. If your AI feature is a single-shot chat completion, caching saves you maybe 20%. If your AI feature is an agent loop with 8-15 tool calls per task — which is what real production AI workloads look like in 2026 — caching saves you 60-85%. The bigger the agent, the bigger the win.

How prompt caching actually works under the hood

Skip this section if you already know. Otherwise, here is the mental model that took me two weeks to internalize and that I wish someone had drawn for me on day one.

An LLM processes a prompt by computing internal representations (key-value tensors, in transformer terms) for every token in the input. For a 100K-token prompt, that is a significant amount of compute. When you send the same prompt prefix again — say, a long system message followed by tool definitions, followed by example trajectories — the model would normally redo all that work from scratch.

Prompt caching stores those internal representations on the provider's side after your first request. On subsequent requests with the same prefix, the provider loads the cached tensors instead of recomputing them. You pay a small fee to write the cache (typically 1.25x normal input cost on Anthropic, similar on OpenAI), and then every subsequent read costs 10% of normal input pricing.

The catch: caches expire. Anthropic's default TTL is 5 minutes. The 1-hour TTL costs slightly more to write but pays back if your traffic patterns are bursty. OpenAI's caches are more opaque — they refresh on access, with no documented TTL guarantee, which makes capacity planning slightly trickier.

What this means in practice: structure your prompts so the static stuff sits at the top, the dynamic stuff sits at the bottom, and the cache breakpoint sits exactly between them. Get the ordering wrong and you cache nothing useful.

Provider-by-provider comparison: who has the best cache in 2026

I'll be honest about which provider's caching I actually run in production for which workload. We use all three depending on the product, and the choice matters.

The pricing math: if you have a 60K-token static prefix you reuse 1,000 times in an hour, Anthropic charges you about $0.225 in writes + $18 in cached reads. Without caching, the same workload costs $180 in input tokens. That is the 90% reduction Anthropic markets — but only if your prefix is genuinely static and your traffic is dense enough to amortize the writes.

OpenAI's automatic caching is the easiest to ship — you literally do nothing and the API just starts caching common prefixes. The downside is you cannot force a cache write or guarantee a hit. For predictable cost savings in a known-traffic workload, I prefer Anthropic's explicit model. For unpredictable consumer traffic, OpenAI's transparent approach is less work.

Gemini's caching is the right pick for one specific shape: a single huge context (a 400K-token codebase or a 200-page PDF) that many users will query against. The minimum cacheable prefix is 32K tokens, which makes it useless for normal chat workloads, but exceptional for document Q&A.

Implementation walkthrough: how I cache prompts across our six AI products

Here is the actual pattern that drives the cost reductions I quoted at the top. I'll use Anthropic's API since it's where I get the biggest wins, but the structural approach translates to OpenAI and Gemini.

Step 1: identify your cacheable prefix

For SmartExam AI Generator — our exam question generation product running on Laravel + a Python service — the cacheable prefix breaks down as:

• System instructions (about 4,200 tokens) — never changes
• Tool definitions for retrieval and validation (about 3,800 tokens) — changes ~once a month
• Few-shot examples for question quality (about 18,400 tokens) — changes ~once a quarter
• Per-user context (subject, grade level, language) — changes every request
• Conversation history — changes every turn

The first three blocks total 26,400 tokens and are stable for at least a day, often a week. Those go above the cache breakpoint. The per-user context and conversation history go below, where the cache is irrelevant anyway because they change constantly.

Step 2: place the cache_control marker correctly

This is the single most common mistake I see in code reviews. Engineers slap cache_control on the wrong block and end up caching nothing. The Anthropic API caches everything from the start of the messages array up to and including the block marked with cache_control. So you mark the last static block, not the first.

Concretely, in Python:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": SYSTEM_INSTRUCTIONS},
            {"type": "text", "text": TOOL_DEFINITIONS},
            {
                "type": "text",
                "text": FEW_SHOT_EXAMPLES,
                "cache_control": {"type": "ephemeral"}
            },
            {"type": "text", "text": user_query}
        ]
    }
]

The cache_control on FEW_SHOT_EXAMPLES tells Anthropic to cache everything from SYSTEM_INSTRUCTIONS through the end of FEW_SHOT_EXAMPLES. The user_query is below the breakpoint and isn't cached, which is correct — it changes every request.

Step 3: verify your cache is actually hitting

The response includes a usage object with two new fields: cache_creation_input_tokens and cache_read_input_tokens. On a successful cache hit, you should see the read field populated and the regular input_tokens dropped to just the uncached portion.

I log every request's cache stats to our observability stack (we use Langfuse for LLM tracing). The first thing I check after deploying any cache-aware code path is whether the cache hit rate is above 70% in the first hour. Below that and something is wrong — usually a non-deterministic field somewhere in the supposed-static prefix, like a timestamp accidentally baked into a tool definition.

Step 4: use the 1-hour TTL when traffic is bursty

The default 5-minute TTL works for steady traffic. For BizChat Revenue Assistant, where one customer session might span 15-25 minutes of intermittent activity, the default kept expiring between turns. Switching to the 1-hour TTL ("cache_control": {"type": "ephemeral", "ttl": "1h"}) cost slightly more on the first write but eliminated re-write costs throughout the session.

For ContentForge AI Studio, which runs nightly batch jobs at peak load, the 1-hour TTL is overkill — the workload is dense enough to keep the 5-minute cache warm. Different products, different settings. Measure first.

Real numbers from production across our six AI products

I logged the before/after for each product over a 30-day window in April 2026. These are real numbers from real customer traffic, not synthetic benchmarks.

The 73% number I opened with is the same workloads measured a month later after a second round of optimization — moving DiabeCheck's image preprocessing to a smaller model and merging redundant tool definitions in ServiceBot. The raw caching win is 63%; the rest came from cleanup that the caching audit surfaced.

Notice the workloads where caching barely moved the needle. DiabeCheck and DocSumm have minimal repeatable prefix — every food image is new, every document is new. There is nothing meaningful to cache. If your AI feature looks like these two, caching is not your highest-leverage cost lever; model selection probably is.

The mistakes that cost me real money before I figured them out

Five lessons from the first week of botched implementations. If you avoid these, you'll get to the savings faster than I did.

Mistake 1: putting cache_control on the wrong block. I initially put the marker on the first block (system instructions) instead of the last static block (few-shot examples). Result: only the 4,200-token system prompt was cached, leaving 22,200 tokens of examples to be paid for in full on every request. The fix took 30 seconds; identifying the bug took 4 hours of staring at our cost dashboard.

Mistake 2: variable substitution inside the cached prefix. Our SmartExam tool definitions had a templated {{exam_subject_hint}} that got filled in per request. Even though it looked like a tiny variation, it broke the cache key for the entire prefix. Move all variability below the cache breakpoint, always. If a value must appear in a tool description, hard-code it in a per-user wrapper instead of inlining it into the cached block.

Mistake 3: not enabling caching on tool definitions. The Anthropic API caches tool definitions if you mark the last tool with cache_control. For ServiceBot AI Helpdesk, which has 14 tools totaling 9,800 tokens, this single fix dropped per-turn costs by 31%. Tool definitions are almost always stable across a deploy and are the highest-leverage thing to cache after system prompts.

Mistake 4: ignoring the cache write penalty in low-traffic periods. For ContentForge, we have peak batch hours at 2 AM Asia/Jakarta time and very low traffic during US working hours. With 5-minute TTL, the cache kept expiring during the slow period, paying the 1.25x write penalty on the first request each cycle without ever amortizing it. Switching to 1-hour TTL for the off-peak window solved the issue, with the trade-off that the per-write fee is slightly higher.

Mistake 5: caching when you shouldn't. Not every workload benefits. If your prefix changes more than once per minute, or if your total request volume is below ~50/hour against the same prefix, you're paying for cache writes that never amortize. Measure cache hit rate weekly. If it sits below 40%, turn caching off for that endpoint and reinvest the effort elsewhere.

When to cache, when not to: a decision matrix

I run new AI features through this checklist before deciding whether to invest engineering time in caching:

• Is the prefix at least 1,024 tokens? If no, skip — below Anthropic's minimum, you can't cache anyway.
• Does the same prefix repeat in at least 50 requests/hour? If no, the write overhead probably exceeds the read savings.
• Is the prefix stable for at least an hour? If no, you'll burn write costs constantly. Either restructure the prompt to expose a more stable core or skip caching for now.
• Does the workload run agent loops or multi-turn conversations? If yes, caching is high-leverage — prioritize it. Single-shot workloads have less to gain.
• Is your current spend on this workload above $50/month? Below that, the engineering time costs more than the savings. Above that, the math almost always works.

For a typical SaaS product launching an AI feature in 2026, the answer to all five questions is "yes" within a quarter of product-market fit. Plan for caching in your architecture from day one. Retrofitting is cheap, but designing for it is cheaper.

Frequently asked questions

Do I need a special SDK to use prompt caching?

No. The Anthropic Python SDK supports caching natively via the standard cache_control field. OpenAI's caching is automatic with the Responses API and requires no code changes. Gemini caching requires explicit cache object creation via the Google AI SDK. All three are documented in their respective official docs.

What about Bedrock and Vertex AI proxies for Claude and Gemini?

Bedrock supports Anthropic's prompt caching as of late 2025. Vertex AI supports Gemini's context caching. The feature parity is now solid enough that you can pick your deployment surface (direct API, Bedrock, or Vertex) based on procurement and compliance, not caching support.

How does caching interact with extended thinking / reasoning tokens?

Cached prefixes work with extended thinking on Claude. The reasoning tokens themselves are not cached — they're generated fresh per request — but the prompt prefix is, so the input cost savings still apply. For Opus 4.7 with deep reasoning, this matters a lot because the reasoning chain can run 20-50K tokens of output even before the final answer.

Will my data be leaked through someone else's cache hit?

No. Caches are scoped to your API key (and on some providers, to your organization). Another customer cannot get a cache hit on your prefix even if they happen to send identical text. Anthropic, OpenAI, and Google all document this explicitly in their security docs.

Should I cache user input?

Almost never. User input changes every request and won't generate cache hits. The only exception is if you have a small set of templated user queries (e.g., a "summarize this URL" feature where the URL is the only variable) — but even then, you'd cache the templated wrapper, not the user-provided value.

Does caching change the model's output quality?

No. The cached representations are identical to what the model would have computed without caching. Output quality is unchanged.

What I'd build today if starting from scratch

If I were spinning up an AI product in 2026 with no legacy commitments, the architecture would assume caching from day one. Concretely:

• A clear separation between "platform layer" (system instructions, tool definitions, few-shot examples) and "request layer" (user input, session state, dynamic context). The platform layer goes above the cache breakpoint. The request layer goes below.
• An observability hook on every LLM call that logs cache hit rate, write count, and total token cost. We use Langfuse, but Helicone or OpenTelemetry-based stacks work equally well.
• A weekly review of cache hit rates per endpoint. Anything below 60% gets investigated. Anything below 40% gets reverted to non-cached unless I can identify the cause.
• For agent workflows, a "cache warming" job that fires synthetic requests every 4 minutes during the user's active window, keeping the 5-minute cache from expiring between turns. This costs ~$0.50/day for ServiceBot and saves about $15/day in cache rewrites.

The wider point: caching is not a clever optimization you bolt on at the end. It's an architectural assumption you bake in at the start. The teams I see struggling with LLM costs in 2026 are almost always teams that ignored prompt structure until the bill got scary, then tried to retrofit caching onto a codebase whose prompts were never designed to be cached. The teams winning are the ones where the staff engineer thought about prompt structure on day one.

If you take one thing from this post, take this: open your last 100 LLM API calls, look at how much of each request is identical, and ask yourself why you're paying full price for the same tokens 100 times. Then go fix it. The win is sitting there waiting for you.