Prompt Caching and Cost Optimization

In production AI apps, the same long preamble — system prompt, few-shot examples, long retrieved documents — is often sent on every request while only a short user message changes. Processing that preamble every time is wasteful. Prompt caching lets the provider store a processed version of the prefix and reuse it.

Prompt structure for caching

// Anthropic Messages API
{
  "model": "claude-sonnet-4-5",
  "system": [
    {
      "type": "text",
      "text": "<long system prompt, role, rules, policies...>",
      "cache_control": {"type": "ephemeral"}
    },
    {
      "type": "text",
      "text": "<knowledge base, 50KB of docs>",
      "cache_control": {"type": "ephemeral"}
    }
  ],
  "messages": [
    {"role": "user", "content": "What's our refund policy for digital goods?"}
  ]
}

What to cache

• System prompts longer than ~1K tokens.
• Few-shot example blocks that don't change between requests.
• Retrieved documents that are reused across a session.
• Tool definitions and schemas.
• Large style guides, glossaries, or brand-voice references.

What NOT to cache

• The user's per-request message (defeats the purpose).
• Content under ~1K tokens (overhead > benefit).
• Personalized data that changes every request.
• Anything behind the cache's expiration window if requests are sparse.

Cache hierarchy

Caching works best when prefixes are layered in order from most-stable to least-stable. Put static rules first, then slowly-changing context (user profile, session info), then the per-request message last. That way the maximum possible prefix hits the cache.

[STABLE SYSTEM]       <- cache forever (or until product change)
[KNOWLEDGE BASE]      <- cache for the day
[USER PROFILE]        <- cache for the session
[CONVERSATION HISTORY]<- cache up to last turn
[CURRENT USER MSG]    <- never cache

Measuring impact

1. 1Baseline: log tokens-in / tokens-out / latency / cost per request for a representative day.
2. 2Add cache_control markers.
3. 3Re-measure. Compare per-request cost and P50/P95 latency.
4. 4Track cache hit rate (providers return it in responses). Aim for >80%.
5. 5Revisit when the preamble changes — cache misses spike after a prompt update.