Blog

GPT prompt caching: how to improve cache hit rates

· the plori team

The short answer: treat prompt caching as a request-shape problem. A cache key does not make two prompts equivalent. The model can reuse only an exact prefix that a prior request wrote at an eligible breakpoint. Put stable instructions, tools, schemas, and durable history first. Put timestamps, request IDs, retrieved state, and other changing material after the reusable prefix. For GPT-5.6 and later, use an explicit breakpoint when the default breakpoint would include a volatile tail.

That advice sounds obvious. It is still easy to get wrong in a multi-turn agent because the request can look append-only at the message level while its cached entry ends with a block that changes every round. We found this in a real workload, reduced it to a controlled probe, and measured the request shape before and after the fix.

A real before-and-after

The production trace came from one long-running agent session on GPT-5.6 Luna, served through OpenRouter. Product, account, session, tool, infrastructure, and prompt content have been removed. Token counts and billing records are unchanged, except that the prompt range is rounded to the nearest thousand.

Production measurement Observed value
Consecutive model rounds 39
Prompt size 401K to 515K tokens
Gap between some rounds about 10 seconds
Cached tokens per round usually 3,467 or 0
Cache hit rate 0% to 1%
Cache writes approximately the full prompt each round
Representative 437K-token round about $0.218

The prompt's system instructions and tool definitions were byte-stable across rounds. The cache was warm, the requests stayed on the same provider, and the context window was not full. Yet nearly every round paid to write hundreds of thousands of tokens again.

We then replayed synthetic content against the same live model endpoint and varied one property at a time. These were billed API calls, not mocked tests.

Probe Request shape Result
Broken shape Same key, but the new request diverged before the end of the prior cached entry 1,215 cached tokens, only the common head stub
Optimized shape Same key and a strict extension of the broken-shape request 45,786 / 45,805 cached tokens, 99.96%
Repeat control Another strict extension of its seed request 45,731 / 45,753 cached tokens, 99.95%; reported request cost was about one-twelfth of the cold predecessor
Key control Byte-identical prompt with a different prompt_cache_key 0 cached tokens

The optimized shape returned 37.7 times as many cached tokens as the broken replay: 45,786 instead of 1,215. That is a cached-token comparison, not a claim that every bill will fall by 37.7 times. Output tokens, images, uncached suffixes, gateway pricing, and long-context price tiers still apply.

The production trace and the controlled replay answer different questions. The 39-round trace shows that the failure mattered at realistic context sizes. The smaller replay isolates exact-prefix matching without spending dollars on every probe. We will not call the large production workload fixed until the optimized request shape has gone through its normal deployment and the same production metrics have been checked again.

What the cache actually matches

For operational purposes, treat a prompt-cache lookup as depending on all of these:

model and provider route
+ prompt_cache_key
+ exact rendered bytes from the start of the request through a cache breakpoint
+ a live cache entry

The key helps route related requests toward the same cache. It does not turn a fuzzy match into an exact one. A single changed tool description, reordered schema field, timestamp, image setting, or message before the breakpoint can make the entry unusable.

The prefix starts earlier than many teams expect. Tool definitions and structured-output schemas can contribute to the rendered prefix, as can images, files, and request settings. Stable message text is not enough if the tool array changes order on every request.

GPT-5.6 changed the failure mode

OpenAI's prompt-caching guide draws a useful line between GPT-5.6 and earlier models.

Behavior GPT-5.6 and later Earlier cache-capable GPT models
Match behavior Exact matching at eligible breakpoints Automatic best-effort reuse of matching prefixes
Default breakpoint Latest user or tool message Managed automatically
Fallback to an earlier unmarked prefix No Best-effort matching may reuse an earlier prefix
Explicit breakpoints Supported Not supported
Minimum cacheable prefix 1,024 tokens 1,024 to 2,048 tokens, model-dependent
Cache write rate 1.25 times uncached input No separate write rate

On GPT-5.6, the default implicit breakpoint does not search backward for the longest unmarked match. If the latest user message contains a fresh timestamp or a regenerated runtime block, the service may write that changing prefix again even though most of the request is stable.

The transient-tail trap in agent loops

Suppose an agent assembles each round like this:

Round 1: durable history A -> transient runtime T1 [implicit breakpoint]
Round 2: durable history A+B -> transient runtime T2 [implicit breakpoint]

T1 is not part of round 2, so round 1's complete cached entry is not a prefix of round 2. The stable A portion exists inside both requests, but there is no eligible cached entry ending there. A stable cache key cannot repair that shape.

For this request shape, use explicit-only caching when the suffix is genuinely disposable:

Round 1: durable A [current breakpoint] -> transient T1
Round 2: durable A [previous breakpoint] -> durable B [current breakpoint] -> transient T2

The previous marker gives round 2 an entry it can read. The current marker writes the extended durable prefix for round 3. The changing tail stays outside both entries.

This detail matters: do not keep only a moving current breakpoint. On round 2, that new position has not been written by any earlier request. Retain the previous durable boundary for the read and add the current boundary for the next write. On the first round there is no previous boundary, so mark the current one to seed the cache.

Choose the caching mode from the request shape

Use implicit caching when a conversation grows by appending content that future requests will repeat. It is the simplest option and may both read an earlier prefix and write a new checkpoint.

Add an explicit breakpoint while leaving implicit mode on when you want a stable shared prefix and also expect the latest conversation history to be reused. Be aware that the implicit breakpoint can still write the changing suffix.

Use explicit-only mode when stable content is followed by request-specific content that should not be cached. Set prompt_cache_options.mode to explicit and add at least one valid prompt_cache_breakpoint. Explicit mode with no marker disables prompt caching for that request.

Here is the smallest useful Chat Completions shape for GPT-5.6:

{
  "model": "gpt-5.6",
  "prompt_cache_key": "conversation:7f3c...",
  "prompt_cache_options": {
    "mode": "explicit",
    "ttl": "30m"
  },
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "Stable instructions and reference material...",
          "prompt_cache_breakpoint": { "mode": "explicit" }
        }
      ]
    },
    {
      "role": "user",
      "content": "Request-specific input..."
    }
  ]
}

The content through the marker must render to at least 1,024 tokens. In the Responses API, put the marker on a supported input_text, input_image, or input_file block. Top-level instructions cannot carry a breakpoint, so reusable developer instructions need to live in a developer message when you want to mark them.

If you use a gateway, verify that it passes these fields to the provider. The OpenRouter caching guide documents prompt_cache_key, prompt_cache_options, and block-level breakpoints for both Chat Completions and Responses. It also documents session_id for provider stickiness. Other gateways may translate, drop, or reject the same fields.

Build the prefix deliberately

A reliable request usually has this order:

  1. Stable system or developer instructions.
  2. Tool definitions in a deterministic order, with stable JSON schemas.
  3. Shared examples, reference files, or images with identical settings.
  4. Durable conversation history that later rounds reproduce byte-for-byte.
  5. The cache breakpoint.
  6. Timestamps, request IDs, live runtime state, newly retrieved documents, and other request-specific material.

Do not sort only the top-level tools and assume the job is done. Canonicalize any map or set that feeds the rendered request. Keep tool descriptions and schema defaults stable. Do not insert a timestamp into a system prompt. If a value changes every call, it belongs after the breakpoint unless the model truly needs it earlier.

Append-only history has another trap: compaction and truncation change the prefix. Treat the first request after either operation as a cold start. Give the compacted history a new stable cache family rather than pretending it extends the old bytes.

Choosing and versioning cache keys

Use one deterministic key for requests that should share a prefix. A session-scoped hash or a versioned workload-family hash is usually enough. Do not put raw account IDs, email addresses, prompt text, or other sensitive values in the key.

Changing the key is an intentional cache reset. That is useful after a prompt template, tool schema, or compaction version changes. It is expensive when a random UUID is minted for every round.

OpenAI recommends keeping traffic for each key to roughly 15 requests per minute. If a shared prefix serves more traffic, partition it with a stable bucket suffix. Random partitioning preserves neither routing nor reuse.

When a gateway can move traffic between providers, cache affinity has another layer. Pin the conversation to a provider using the gateway's supported session mechanism, then use the provider cache key inside that route. A perfect prefix cannot hit an entry on a different provider.

Measure reads and writes separately

For Chat Completions, record these values from usage.prompt_tokens_details:

{
  "cached_tokens": 45786,
  "cache_write_tokens": 19
}

For Responses, the same fields live under usage.input_tokens_details. Also record total input tokens, model, a hashed cache-key bucket, prompt-template version, and hashes of the system, tools, and durable-prefix segments. Hashes let you find drift without logging raw prompts.

Read the pair, not one number:

Pattern Likely meaning
High reads, small writes Healthy append-only extension
Low reads, near-full writes Breakpoint includes changing content, wrong key, cold entry, or provider drift
Reads and writes both zero Below the minimum, explicit mode without a valid marker, or unsupported fields
Reads and writes both nonzero An earlier prefix was read and a later prefix was written; normal in implicit mode
Hits fall only at high volume Too much traffic on one key or loss of provider affinity

Do not report only "cache hit requests." A request that reads 3,467 tokens from a 500,000-token prompt technically hit the cache, but it reused less than 1% of the input. Track token-weighted hit rate:

sum(cached_tokens) / sum(input_tokens)

For GPT-5.6, also track write amplification:

sum(cache_write_tokens) / sum(input_tokens)

The worst pattern is a low read ratio next to a high write ratio. It means the system is paying the cache-write premium without collecting the read discount.

Debug with a five-request probe

Before changing production prompts, run a small controlled sequence against the exact model and provider path you use:

  1. Send prefix A with one stable key. Expect a write.
  2. Send A again with the same key. Expect a read.
  3. Send A plus a new suffix with the same key. Expect A to be read.
  4. Send a strict append-only extension and check whether the new boundary is written.
  5. Repeat identical bytes with a different key. Expect no read from the first key.

Keep the requests above the model's minimum prefix length and within the TTL. Change one variable per probe. This separates prefix behavior from provider routing, key scoping, expiry, and application-level serialization.

If step 2 misses, compare the serialized bytes before the breakpoint. If the bytes match, inspect model selection, cache key, provider affinity, TTL, and whether the gateway forwarded the cache fields. If step 2 hits but step 3 misses, the breakpoint is probably later than the stable prefix you intended.

Separate reads, writes, and uncached input in the cost math

For GPT-5.6 and later, OpenAI currently bills cached reads at 0.1 times the uncached input rate and cache writes at 1.25 times that rate. The 1.25 multiplier is the total rate for written tokens, not an extra 1.25 charge added on top of a full input charge. For the same number of tokens, a write therefore costs 12.5 times a read.

That ratio explains why repeated full-prefix writes hurt so much. It does not predict the whole request bill. Calculate each segment separately:

input cost =
    cached_read_tokens * cached_read_rate
  + cache_write_tokens * cache_write_rate
  + ordinary_uncached_tokens * uncached_input_rate

Then add output, images, tools, gateway fees, and any long-context pricing tier. Use the rates from the endpoint that actually served the request. In our real 437K-token example, the provider's long-context rate, not the short-context list price, was needed to explain the observed $0.218 round.

What prompt caching does not solve

Prompt caching does not make outputs deterministic and does not remove input tokens from rate-limit accounting. It does not guarantee that an entry lives forever. GPT-5.6's documented TTL is 30 minutes and refreshes on reuse, but an application should tolerate a cold miss at any time.

It also cannot rescue a request whose early bytes genuinely change. If every request has different tools, instructions, or reference material, the correct cacheable prefix may be small. Do not distort the prompt until it is wrong just to raise a metric.

The practical target is not 100% on every call. It is a stable, explainable boundary: cold writes when a prefix is first created or deliberately versioned, large reads while that prefix is reused, and small writes when durable history grows. When the graph shows full writes beside tiny reads, inspect the request bytes before tuning anything else.

Sources and measurement note

  • OpenAI prompt caching guide, checked 2026-08-22 for GPT-5.6 matching, breakpoints, pricing ratios, TTL, metrics, and troubleshooting guidance.
  • OpenRouter prompt caching guide, checked 2026-08-22 for gateway field support, cache metrics, and session stickiness.
  • The before-and-after figures are from 2026-08-22 production traces and paid controlled probes. Identifiers and prompt content were removed; token and cost measurements were retained as described above. No publishable latency measurement was collected, so this article makes no latency-improvement claim.