Implicit vs Explicit Prompt Caching on GPT-5.6 Luna
When we moved MinuteShark AI onto GPT-5.6 Luna, the model swap was easy. The caching story was not.
We already shaped our prompts for reuse: a large static system block first, then stable per-user context, then a small dynamic tail that changes every turn. On earlier models, that layout mostly “just worked” with automatic prompt caching.
On Luna, the cache hits we expected stopped showing up the way we were used to.
Bottom line: On GPT-5.6, don’t assume old implicit partial-prefix caching still saves you money. For agent-style loops, mark the reusable prefix with explicit breakpoints — and measure write cost against read savings.
What We Assumed
If the first chunk of a request stays identical across turns, those tokens should get cached — even when the user’s latest message changes.
That assumption came from older automatic / implicit caching behavior:
- Stable prefix stays the same
- Changing suffix updates every turn
- Partial reuse still kicks in
For a product assistant with long instructions and tool calling, that pattern is everything. Most of the prompt is policy and structure. Only a little of it is “what the user said this second.”
What Luna Actually Does
On GPT-5.6, implicit caching still exists — but it behaves differently.
In implicit mode, OpenAI places a managed breakpoint near the latest message. It does not reliably give you the old “large stable prefix + changing question” partial-prefix hit.
So if your system instructions are huge and reusable, but the trailing chat content changes every turn (which it always does), you can look like you’re “using caching” while still reprocessing the expensive prefix.
The trap
Your prompt layout can be perfect for caching, and still miss the savings — because implicit mode is caching at the wrong boundary for agent workloads.
The Fix: Explicit Caching
For loops like ours — tool calling, multi-step confirmations, long instructions — we switched to explicit mode.
What we changed
- Set
prompt_cache_options: { mode: "explicit", ttl: "30m" } - Put a
prompt_cache_breakpointat the end of the reusable prefix - Keep a consistent
prompt_cache_keyso related requests can find the same cache - Leave changing chat / session / dynamic context after the breakpoint
In plain English: tell the API exactly where the reusable part ends.
Once we did that, cache reads became predictable again. The model still saw the full prompt. We just stopped paying full price for the static front matter on every turn.
The Other Gotcha: Cache Writes Aren’t Free
On GPT-5.6 models, cache writes are billed at 1.25× the normal input rate. Cache reads are still discounted.
So “turn caching on everywhere” is not free optimization. Bad breakpoint placement can mean:
- Writing too often
- Writing prefixes that rarely get reused
- Paying a write premium without enough later hits to earn it back
Explicit mode helps here too: you cache only what you intend to reuse.
How We Split the Prompt
MinuteShark AI never lets the model write workspace data directly — it proposes actions and waits for confirmation. That same discipline applies to caching.
Cacheable
Static rules, tools, product policy, and requester identity / workspace permissions that stay stable in a session.
After the breakpoint
Current chat, live stats, today/now context, and anything that churns from turn to turn.
We also started treating cached_tokens and cache_write_tokens as first-class telemetry — not nice-to-have logs. Without those numbers, “we have prompt caching” is just a belief.
Takeaway
If you’re building a production assistant on GPT-5.6 Luna:
- Don’t assume old implicit partial-prefix behavior still saves you money
- Shape the prompt so the stable part is truly stable
- Use explicit breakpoints for agent loops
- Measure write cost against read savings
Model choice still matters for speed and price. But for us, the bigger cost lesson was simpler:
Caching only helps when you control the cache boundary.