A context cache reuses the stable parts of a prompt across many model calls so you stop paying, in tokens and in latency, to re-send the same context every time. On high-volume generation workloads, that repeated context is often the bulk of what you send, and resending it is pure waste. CoreReflex exposes context caching as a first-class platform primitive so repeated, large-context jobs run faster and cost less, without changing the output.
What the context cache does
Most serious generation prompts have two parts: a large, stable block, system instructions, brand guidelines, a knowledge-base excerpt, a style guide, a long set of few-shot examples, and a small, variable block, the actual per-call request. Without caching, you pay to process that entire stable block on every single call, even though it never changes.
The context cache stores the processed stable block once and references it on subsequent calls. Instead of re-sending and re-processing tens of thousands of tokens of unchanging context for each request, you send the cache reference plus the small variable part. The model behaves identically, same grounding, same instructions, same output quality, but the cost and latency of the repeated context drop sharply because the heavy lifting happened once.
Two benefits follow directly:
- Lower token cost. You are billed for the cached context at a reduced rate rather than full price on every call.
- Lower latency, the model skips re-ingesting the stable block, so time-to-first-token shrinks on every cached request.
When context caching pays off
Caching is not free to set up. There is overhead in writing and maintaining the cache, so it pays off in proportion to how often you reuse the same context. The rule of thumb: the cache wins when the stable block is large and reused many times within its lifetime.
Clear winners include:
- Batch generation over a shared brief, generating dozens of variants against one brand kit, style guide, or product spec.
- High-volume API workloads where a system prompt and grounding context repeat across thousands of calls.
- RAG pipelines where the same retrieved knowledge base context anchors many related generations, closely related to how the knowledge base API grounds every generation.
- Scheduled, hands-off runs where an Autopilot recipe fires the same context-heavy prompt on a cadence, see how Autopilot schedules hands-off generation.
Where caching does not pay off is one-off calls, or workloads where the context changes on every request. If there is nothing stable to reuse, there is nothing to cache. The honest answer is to enable it where context is both large and repeated, and skip it where it is not.
How it fits the CoreReflex platform
The context cache is one primitive in a programmable platform, not a standalone trick. CoreReflex exposes a JSON engine over editor state, a public API with API keys, an SDK, and an MCP server, alongside RAG, eval, batch, and the cache. These compose: you ground a generation with the knowledge base, cache the stable grounding context, run the job in batch, and score the results with eval, all on one owned stack on Google Vertex AI.
Because the whole platform runs on Vertex, caching applies across the same models that power the rest of the studio. Gemini for reasoning and text, and the generation models behind video, image, and audio. You are not bolting a cache onto a third-party API you do not control; it is part of the same owned pipeline. If you are wiring this from scratch, the API quickstart for developers is the place to start, and the broader developer guides cover the surrounding primitives.
Caching and Workflow recipes
Context caching is especially powerful inside Workflow recipes, where a pipeline runs the same stage repeatedly with shared instructions and governance around it. Caching the stable stage context cuts the marginal cost of each run, which compounds when a recipe processes many items. The relationship between caching and the governance layer is worth understanding alongside Workflow recipes with quality and budget gates, the cache lowers cost per run while budget gates cap total spend.
Caching and eval
When you score outputs programmatically, the eval rubric and reference context are themselves stable across many scored items. Caching that rubric context makes large eval runs cheaper, which matters when you are scoring at volume. The mechanics of programmatic scoring are covered in the eval API guide; the cache is simply the cost lever that makes running it at scale practical.
A practical setup pattern
The pattern for adopting context caching cleanly is straightforward:
- Identify the stable block. Separate what repeats (instructions, brand kit, grounding, examples) from what varies per call (the specific request).
- Confirm the reuse count. Estimate how many calls will share that block within its lifetime. If it is many, caching wins; if it is a handful, it may not be worth the overhead.
- Cache the stable block and reference it on each call, sending only the variable part as the live prompt.
- Measure. Compare latency and token cost before and after on a representative batch. The savings should track the size and reuse of the cached block.
- Set the lifetime sensibly. Keep the cache alive across the burst of related calls, and let it expire when the context goes stale so you are never grounding on outdated material.
The key discipline is keeping the cached context stable. If you find yourself editing the cached block frequently, it probably belongs in the variable part instead.
What you get
Used well, the context cache turns high-volume generation from something you ration into something you run freely. The same brief, brand kit, or knowledge base that anchors a thousand generations is processed once, not a thousand times, so latency and token cost fall without touching output quality. That is the end-state the platform is built for: the full agentic studio, video, image, voice, and the rest, exposed as composable, programmable primitives you can run at scale on a stack CoreReflex owns end to end. Caching is the lever that keeps that scale affordable.
Frequently asked questions
What does the context cache do?
It stores the large, stable portion of a prompt, system instructions, brand guidelines, grounding context, few-shot examples, so it is processed once and referenced on subsequent calls instead of re-sent every time. The model still sees the full context and produces identical-quality output, but you pay a reduced rate on the cached portion and skip the latency of re-ingesting it. The benefit scales with how large the stable block is and how often it is reused.
When should I enable context caching?
Enable it when a large block of context is reused across many calls within a short window, batch generation against a shared brief, high-volume API workloads with a repeated system prompt, RAG pipelines anchored to the same knowledge base, or scheduled Autopilot runs. Skip it for one-off calls or workloads where the context changes on every request, since there is nothing stable to reuse and the setup overhead would not pay off.
Does caching change the output?
No. The cache only changes how the stable context is delivered to the model, not what the model sees. The full context is still applied to every call, so grounding, instructions, and output quality are unchanged, caching is purely a cost and latency optimization, not a quality trade-off.
How does the cache work with other platform primitives?
It composes with them. You ground a generation with the knowledge base, cache the stable grounding so repeated calls are cheap, run the job through batch, score results with eval, and orchestrate the whole thing through the JSON engine, API, SDK, or MCP. Because everything runs on the same owned Vertex AI stack, caching applies consistently across the models behind text, video, image, and audio.
Build it on the platform
The context cache is how high-volume, context-heavy generation stays fast and affordable: process the stable block once, reference it everywhere, and pay full freight only on what actually changes. It is one primitive in a programmable platform. JSON engine, API, SDK, MCP, RAG, eval, and batch, that exposes the whole agentic studio to your code. Get an API key and start building, start free, no credit card required.