An eval API lets you score generated outputs programmatically, running each result against your own checks, reading back a pass or fail, and gating a release before anything reaches a customer. For engineering teams building on top of generative pipelines, that's the difference between shipping on vibes and shipping on evidence. CoreReflex exposes Eval as part of its platform so the same rigor that scores a film inside the editor is available to your code, your CI, and your agents.
What the eval API is
Eval is a programmable scoring layer, you define what "good" means for your use case as a set of checks, point the eval at a batch of generations, and get back structured results, a score per check, an overall pass or fail, and the trace behind each decision. Because it's an API rather than a screen, you can run it anywhere your other automation runs: a backend service, a scheduled job, a pull-request check, or an agent loop.
The key idea is that scoring becomes data. Instead of a human eyeballing outputs and forming an opinion, you get machine-readable verdicts you can branch on, log, alert on, and trend over time, that's what makes quality a property you can enforce rather than a hope you carry into a launch. Eval sits alongside the rest of the platform's programmable surface, the JSON engine, public API, SDK, and MCP, which our API quickstart for developers introduces end to end.
Eval vs the per-shot quality gate
This is the most common point of confusion, so it's worth being precise. The per-shot quality gate runs inside the Director loop during the CRITIQUE stage: as each shot is generated, it's scored on prompt match, sharpness, motion coherence, on-screen text legibility, on-brand, and claims risk, and failures regenerate selectively. The gate's job is to keep a single film's shots above the bar while it's being made.
The eval API operates at a different altitude. It scores outputs, finished generations, batches, or whole workflow runs, against checks you define, after or independent of any single film's production. Think of the quality gate as the inline reviewer that protects one cut, and Eval as the test suite you run across many outputs to catch regressions, compare configurations, and decide whether a change is safe to ship.
| Per-shot quality gate | Eval API | |
|---|---|---|
| When it runs | Inline, during CRITIQUE | On demand, in batch or CI |
| Scope | One film's shots | Many outputs across runs |
| On failure | Selective regeneration | Surface the result, gate the release |
| Who calls it | The Director loop | Your code, CI, or an agent |
The two are complementary. The gate keeps individual films clean; Eval keeps your pipeline honest as prompts, models, and recipes change underneath it.
What you can score
Eval isn't limited to one media type. Because CoreReflex owns its stack on Google Vertex AI. Gemini, Veo, Kling, Lyria, Imagen, embeddings, STT, and TTS, the same scoring layer reaches across the things those models produce:
- Video, does the cut match the brief, hold motion coherence, and stay on-brand across shots.
- Images, sharpness, prompt fidelity, and brand alignment for stills from Image Studio.
- Copy, does written output respect the brand-voice guard and avoid risky claims.
- Voice, does narration match the intended voice and read cleanly.
You decide which checks apply to which outputs. A claims-risk check matters enormously for an ad; a motion-coherence check is irrelevant to a static image. Defining the right checks per surface is the work that makes an eval meaningful rather than ceremonial.
A typical eval workflow
Here's the shape of a real integration, from definition to gate.
- Define your checks. Encode what "good" means for the output type, the dimensions you care about and the threshold each must clear.
- Assemble a set to score. Point the eval at fresh generations, a stored batch, or the outputs of a workflow run.
- Run the eval. Call the API and let it score every item against your checks. Scoring leans on the same free planning-and-scoring economics as the rest of the platform, only generation costs credits, so you can evaluate liberally.
- Read structured results. Get back a per-check score, an overall verdict, and the provenance behind each item: model, prompt, parameters, and score.
- Branch on the verdict. Accept, regenerate, flag for human review, or fail the build, your logic decides.
This loop mirrors how you'd wire any generation into your product, which we cover in adding AI video to your app with the API and the more film-specific AI video API: generate film from one sentence.
Gating releases and catching regressions
The highest-value use of an eval API is as a gate. Wire it into CI and a pull request that changes a prompt template, swaps a model, or edits a workflow recipe has to clear the eval before it merges. If the change quietly degrades quality on a dimension you care about, the eval catches it as a failing check instead of a customer catching it in production.
Run the same eval on a schedule and you get regression detection over time. Because each result carries a provenance trace, a drop in scores isn't a mystery, you can see which model and parameters produced the regression and reproduce it exactly to debug. That auditability is what separates a real test suite from a one-time spot check, and it's why teams weighing whether to assemble their own pipeline often land here, a trade-off we examine in build your own Veo pipeline vs CoreReflex.
Where eval fits in the platform
Eval is one capability in a larger programmable surface, and it gets more powerful next to the others. Pair it with batch to score large sets in one pass. Use RAG and the knowledge base to ground checks in your own reference material. Combine it with workflow recipes, which carry budget and checkpoint gates, so an automated run can stop or pause on a failing eval rather than barreling ahead, the pattern in workflow recipes with quality and budget gates. And put it downstream of Autopilot, so scheduled, hands-off generation is continuously scored rather than trusted blindly, as covered in Autopilot: schedule hands-off generation. For the full reference and the rest of the developer surface, see the developer documentation and the broader developers library.
Frequently asked questions
Can I evaluate generations programmatically?
Yes. The eval API lets you define checks, point them at a set of generations, run the scoring, and read back structured per-check scores and an overall verdict. Because it's an API, you can run it from a backend service, a scheduled job, a CI pipeline, or an agent, and branch your logic on the result.
How does eval differ from the per-shot quality gate?
The per-shot quality gate runs inline during the Director's CRITIQUE stage to keep one film's shots above the bar, regenerating failures selectively. The eval API runs on demand or in CI across many outputs to catch regressions, compare configurations, and gate releases. The gate protects an individual cut; Eval protects your pipeline as it changes.
What can I score with the eval API?
You can score any output the platform produces, video, images, copy, and voice, against checks you define for that surface. You choose which checks apply where, so a claims-risk check can gate ad copy while a motion-coherence check gates video, and a brand-voice check gates written content.
Does running an eval cost credits?
Scoring and planning operations follow the same economics as the rest of the platform: they're free, and only generation consumes credits. That means you can evaluate liberally, across large batches and on every pull request, without the cost of the evaluation itself becoming the bottleneck.
Are eval results reproducible?
Yes. Every generation carries a portable provenance trace recording the model, prompt, parameters, and score, so a passing or failing eval result is auditable and replayable. When scores regress, you can see exactly which configuration caused it and reproduce that state to debug.
Ship on evidence, not vibes
An eval API turns quality from a gut feeling into a gate: define your checks, score every output, catch regressions before they ship, and keep a trace you can replay. That's exactly what Eval gives you inside CoreReflex's owned, programmable stack, the same rigor the Director applies to a film, available to your code. Start free with no credit card and put your first eval in the loop.