# The AGENT Kernel

The suite-wide conversational agent: a natural-language request becomes a validated, cost-estimated, multi-step plan over the real JSON command engine — and nothing executes until a human approves it.

**On this page**

- [Overview](#overview)
- [Architecture](#architecture)
- [Key files and seams](#key-files-and-seams)
- [Phase 1 — PLAN (`interpret`)](#phase-1--plan-interpret)
  - [Reasoners: tool-use loop, one-shot, mock](#reasoners-tool-use-loop-one-shot-mock)
  - [Step taxonomy (`classifyStep`)](#step-taxonomy-classifystep)
  - [Normalization and the untrusted-input choke point](#normalization-and-the-untrusted-input-choke-point)
  - [Validation (`validateAgentPlan`)](#validation-validateagentplan)
  - [Reference resolution (`resolveReferences`)](#reference-resolution-resolvereferences)
  - [Feasibility gating (`planFeasibilityErrors`)](#feasibility-gating-planfeasibilityerrors)
  - [Clarifying questions](#clarifying-questions)
- [The tool registry](#the-tool-registry)
- [Phase 2 — EXECUTE (`runAgentPlan`)](#phase-2--execute-runagentplan)
  - [Atomic engine batches (`compileEngineBatch`)](#atomic-engine-batches-compileenginebatch)
  - [Editor-side apply (undoable runs)](#editor-side-apply-undoable-runs)
  - [Generative step dispatch](#generative-step-dispatch)
  - [Asset-flow chaining (`@asset:step-N`)](#asset-flow-chaining-assetstep-n)
  - [Typed failure recovery](#typed-failure-recovery)
  - [Dry run and the approval-card preview](#dry-run-and-the-approval-card-preview)
- [Governance](#governance)
  - [Approval policies and the hard confirm floor](#approval-policies-and-the-hard-confirm-floor)
  - [Policy tiers (tool-class allowlists)](#policy-tiers-tool-class-allowlists)
  - [Input redaction and moderation](#input-redaction-and-moderation)
  - [Budget: pre-flight check and run ledger](#budget-pre-flight-check-and-run-ledger)
- [Audit and resume: `agent_runs`](#audit-and-resume-agent_runs)
- [Observability](#observability)
  - [Live progress over SSE](#live-progress-over-sse)
  - [Provenance traces](#provenance-traces)
  - [The eval harness](#the-eval-harness)
- [Adjacent verbs](#adjacent-verbs)
- [HTTP surface](#http-surface)
- [Configuration](#configuration)
- [Testing](#testing)
- [Gotchas](#gotchas)
- [History: why this is in-house](#history-why-this-is-in-house)
- [Related pages](#related-pages)

---

## Overview

The AGENT kernel (`src/agent.js`) is the layer that unifies the [JSON command engine](json-engine.md), the Director film pipeline, recipes, and the cross-studio generation verbs behind one conversational surface. A user types "delete the last clip, make it vertical, and add a title card" — the kernel turns that into a typed `AgentPlan`, shows the user exactly what it will change and what it will cost, and only then executes it atomically.

Two invariants hold everywhere, by construction:

1. **Two-phase.** `interpret()` PLANS with zero side effects. `runAgentPlan()` EXECUTES, only after the caller approves. There is no code path where planning mutates state or spends credits — during planning only pure reads (`query`/`describe`) ever run.
2. **The engine is the single mutation authority.** Every timeline edit the agent makes is an engine command compiled into one atomic batch. The agent never invents its own edit primitives, so an agent edit is exactly as validated, undoable, and testable as a human edit.

The planning model is Gemini on Vertex AI (a Google integration we orchestrate — see [generation-and-models.md](generation-and-models.md)). A deterministic mock reasoner (`mockAgentPlan`) mirrors the Director's `mockReason` pattern, so the whole kernel plans, validates, and executes offline in tests without credentials.

## Architecture

```
             ┌──────────────────────── PHASE 1: PLAN (no side effects) ───────────────────────┐
user message │                                                                                │
     ──────► │ redactSecrets ─► reasoner (Gemini tool-use loop │ one-shot │ mock)             │
             │        ▲                │ reads (query/describe) run PURE to ground the model  │
             │        │                ▼                                                      │
 grounding   │ assembleAgentContext   normalizeAgentPlan ─► resolveReferences (@last → ids)   │
 (KB, brand, │ (agent-context.js)     ─► validateAgentPlan (+ policy + feasibility + caps)    │
 credits,    │                        ─► planConfirmations + estimatePlanCost + dryRun        │
 transcript) │                                                                                │
             └─────────────────────────────► { plan, valid, errors, preview, confirmations } ─┘
                                                          │  human approves (+ confirms)
             ┌──────────────────────── PHASE 2: EXECUTE ──▼───────────────────────────────────┐
             │ runAgentPlan: re-validate ─► approval gate ─► assertConfirmed ─► assertAffordable
             │   engine steps  ─► compileEngineBatch ─► ONE atomic batch (rolls back wholesale)
             │   generate steps ─► meter credits ─► dispatch seam (image/music/voice/recipe)  │
             │   director steps ─► hand off to the durable film pipeline (background runId)   │
             │   deferred verbs ─► reported `deferred` (durable executors are follow-ons)     │
             │   @asset:step-N  ─► inject addAsset into the placing step's batch              │
             │ progress ─► SSE (realtime-bus)   audit ─► agent_runs (migration 032)           │
             └────────────────────────────────────────────────────────────────────────────────┘
```

Execution has two runners with identical safety semantics:

- `runAgentPlan(user, plan, options)` — the full executor: engine batches, generative dispatch, budget ledger, retry recovery, progress events. The `/api/agent/execute` route uses it for both stored compositions (engine batches persist through the row-locked `evaluateComposition` path via an injected `runBatch`) and inline manifests.
- `runAgentPlanInline(plan, state, options)` — engine-only synchronous apply; generative steps report `pending`. Kept for callers that only need deterministic edits.

## Key files and seams

| Path | Role |
|---|---|
| `src/agent.js` | The kernel: `interpret`, `runAgentPlan`, validation, classification, confirmations, dry-run, recovery, `@asset` chaining |
| `src/agent-tools.js` | Canonical tool registry (`buildToolRegistry`, `toFunctionDeclarations`, `functionToolMap`) |
| `src/agent-runs.js` | `agent_runs` audit ledger: start/finish/fail, list/get, transcript, credit SUM |
| `src/agent-context.js` | Planner grounding bundle (composition summary, brand voice, KB, credits, accounts, transcript) |
| `src/agent-ask.js` | Read-only factual Q&A (`askProject`), no plan round-trip |
| `src/agent-recipe.js` | `planToRecipe` — an approved plan becomes a reusable `{{slot}}` recipe |
| `src/agent-eval.js` | Plan-quality eval harness (fixture cases + rubric scoring) |
| `src/budget.js` | `createLedger` (reserve → reconcile → refund), `estimateRecipe` |
| `src/checkpoints.js` | `CHECKPOINT_POLICIES` (`guided`/`manual`/`auto`) + `stageNeedsApproval` — the approval vocabulary the agent reuses |
| `src/inference.js` | `newTrace`/`runTraced` — the portable provenance envelope |
| `src/chat-safety.js` | `redactSecrets` + `moderateInput` (injection/exfil flags) |
| `src/json-engine.js` | `describeSchema()` / `evaluate()` — the command engine the agent plans over |
| `src/api.js` | The `/api/agent/*` routes (search for `api/agent`) |
| `src/voice-agent/tools.js` | `agentToolHandlers` — voice `plan_creative_edit` (plan-only) |
| `migrations/032_agent_runs.sql` | The `agent_runs` table |
| `editor/src/editor/agent/` | Editor Agent panel, SSE activity bridge, undoable apply (`apply-run.ts`, `use-apply-agent-run.ts`) |
| `editor/src/editor/json-api/` | The editor-side engine mirror (commands, `evaluate`, reducer-parity test) |

## Phase 1 — PLAN (`interpret`)

`interpret(message, context, options)` in `src/agent.js` is the front door. It:

1. Redacts secret-shaped substrings from the message (`redactSecrets`) **before** the model ever sees it.
2. Picks a reasoner: the Gemini tool-use loop when Vertex is configured, else the deterministic mock. A reasoner throw degrades to the mock and sets `plan.degraded` — the front door never hard-fails.
3. Normalizes the raw plan (`normalizeAgentPlan`), binds fuzzy reference tokens against the live state (`resolveReferences`), computes `estCredits` and `confirmations`, and validates.
4. Returns `{ plan, valid, errors, degraded, safety, confirmations }` — or a clarification result (see below) when the request is ambiguous.

### Reasoners: tool-use loop, one-shot, mock

- **`geminiAgentToolUse`** (the default when Vertex is up, AGENT-003): declares every registry verb as a typed function tool (`toFunctionDeclarations`) and lets Gemini compose the plan by calling tools by name, up to `AGENT_MAX_TOOL_TURNS` (default 6) turns. Only side-effect-free reads (`query`/`describe`) actually execute during planning — `runReadForPlanning` runs them against the supplied state so the model grounds on real rows (capped at 50 rows per read). Every mutating/generative call is *recorded* as a plan step and answered with a synthetic `{ queued: true }` so the model keeps composing while nothing runs. Tool names are provider-sanitized (`generate.image` → `generate_image`) and mapped back via `functionToolMap`. If the model never calls a tool, it falls back to the one-shot planner.
- **`geminiAgentPlan`** (one-shot): a single structured-output call with `responseSchema: AGENT_PLAN_SCHEMA` — `{ summary, steps: [{ kind, tool, params, targetStudio, rationale }] }`. The system prompt lists every engine command live from `describeSchema()` plus the canonical keyframe-animation recipes (fade-in, slide-in, pop, pulse, ken-burns) so the model composes real motion vocabulary.
- **`mockAgentPlan`** (deterministic, offline): keyword → multi-step plan over the engine vocabulary (`mute`, `delete … last/first/longest`, `color grade`, `silence`, `caption`, "N vertical shorts", `publish to <platform>`, `vertical/9:16`). Anything unrecognized becomes a safe read-only `query` — the mock never invents a mutation it can't ground. This is what makes the whole kernel key-free testable.

Both live reasoners attach a replayable provenance trace as `raw._trace` (see [Provenance traces](#provenance-traces)).

### Step taxonomy (`classifyStep`)

Every step is classified into one of three classes; the class drives approval, policy tiers, and budgeting:

| Class | Meaning | Examples |
|---|---|---|
| `read` | Never mutates | `query`, `describe`, `kind:'studio'` (a UI switch hint) |
| `mutate` | Deterministic engine edit — free, undoable | `updateItem`, `deleteItems`, `setCompositionSize` (any of the 47 engine commands) |
| `generate` | Spends model credits and/or leaves the platform — irreversible | `generate.image/music/voice`, `director`, `recipe`, and the deferred verbs (`upscale`, `publish`, `create.short`, `auto.reframe`, `cut.silences`, `add.captions`) |

Unknown tools classify as `mutate` (gated) and are then rejected by validation. `step.reversible` is `class !== 'generate'`: engine edits undo; spends don't.

Costing: `estimateStepCredits` prices `generate` steps from `STEP_COST` (single-sourced against `billing.USAGE_COSTS`); `estimatePlanCost(plan)` returns a `{ low, expected, high, uncertain }` range for the approval card — a `director` step spans 1–8 shots, a `recipe` fans out to a ~4x ceiling, both flag `uncertain: true`. `committedCredits(resultSteps)` is the honest post-run figure (a metered-but-failed step still counts, because the meter fires before the seam).

### Normalization and the untrusted-input choke point

`normalizeAgentPlan(raw, message)` is the single choke point both `interpret` **and** the untrusted `/api/agent/execute` path pass through. It assigns stable step ids (`step-1`…), infers `kind` from the tool, coerces `params` to an object, clamps `targetStudio` to a known studio, and — critically — applies safety at rest:

- The message and summary are secret-redacted and length-capped.
- Free-text `rationale` is redacted (a forged `/execute` plan could smuggle a secret into `agent_runs` otherwise). Structured `params` are **not** blanket-redacted — the model only ever sees the already-redacted message, so it cannot echo a real secret into a param, and redacting params would corrupt legitimate creative text.
- `moderateInput(message)` attaches `plan.safety = { flags, action }` (advisory `prompt_injection` / `credential_probe` flags surfaced on the approval card).
- A `kind:'studio'` claim on a step whose tool is a real engine op is stripped — a studio marker can never disguise an engine command past the gates.

### Validation (`validateAgentPlan`)

A deterministic gate that rejects a plan **before** any spend. It fails on: an empty plan; more than `AGENT_MAX_STEPS` (200) steps; a malformed (non-object) step; a step class the caller's [policy tier](#policy-tiers-tool-class-allowlists) forbids; an engine op missing a schema-required param (checked live against `describeSchema()`); any tool the executor cannot actually run (e.g. `generate.video` — pricing an unrunnable verb would let a plan spend earlier steps' credits and then abort); and a worst-case plan cost above `AGENT_MAX_PLAN_CREDITS` (5000) — the "make 500 films" hard cap, independent of the workspace balance. When the caller supplies `options.feasibility`, feasibility errors are appended (below). Returns `{ ok, errors[] }` — errors are messages, never throws.

### Reference resolution (`resolveReferences`)

Plans may carry fuzzy reference tokens instead of concrete ids, so the reasoner can say "delete the last clip" without knowing ids. `resolveReferences(plan, state)` is a pure pre-pass that binds tokens against the **live** state via the same engine query surface, before validation and execution:

- Grammar: `@first`, `@last`, `@longest`, `@shortest`, `@all`, `@selected` (from `context.selectedItemIds`), type-scoped `@first:video` / `@last:audio`, and `@track:first` / `@track:last`.
- Recognized in the single-id param keys `id`, `itemId`, `assetId` and the list key `ids` (including tokens inside arrays).
- Returns a new plan plus an `unresolved[]` list — an unresolvable token becomes a hard error (or a clarifying question), never a wrong-item edit.

Both executors re-run resolution at execute time against the authoritative state, so a stale client-side binding can't mutate the wrong item.

### Feasibility gating (`planFeasibilityErrors`)

AGENT-019/022: reject a step the user *can't actually run* so the agent never offers a dead step. The check is pure and fail-open — it only fires on pre-resolved context the route supplies (`{ plan: <tier>, publishConnected }`):

- **Tier gating** is mechanically present (`TOOL_MIN_PLAN`, rank-ordered `free < pro < enterprise`) but the map is **intentionally empty**: the agent's cross-studio verbs are gated by credits (the free plan includes 0) or a runtime precondition, not by plan tier. Add a verb only if its own route genuinely enforces `planAtLeast(...)`.
- **Publish precondition**: `publish` with `publishConnected === false` errors ("connect an account first"). The `/api/agent/plan` route resolves the connection status live — but only when the plan actually contains a `publish` step.

### Clarifying questions

AGENT-046: when intent is ambiguous, the agent asks **one** targeted question and pauses instead of guessing. Two conservative triggers:

- **Deterministic**: a reference token failed to resolve *and* the composition has candidate items — the error is upgraded to a `select_item` question whose choices carry real item ids (the client re-plans with `context.selectedItemIds = [choice]`).
- **Model-driven**: the tool-use loop declares an `ask_user` function tool; the model calling it pauses planning (any steps sketched so far are dropped — they'd be guesses on an intent the model itself flagged).

A question turn returns `{ valid: false, needsClarification: true, question: { question, kind, choices? } }` with an empty plan envelope — nothing can run. `normalizeClarifyingQuestion` caps/redacts the text and caps choices at 6.

## The tool registry

`src/agent-tools.js` (AGENT-016) is the single canonical catalog of everything the agent can reach. It is **auto-derived live**, not hand-listed:

1. **Read tools** — `query` and `describe`, with the engine's query surface typed out.
2. **Engine commands** — pulled from `describeSchema().commands` at build time, so a new engine command is automatically visible to the agent the moment it lands in the engine (`class: 'mutate'`, `reversible: true`, `cost: 0`).
3. **Capability tools** — declared here because they live outside the engine: `director`, `generate.image/music/voice`, `recipe`, and the deferred cross-studio verbs (`upscale`, `cut.silences`, `add.captions`, `create.short`, `auto.reframe`, `publish`). Each carries `class`, `reversible`, `async`, `deferred`, the billing meter `reason`, and a `cost` single-sourced from `USAGE_COSTS` (with `costVaries` where the true cost is only known at run time).

Helpers: `listTools`, `getTool`, `toolNames`, `describeToolsForStudio(registry, studio)` (reads + engine everywhere; generate verbs scoped to their studio — AGENT-023 groundwork).

`toFunctionDeclarations(registry, { only })` emits a provider-style function-declarations array (JSON-Schema params, sanitized names via `functionName` — providers require `^[a-zA-Z0-9_-]{1,64}$`, so `generate.image` → `generate_image`); each declaration carries the canonical verb under `tool`, and `functionToolMap` gives the reverse lookup. The same registry backs the Gemini tool-use loop, `GET /api/agent/tools`, and any future MCP/public-API surface.

**The parity test is the contract.** `test/agent-tools.test.js` (AGENT-021) locks:

- every `describeSchema()` command is a registry engine tool, and vice versa (no phantom tools);
- every engine tool's declared params **deep-equal** the schema's (no typed-surface drift);
- every tool's `class` agrees with `agent.js classifyStep` (one source of truth for read/mutate/generate);
- every tool's `cost` and meter `reason` match `estimateStepCredits`/`stepMeterReason` (the approval card can never show a stale price);
- every emitted function name is provider-valid and reverse-maps with no collisions.

If you add an engine command, the registry picks it up automatically. If you add a capability tool, update `CAPABILITY_TOOLS` *and* `agent.js` (`STEP_COST`, `GENERATIVE_TOOLS`/`DEFERRED_TOOLS`, `GEN_KINDS` if it runs inline) — the parity test fails until the two agree.

## Phase 2 — EXECUTE (`runAgentPlan`)

`runAgentPlan(user, plan, options)` re-runs the full safety stack (never trusting that the caller did): `resolveReferences` → `validateAgentPlan` → `planNeedsApproval` (412 without `approved: true`) → `assertConfirmed` (412 without `confirm: true` when confirmations exist) → `assertAffordable` (402 before any spend). Then it walks the steps in order.

### Atomic engine batches (`compileEngineBatch`)

Consecutive engine steps (reads + commands) buffer up and flush as **one** `{ kind: 'batch', requests }` engine request. The engine batch is atomic: any step failing rolls the whole batch back, so a multi-clip edit never half-applies. Two details worth knowing:

- Read steps compile through `compileReadRequest`, which whitelists exactly `target/where/select/orderBy/limit/offset/aggregate` — a read step's params can never inject `kind`/`op` and smuggle a mutating command past classification (a raw spread would allow exactly that privilege escalation; AGENT-084 review).
- On a batch failure, the engine's `requests[i]` error path is mapped back to the plan step, so the failure report says "step 4 (muteTrack)" rather than an opaque batch error.

For **stored compositions**, the route injects `runBatch` so batches persist through the same row-locked `evaluateComposition` path as `/api/json` — auth, workspace scope, locking, and persistence stay single-sourced. For **inline** state, the default runner tracks the working manifest and returns it.

### Editor-side apply (undoable runs)

The editor never round-trips the open document through the server executor (the director-overlay lesson: no blocking server calls against a live editing session). Instead:

- `editor/src/editor/agent/apply-run.ts` (AGENT-030) is a framework-free core that compiles a run's mutating engine steps into the same atomic batch shape and applies it via the editor's headless `json-api` engine. The `json-api` **reducer-parity test** (`editor/src/editor/json-api/reducer-parity.test.js`) guarantees `evaluate()`'s commands equal the standalone editor reducer actions — so an agent run applied through the reducers yields the *same document* a headless or server apply would.
- `use-apply-agent-run.ts` runs that transform inside `useWriteContext.setState`, so the run lands on the undo/redo stack. Passing a `runId` coalesces every commit of the run into **one undo entry** (`undoCoalesce` key `agent-run:<runId>`, 15-minute window — AGENT-031 via EDIT-091): one Cmd-Z undoes the whole agent run. Selection pointing at deleted items is dropped; a no-op run keeps the state ref and pushes no undo entry.
- `agent-panel.tsx` (AGENT-094) is the chat UI: plan card (summary, steps, cost range, dry-run change list, confirmations, safety flags), typed clarifying-question choices, approve-and-run. Flag-gated by `FEATURE_AGENT_PANEL` in `editor/src/editor/flags.ts`, toggled from the StudioBar.

### Generative step dispatch

`dispatchGenerativeStep(user, step, { seams, meter })` runs one generative step: **meter first** (the billing charge fires before the seam, so a seam failure is still an honest spend — the error is annotated with `chargedStep` so the audit ledger records it), then call the seam. Seams are lazily imported (`imagen.js`, `music.js`, `voice.js`, `recipes.js`) and fully injectable for offline tests:

| Kind | Meter reason | Behavior |
|---|---|---|
| `generate.image` | `usage:imagen` | Synchronous; returns `{ kind:'image', url, objectKey }` |
| `generate.music` | `usage:lyria` | Synchronous; returns `{ kind:'audio', url, durationInSeconds }` |
| `generate.voice` | `usage:voice` | Synchronous; returns `{ kind:'audio', url, voice, durationInSeconds? }` |
| `recipe` | none (meters its own steps) | Background — `runRecipeById(..., { background: true })` returns a `runId` to poll; never blocks the HTTP handler |
| `director` | none (meters per shot inside the runner) | Hands off to the durable film pipeline (`runStarterFilm`: planFilm → produceFilm → assemble → persist) as a background run; returns `status:'running'` + `runId`. Never inline — the repo's own `/api/director/produce` 502 lesson |
| `upscale`, `publish`, `create.short`, `auto.reframe`, `cut.silences`, `add.captions` | none yet | **Deferred** (AGENT-017/018): planned, priced, validated — but reported `status:'deferred'` with no spend; their durable executors are follow-on projects |
| `kind:'studio'` | none | A client-side UI transition — reported done, never dispatched |

### Asset-flow chaining (`@asset:step-N`)

AGENT-066: a later step may reference an earlier generative step's output with the whole-string token `@asset:step-N` (any param, including inside id arrays). At execute time — after that generation actually lands — `chainAssetTokens` resolves the token by injecting one internal `addAsset` engine step (registering the produced media as a real manifest asset with a deterministic id `agent-<runTag>-step-N`) into the **same atomic batch** as the placing step, then substituting the asset id. Consequences:

- A mis-chained plan (placing before generating) throws a clean 422, and the batch rolls back wholesale.
- If the producing step **deferred**, the token rides along unresolved for the durable executor to bind later.
- The run tag is random per run (`options.runId` or a UUID slice) — a constant fallback once made predicted asset ids collide across runs and place stale media (review finding).
- Injected `addAsset` steps are `internal: true` bookkeeping: they never appear in results or the progress stream, and if the engine uniques a colliding asset id the run aborts loudly rather than binding the wrong media.
- Dry-run treats `@asset` steps as `chained` work (the asset doesn't exist in a preview clone), via `hasAssetToken`.

### Typed failure recovery

AGENT-048/049: when a generative step fails, `classifyStepError` types the failure so the executor reacts instead of aborting blindly:

| Kind | Trigger | Action |
|---|---|---|
| `transient` | 408/425/429/5xx, network codes, timeout/overloaded wording | Retry with exponential backoff (`retryBackoffMs`: 250ms → 4s cap), up to `maxStepRetries` (default 2). Retries skip the pre-meter — one generation is charged exactly once |
| `quota` | 402 / out-of-credits wording | Pause: rethrown as 402 (`action: 'pause'`) — top up or approve a smaller plan |
| `bad_param` | 400/422 / validation wording | Re-plan: rethrown as 422 (`action: 'replan'`) — the same params fail every time |
| `unknown` | everything else | Fail as-is |

Failures carry `err.recovery = { kind, action, attempts, reason }` and `err.failedStep`; `describeRunFailure` turns that into the creator-facing report: which step ("step 4 of 7 (generate.music)"), why (one clean first line — never a stack trace), the atomicity guarantee ("no changes from the failed step were applied"), and exactly two concrete fixes per failure kind. Mid-run failures attach `err.partialSteps` so the audit row records what already ran (and already spent) — never a false `credits: 0`.

### Dry run and the approval-card preview

`dryRun(plan, state)` (AGENT-026/027) is the content of the approval card, computed with zero side effects: engine steps run against a `structuredClone` of the state (never the real composition, never persisted), producing a structured `diffState` (items added/removed/changed, composition size/fps, tracks, color grade, master audio) plus described-not-run generative steps and the cost range. `summarizeDryRun` renders the plain-English line — "removes 2 clips, resizes to 1080×1920, will generate an image (a title card)". The `/api/agent/plan` route attaches this as `preview` whenever the plan is valid and a state is available.

## Governance

### Approval policies and the hard confirm floor

Two independent gates stack at execute time:

**1. Approval (`planNeedsApproval`, AGENT-028)** — reuses the `checkpoints.js` vocabulary (`auto` / `guided` / `manual`) per step class:

- Reads are always auto.
- `generate` steps, deferred/external verbs, and **destructive engine ops** (`deleteItems`, `cutItems`, `deleteTrack`, `removeRangesFromItem`, `addAsset`) are gated under **every** policy — even `auto` never runs a spend or a deletion unattended.
- Plain engine mutates are what the policy governs: `auto` runs them unapproved (undoable edits), `guided` (the default) gates them, `manual` gates everything non-read.
- An unknown policy resolves to `guided` — bad input can never escalate to auto.

Executing a gated plan without `approved: true` throws 412.

**2. Confirmations (`planConfirmations` + `assertConfirmed`, AGENT-082/029)** — deterministic, computed on the *bound* plan, and enforced on top of approval (412 without `confirm: true`):

- `bulk_delete`: more than `AGENT_MAX_BULK_DELETE` (25) **distinct** ids deleted cumulatively across the plan (splitting a big delete into sub-threshold steps can't slip past; padded id lists can't inflate). `deleteTrack { deleteItems: true }` always confirms (an implicit, unbounded wipe).
- `external`: `publish` always confirms — content leaving the platform can't be un-sent.
- `bulk_generate`: more than `AGENT_MAX_UNCONFIRMED_GENERATIONS` (8) irreversible generations, even if each is cheap.
- `high_spend`: expected spend at/over `AGENT_CONFIRM_CREDITS` (100) — using the **uncertain ceiling** for director/recipe fan-outs so a plan can't sit just under the line while its real cost is ~4x.

Confirmations don't invalidate a plan (the preview still renders); they require explicitness at execute.

### Policy tiers (tool-class allowlists)

AGENT-084: `AGENT_POLICIES` names the step classes a calling context may run — `full` (read+mutate+generate, the default), `editor` (read+mutate — deterministic edits, no spend), `readonly` (query/describe only — a shared/proof surface). Validation rejects any step whose class the tier forbids *before anything else*. A caller can only downgrade itself (`resolvePolicy` uses `Object.hasOwn` so a prototype-key `policy` like `__proto__` falls back to `full` instead of crashing) — never escalate.

### Input redaction and moderation

`src/chat-safety.js` (shared with the chat system — see [chat-system.md](chat-system.md)):

- `redactSecrets` replaces secret-shaped substrings (bearer tokens, JWTs, provider API keys, `password=`-style assignments) before the message reaches the model and before any free text lands in `agent_runs`. Because the model only ever sees redacted text, it cannot echo a secret into a generated param (AGENT-083).
- `moderateInput` flags `prompt_injection` / `credential_probe` (advisory, `log_only`) — surfaced as `plan.safety` on the approval card.
- KB grounding is treated as **untrusted data**: `agent-context.js` moderates each KB snippet and *excludes* injection-flagged docs from the prompt entirely (exclusion beats annotation on a spend surface), reporting them as `kbSafety`; the route merges those flags into the plan's safety field (`kb:prompt_injection` etc.) so the human learns their KB holds hostile-looking material.

### Budget: pre-flight check and run ledger

Three layers, from plan to per-step:

1. **Hard cap** (`MAX_PLAN_CREDITS`, validation): worst-case cost above 5000 credits is never plannable, regardless of balance.
2. **Pre-flight affordability** (`assertAffordable`, AGENT-038): before any spend, the plan's worst-case `estimatePlanCost().high` is checked against remaining workspace credits (`billing.getEntitlements`) — 402 with the shortfall if it can't afford it. No-op when billing is disabled.
3. **Run-scoped ledger** (`createLedger` from `src/budget.js` in `cap` mode, AGENT-037): built only when the plan spends inline (`generate.image/music/voice`), budgeted at the workspace's remaining credits with `reservePct: 0` (the pre-flight already gated the whole plan). Each inline step **reserves** its estimate before dispatch — a reservation that would overspend halts the run with a 402 at the exact step, before the charge — then **reconciles** to actual committed credits on success or **refunds** on failure. `ledger.summary()` returns `{ mode, budgetCredits, holdback, reserved, spent, usable }` on the run result, and every progress event carries `creditsSpent` + `budgetRemaining` (AGENT-040 live cost meter).

Director and recipe steps are deliberately *not* reserved: neither spends synchronously here (director meters per shot inside its durable runner; recipes meter internally), so reserving them would falsely halt a run.

## Audit and resume: `agent_runs`

`migrations/032_agent_runs.sql` + `src/agent-runs.js` (AGENT-010). Every execution is an auditable row: workspace/composition/user, the natural-language `message`, the full validated `plan` JSONB (including its provenance trace), the per-step `steps` record in original plan order, `status` (`planned | running | done | failed`), `credits` **committed** (from `committedCredits` — the honest figure including charged-but-failed steps), and a plain-English `verdict` (or failure reason). Indexed by workspace and by composition, newest first.

Design rules (both mirror `composition-versions.js`):

1. Every write is injectable (`query`) so the module unit-tests offline.
2. Ledger writes are **best-effort**: `startAgentRun`/`finishAgentRun`/`failAgentRun` swallow their own errors (returning `runId: null` / `false`) — an audit-write failure must never fail the actual execution. The read APIs (`listAgentRuns`, `getAgentRun`) *do* throw; a user asking for history should see a real failure.

The route opens the row before execution and closes it after; a mid-run throw records `err.partialSteps` + committed credits and marks the run `failed` with the failure report as the verdict. The stored plan + per-step record is the substrate a resume/replan step reads from.

Projections over the table (no extra tables):

- `getAgentTranscript` (AGENT-056): the last N `(asked → did)` turns, oldest-first — the agent's conversation memory, fed back into planning context so "make it like last time" grounds on real history.
- `sumAgentRunCredits` (AGENT-057): a SQL `SUM`, never a paged sample — `listAgentRuns` caps at 200 rows, so summing a page would silently under-report a busy workspace.
- `getAgentRun` is workspace-scoped and returns 404 (never 403) on a foreign or malformed id, so run ids can't be probed.

## Observability

### Live progress over SSE

AGENT-044/045: for stored-composition runs, `/api/agent/execute` wires `onProgress` to `publishRealtime` on the composition channel (`src/realtime-bus.js` — the same SSE bus as in-editor comments). Event names are **dotless** (the bus's `EVENT_RE` rejects dots): `agent_plan`, `agent_start`, `agent_step`, `agent_recovery`, `agent_done`, `agent_error`. Every `agent_step` carries `{ index, total, id, tool, class, status, creditsSpent, budgetRemaining }`. Emission is best-effort by construction (`makeEmitter` swallows sink errors) — a streaming failure can never fail the run.

The editor subscribes via `GET /api/agent/compositions/:id/events` (ownership verified via `loadComposition` **before** the socket is held open). `agent-activity-bridge.tsx` maps the events onto the editor's global `activityBus` (top progress bar + event log + toasts) through the pure, node-tested mapper `agent-activity-map.ts`. Inline runs have no channel — the panel applies results directly.

### Provenance traces

AGENT-012, over `src/inference.js`: every live planning call attaches a portable trace (spec `{ id: 'agent.kernel', version }`, task `agent-plan` or `agent-plan-tooluse`, params, an `inputsDigest` and `promptSha256` — inputs are digested/hashed, never stored raw, and the message reaching the reasoner is already redacted, so a trace cannot leak a secret), plus timing. `normalizeAgentPlan` preserves it as `plan.trace`, and the plan JSONB in `agent_runs` carries it — a plan is auditable and re-runnable after the fact. This mirrors the Director's `runTraced` pattern; the mock/offline path has `trace: null`.

### The eval harness

`src/agent-eval.js` (AGENT-081): plan quality measured, not vibes. A fixture set (`AGENT_EVAL_CASES`) of intent → expected-shape cases — vertical reframe, grounded delete-last (must bind to the real id `i2`), mute, color grade, combined intents, the flagship cross-studio exemplar ("cut the silences, add captions, make 3 vertical shorts, publish to tiktok"), and two restraint cases (a question and pure chatter must never mutate). `scoreAgentPlan` scores five dimensions 0–5 (tools-in-order, restraint, validity, minimality, grounding) into an overall 0–1; `runAgentEval` runs the whole set through `interpret()`. It runs in CI against the offline mock (a planner regression fails the suite the moment it lands — `test/agent-eval.test.js`) and accepts any reasoner (`geminiAgentToolUse`) for live scoring. The rubric mechanics mirror `src/eval.js`.

## Adjacent verbs

- **`agent-ask.js`** (AGENT-057): read-only "ask the project" Q&A. Deterministic keyword routing (`classifyProjectQuestion`, length-capped input as the ReDoS guard) answers factual questions instantly from engine aggregates and the `agent_runs` ledger — durations, counts, longest/shortest, muted/hidden tracks, size/fps, "how much did this cost", "what did the agent do last". No mutation is *possible* by construction: the module only imports the pure `evaluate()` and the runs readers. Unsupported questions return `{ ok: false, reason }` so the client falls back to the full planner.
- **`agent-recipe.js`** (AGENT-073): `planToRecipe` converts an approved one-off plan into a reusable, parameterized recipe. Honest contract: generative steps convert (prompt-like params lifted into `{{params.slot}}` definitions with the original text as the default); engine edits are **skipped and reported** (they bind composition-specific ids — a recipe is a repeatable generation workflow, not an edit macro); the emitted spec validates against the recipe vocabulary (`STEP_KIND_NAMES`) before it reaches `createRecipe`.
- **Voice, plan-only** (AGENT-091): `src/voice-agent/tools.js` `agentToolHandlers` exposes `plan_creative_edit` to the real-time voice agent — a spoken request routes into `interpret()` and returns a *speakable* plan summary + cost with `needs_approval: true`. Nothing runs from voice; the two-phase invariant holds — the plan executes only after approval in the editor. Clarifying questions come back as a question to ask the caller.
- **Grounding context** (`agent-context.js`, AGENT-054/055/056): `assembleAgentContext` bundles what the planner should always know — composition summary (pure engine aggregates), workspace brand voice, connected publishing accounts (safe fields only — never a token), remaining credits/plan, top workspace-KB hits for the request, and the recent-run transcript. Every read is isolated via `Promise.allSettled` (a failed section is omitted, never blocks planning), and `fitBudget` deterministically trims the bundle under 1,400 chars so the planner's ~2k-char context slice never evicts the client's own selection keys. The route merges client context so client keys win on collision and serialize first.

## HTTP surface

All routes live in `src/api.js`; session auth via `requireUser`, per-user rate limits noted. See [api-reference.md](api-reference.md) for the platform-wide catalog.

### `POST /api/agent/plan` — Phase 1 (30/min)

```json
{
  "message": "delete the last clip and make it vertical",
  "compositionId": "8f9e…",            // or "state": { …inline manifest… }
  "context": { "selectedItemIds": ["i3"] },
  "policy": "full"                      // optional downgrade: "editor" | "readonly"
}
```

Response — `{ plan, valid, errors, degraded, safety, confirmations, preview }`:

```json
{
  "valid": true,
  "errors": [],
  "plan": {
    "summary": "Remove the last clip and reframe to 9:16",
    "steps": [
      { "id": "step-1", "kind": "engine", "tool": "deleteItems",
        "params": { "ids": ["i2"] }, "class": "mutate", "reversible": true, "estCredits": 0 },
      { "id": "step-2", "kind": "engine", "tool": "setCompositionSize",
        "params": { "width": 1080, "height": 1920 }, "class": "mutate", "reversible": true, "estCredits": 0 }
    ],
    "estCredits": 0,
    "confirmations": [],
    "safety": { "flags": [], "action": "allow" },
    "trace": { "traceId": "…", "spec": { "id": "agent.kernel", "version": "1.0.0" }, "promptSha256": "…" }
  },
  "preview": {
    "ok": true,
    "summary": "removes 1 clip, resizes to 1080×1920",
    "diff": { "itemsRemoved": ["i2"], "composition": { "width": { "from": 1920, "to": 1080 } } },
    "cost": { "low": 0, "expected": 0, "high": 0, "uncertain": false }
  }
}
```

A clarification turn instead returns `{ valid: false, needsClarification: true, question: { question, kind, choices? }, … }`.

### `POST /api/agent/execute` — Phase 2 (30/min)

```json
{
  "plan": { "steps": [ … ] },           // echo the /plan result (re-normalized + re-validated server-side)
  "compositionId": "8f9e…",            // stored (persists, streams SSE) — or "state" for inline
  "approved": true,                      // required for any mutating/spending plan (else 412)
  "confirm": true,                       // required when the plan carries confirmations (else 412)
  "approvalPolicy": "guided",           // optional: auto | guided | manual
  "policy": "full",
  "context": { "selectedItemIds": [] }
}
```

Stored-path response: `{ ok: true, runId, steps, summary, persisted: true }` (the manifest persists server-side and is not echoed back). Inline: `{ ok: true, runId, steps, summary, state, changed }`. On failure the route responds with the structured report: `{ error, runId, failure: { message, failedStep, kind, action, attempts, completedSteps, fixes } }` (the failure detail only when the error is exposable — a masked 5xx leaks nothing). For a stored composition the server always binds references against **its own** manifest, never a client-supplied state — otherwise `@last` could resolve to a forged item.

### `POST /api/agent/ask` (60/min)

`{ "question": "how long is track 2", "compositionId": "…" }` → `{ ok: true, intent: "trackDuration", answer: "Track 2 runs 12.5s (375 frames @ 30fps) over 4 item(s).", data: { … } }`, or `{ ok: false, reason: "unsupported" }` (fall back to `/plan`).

### `POST /api/agent/plan-to-recipe` (20/min)

`{ "plan": { "steps": [ … ] }, "name": "Weekly promo" }` → `{ ok: true, recipe: { id, name, slug }, slots: ["imagePrompt"], skipped: [ { id, tool, reason } ] }`. Persists via the same `createRecipe` path as the Recipes UI.

### `GET /api/agent/runs` · `GET /api/agent/runs/:id`

List: workspace-scoped metadata + step counts (optionally `?compositionId=…&limit=…`, cap 200). Detail: the full plan + per-step record; 404 across workspaces.

### `GET /api/agent/tools`

The full registry `{ version, generatedFrom, tools }`; `?format=functions` returns `{ functions: [ …declarations… ] }` for a model tool-use loop.

### `GET /api/agent/compositions/:id/events`

SSE stream of `agent_plan / agent_start / agent_step / agent_recovery / agent_done / agent_error` for that composition's runs (20 streams/min; ownership 404-checked before subscribing).

## Configuration

All env vars are optional with safe defaults; none are required for the kernel to function offline.

| Env var | Default | Meaning |
|---|---|---|
| `AGENT_MAX_STEPS` | 200 | Hard cap on plan size (untrusted input → DoS guard) |
| `AGENT_MAX_PLAN_CREDITS` | 5000 | Hard ceiling on a single plan's worst-case spend — over it, never plannable |
| `AGENT_MAX_BULK_DELETE` | 25 | Distinct deleted ids above which a plan requires explicit confirmation |
| `AGENT_CONFIRM_CREDITS` | 100 | Expected spend at/over which a plan requires explicit confirmation |
| `AGENT_MAX_UNCONFIRMED_GENERATIONS` | 8 | Generative-step count above which a plan requires confirmation |
| `AGENT_MAX_TOOL_TURNS` | 6 | Max Gemini tool-use loop turns per planning call |

Live planning requires the standard Vertex configuration (`vertexConfig().configured` — see [configuration.md](configuration.md)); without it, `interpret` uses the deterministic mock and flags nothing (tests) or sets `plan.degraded` (a live reasoner error). Billing gates (`assertAffordable`, the run ledger) are no-ops when billing is disabled. The editor panel is gated by `FEATURE_AGENT_PANEL` (`editor/src/editor/flags.ts`, currently `true`).

## Testing

Server suites (Node's built-in runner; all offline via the mock reasoner and injected seams/queries):

```bash
npm test                                    # everything
node --test test/agent.test.js              # kernel: plan/validate/execute/recovery/chaining
node --test test/agent-tools.test.js        # registry + the AGENT-021 parity locks
node --test test/agent-tooluse.test.js      # the tool-use planning loop (fake generate)
node --test test/agent-safety.test.js       # redaction, moderation, policy tiers, confirmations
node --test test/agent-runs.test.js         # audit ledger (fake query)
node --test test/agent-ask.test.js test/agent-context.test.js test/agent-recipe.test.js
node --test test/agent-eval.test.js         # the plan-quality rubric thresholds
node --test test/agent-director-handoff.test.js test/agent-voice.test.js
```

Editor suites (also `node --test`, via the TS loader):

- `editor/src/editor/agent/apply-run.test.ts` — the framework-free batch apply.
- `editor/src/editor/agent/agent-activity-map.test.ts` — the SSE→activityBus mapper.
- `editor/src/editor/json-api/reducer-parity.test.js` — the invariant that makes editor-side agent applies equal server-side ones.

When adding a capability tool, run `test/agent-tools.test.js` first — the parity test is designed to fail until `agent-tools.js` and `agent.js` agree on class, cost, and meter reason.

## Gotchas

- **Never call a mutating dispatch from planning.** The tool-use loop's safety rests on `runReadForPlanning` handling only `query`/`describe`; every other call must be recorded, not executed.
- **`/execute` re-normalizes.** Don't assume the client's plan shape — `normalizeAgentPlan` is the shared choke point; skipping it reopens the AGENT-083 secret-into-audit hole.
- **Stored-path reference binding uses the server manifest.** Binding `@last`/`@selected` against a client-supplied state on the stored path would let a forged item id reach storage.
- **Read steps must stay whitelisted.** `compileReadRequest`'s field whitelist is a security boundary — spreading read params flat would let `params.kind: 'command'` smuggle a mutation past classification.
- **SSE event names are dotless.** The realtime bus's `EVENT_RE` rejects dots; `agent.step` silently never arrives.
- **The meter fires before the seam.** A generative failure after metering is a real spend: preserve `chargedStep`/`partialSteps` on every new error path or the audit ledger under-reports.
- **Retries must skip the pre-meter** (`skipPreMeter`) — one generation is billed exactly once.
- **Director/recipe never run inline.** Both are background hand-offs returning a `runId`; an inline await recreates the `/api/director/produce` 502.
- **`estimateStepCredits` honors a step's own `estCredits`** when finite — a forged plan can't *lower* real charges (metering uses `GEN_KINDS` reasons, not the estimate) but tests that hand-build steps should know the override exists.
- **The `@track:` token grammar is currently unreachable in practice**: `resolveToken` handles `@track:first/last`, but `REF_ID_KEYS` omits `trackId`, so a `muteTrack { trackId: "@track:first" }` step passes through unresolved and fails in the engine. Bind track ids from a `query` (or `context.firstTrackId`) instead until the key set is extended.
- **`kind:'studio'` is a UI hint only** — the executor reports it done and spends nothing; the actual studio switch is client-side.

## History: why this is in-house

An earlier proposal ([`../AGENTIC-ADK.md`](../AGENTIC-ADK.md)) recommended adopting the official `@google/adk` TypeScript SDK as the orchestration/memory/eval layer. That proposal was **superseded** by this kernel under the project's no-new-dependencies rule: the plan→execute loop, tool registry, budget ledger, checkpoint policies, provenance traces, and eval harness were built in-house over the existing JSON engine and Vertex seams instead. The ADK document remains in the tree as design history; this kernel is what actually ships.

## Related pages

- [json-engine.md](json-engine.md) — the command engine every agent edit compiles to
- [api-reference.md](api-reference.md) — the full HTTP catalog (Director, recipes, generation)
- [generation-and-models.md](generation-and-models.md) — the Vertex model seams the generative steps dispatch to
- [billing.md](billing.md) — `USAGE_COSTS`, metering, entitlements
- [chat-system.md](chat-system.md) — the shared safety layer (`chat-safety.js`) and SSE patterns
- [editor.md](editor.md) — the editor state model the Agent panel writes through
- [architecture.md](architecture.md) — where the agent sits in the whole platform