# Auto-Clips Application Pipeline

How a raw long video becomes ranked, captioned, voiced vertical shorts — the application-layer code, routes, and contracts. For fleet state, container recovery, and the two production gates, see [auto-clips-infra.md](auto-clips-infra.md).

## On this page

- [Overview](#overview)
- [Architecture](#architecture)
- [Key files and seams](#key-files-and-seams)
- [Detection](#detection)
- [The long-video path (clip proxy)](#the-long-video-path-clip-proxy)
- [Cut and reframe](#cut-and-reframe)
- [Captions](#captions)
- [Voiceover and ducking](#voiceover-and-ducking)
- [Story mode](#story-mode)
- [Publish hook](#publish-hook)
- [Studio page architecture](#studio-page-architecture)
- [API surface](#api-surface)
- [Metering and rate limits](#metering-and-rate-limits)
- [Configuration and env vars](#configuration-and-env-vars)
- [Testing](#testing)
- [Gotchas](#gotchas)
- [Provenance](#provenance)
- [Related docs](#related-docs)

## Overview

Auto-Clips answers one question for a long, un-edited source video: *where are the good moments, and how do I cut them into vertical shorts?* The pipeline is:

```
long source ──(prepare: 1-fps proxy)──▶ detect highlights ──▶ review/select
     │                                                            │
     │                    optional per-clip captions / voiceover / story arc
     │                                                            │
     └────────────────────▶ cut + reframe + enqueue render ◀──────┘
                                       │
                        renders land in storage + Media browser
                                       │
                        optional: schedule to social accounts
```

Every stage is a separate, independently-callable API route; the studio page (`/auto-clips`) is a thin client-side orchestrator over them, exactly like the Director UI pattern — no browser call ever blocks on a long-running server loop.

Design invariants (all enforced in code):

- **One model call replaces a heuristic stack.** Highlight detection is a single Gemini (Vertex) video-understanding call with a structured `responseSchema` — no GPU, no scene-cut tuning, no per-frame motion/audio DSP on the critical path.
- **Pure transforms, injectable I/O.** The manifest shaping (`buildClipManifest`), ranking (`selectDiverseHighlights`), caption normalization (`normalizeCaptions`), and narration splitting (`narrationToCaptions`) are pure functions; every network/DB/model effect is dependency-injected, so the whole contract is unit-testable without Vertex, Postgres, or ffmpeg.
- **Outputs ride the ordinary render path.** Each clip is a real editor composition (valid `UndoableState`) inserted into the DB and rendered by the standard render worker — so every short lands in storage and the Media browser automatically, and is exactly as render-safe as a browser-authored project.
- **Zero new dependencies.** Everything is hand-rolled on Node built-ins plus the already-shipped Vertex/Remotion seams.

## Architecture

What talks to what, per stage:

| Stage | Route | App module | Downstream |
|---|---|---|---|
| Prepare (long video) | `POST /api/highlights/proxy` | `src/autoclips-proxy.js` | render worker `/clip-proxy` (ffmpeg → MinIO) |
| Detect (video-native) | `POST /api/highlights/detect` | `src/highlights.js` | Vertex Gemini (video part + structured JSON) |
| Detect (transcript) | `POST /api/captions/highlights` | `src/highlights.js` `rankHighlights` | Vertex Gemini (text) |
| Captions (generated) | `POST /api/captions/generate` | `src/caption-generate.js` + `src/caption-styles.js` | Vertex Gemini (video part) |
| Captions (transcribed) | `POST /api/captions` | `src/captions.js` | Google STT v2 |
| Story arc | `POST /api/auto-clips/story` | `src/story-narrator.js` | Vertex Gemini (text only) |
| Voiceover | `POST /api/auto-clips/voiceover` | `src/auto-clips.js` `synthesizeClipVoiceover` | `src/voice-design.js` (Chirp HD TTS) |
| Cut + render | `POST /api/auto-clips` | `src/auto-clips.js` `createAutoClips` | `manifest-builder.js` + `shorts.js` + `render.js` → render worker |
| Poll | `GET /api/render/:jobId` | `src/render.js` | `render_jobs` table |
| Publish | `POST /api/auto-clips/publish` | `src/auto-clips.js` `scheduleClipsToSocial` | `src/social.js` `publish` relay |

All routes live in `src/api.js` and require an authenticated API user (`requireApiUser`). Media analysis reuses the qa-analyzer media convention: a `gs://` URI rides to Vertex as `fileData.fileUri` (read Vertex-side, never by our box); an `https` URL is fetched by `src/media-fetch.js` `fetchInlineMedia` (SSRF-hardened: refuses redirects, stream-caps the body) and inlined as base64, capped at ~20 MB.

## Key files and seams

| File | Role |
|---|---|
| `src/highlights.js` | `rankHighlights` (transcript), `detectHighlightsFromVideo` (video-native), `selectDiverseHighlights` (pure), `HIGHLIGHT_TYPE_PRESETS`, `DETECT_SCHEMA` |
| `src/auto-clips.js` | `createAutoClips` orchestrator, pure `buildClipManifest`, `scheduleClipsToSocial`, `synthesizeClipVoiceover`, `AUTO_CLIPS_MAX` |
| `src/manifest-builder.js` | server-side source of truth for a valid editor manifest; `videoAsset`/`videoItem`/`audioAsset`/`audioItem`/`captionAsset`/`captionsItem`/`aspectToDims` |
| `src/shorts.js` | `reframeManifest` — the deterministic center-cover affine reframe (shared with the Shorts feature) |
| `src/caption-styles.js` | the 9 caption personas, `TYPE_TO_STYLE` auto-map, `resolveCaptionStyle`, `isStoryStyle` |
| `src/caption-generate.js` | `generateStyledCaptions`, pure `normalizeCaptions`, `CAPTIONS_SCHEMA` |
| `src/voice-presets.js` | per-persona voice briefs, `resolveVoicePreset`, `DEFAULT_DUCK_DB`, `duckDbForLinear` |
| `src/story-narrator.js` | `generateStory` (one arc across N clips), pure `narrationToCaptions`, `resolveStoryStyle` |
| `src/autoclips-proxy.js` | app-side adapter for the worker's `/clip-proxy` endpoint (mock, deploy gate, shared secret) |
| `render-worker/clip-proxy.js` | pure `ffmpegProxyArgs` + content-addressed `proxyStorageKey` |
| `render-worker/clip-proxy-service.js` | `processClipProxy` (fetch → ffmpeg → MinIO) + `createClipProxyServer` HTTP service |
| `render-worker/worker.js` | opt-in `CLIP_PROXY_PORT` wiring of the clip-proxy server into the worker process |
| `public/auto-clips.html` | the self-contained studio page (client-side pipeline) |
| `src/api.js` | all `/api/highlights/*`, `/api/captions/generate`, `/api/auto-clips*` route handlers, SSRF + workspace guards, metering |
| `src/server.js` | `/auto-clips` → `/public/auto-clips.html` page route |

## Detection

### Video-native: `detectHighlightsFromVideo` (src/highlights.js)

The keystone. Gemini watches the whole video (or its proxy) and returns timestamped moments directly — it "sees" content pure audio/motion signals can't (humor, insight, a quiet emotional beat).

Input contract:

- `videoUrl` — required, `https://` or `gs://`.
- `count` — 1–20, default 8.
- `minSeconds` — ≥3, default 12 (`AUTOCLIPS_MIN_SECONDS`).
- `maxSeconds` — ≥ min, default 60 (`AUTOCLIPS_MAX_SECONDS`).
- `minGapMs` — default 500 (`AUTOCLIPS_MIN_GAP_MS`).
- `durationSeconds` — optional; clamps end timestamps to the real source length.
- `preset` — `general` (default) or `gaming`; or a custom `types` array (up to 12 entries of `{id, description}` or plain strings; ids are sanitized to `[a-z0-9_]`, ≤24 chars).

`HIGHLIGHT_TYPE_PRESETS` is the semantic moment taxonomy that seeds the detection prompt:

- **general** (default, any long-form footage): `hook`, `insight`, `emotional`, `funny`, `surprising`, `action`, `payoff`.
- **gaming** (opt-in): `action`, `funny`, `clutch`, `wtf`, `epic_fail`, `hype`, `skill`.

The internal `defaultVideoAnalyze` makes the real Vertex call: prompt (built by `detectionPrompt` — "watch the ENTIRE video, up to N distinct non-overlapping moments, exact start/end seconds, single best type, 0–100 shareability score, punchy title + hook") plus the media part, with `generationConfig: { responseMimeType: 'application/json', responseSchema: DETECT_SCHEMA, temperature: 0.3 }` so Vertex validates the model's output to the schema. Errors are honest and user-facing: 503 when Vertex is unconfigured, 422 when the video can't be read (not gs:// and too big/unfetchable for inline), 502 on a model failure or malformed output. The `analyze` function is injectable for tests.

### Selection: `selectDiverseHighlights` (pure)

Ranking and de-duplication is deliberately I/O-free:

1. **Normalize** each candidate: accepts `startMs`/`endMs` (numbers) or `startSeconds`/`start` in seconds or `"MM:SS"`/`"HH:MM:SS"` strings; clamps `endMs` to `durationMs`; accepts scores as 0–1 or 0–100.
2. **Duration policy**: drops anything shorter than half of `minMs`; a too-long moment is *trimmed to `startMs + maxMs`* (keep the hook) rather than dropped.
3. **Overlap removal within `minGapMs`** — greedy, best-score-first (a robustness check the workflow this was modeled on lacked).
4. **Two-pass diversity pick**: pass 1 takes the top-scoring moment of each unique type; pass 2 fills remaining slots by score.
5. Returns best-first, capped to `count`, each as `{ startMs, endMs, type, score, title, hook, reason }`.

### Transcript-based: `rankHighlights` (the older V5 path)

Still live at `POST /api/captions/highlights` for footage that already has word-level captions (from `POST /api/captions` transcription). It groups captions into sentence-ish segments (`segmentCaptions`, capped at 400 segments ≈ 1 h of speech), sends the numbered transcript to Gemini text generation (temperature 0.3), then hard-validates the returned JSON array: timestamps must sit inside the transcript span (±500 ms), durations within `[0.5·min, 1.5·max]`, no overlaps; junk entries are dropped, never passed through to cut a timeline in the wrong place. Returns `{ highlights: [{startMs, endMs, title, score, reason}], segments }`.

### Route: `POST /api/highlights/detect`

- `requireApiUser`, rate-limited 12/min per user.
- `videoUrl` guard: a `gs://` URI passes through (read by Vertex, not our server); an `https` URL must pass `safePublicUrl` (https-only, blocks loopback/link-local/private/metadata hosts — `src/net-guard.js`).
- **Workspace binding** (`assetUrlInWorkspace`): if the URL targets our own asset proxy (`/api/assets/...`), the object key must contain the caller's workspace id — asset tokens are HMAC over the key only (non-expiring), so without this check a leaked signed URL would let one tenant analyze another's stored footage. External-host URLs are left untouched.
- Meters `usage:content` *before* the Vertex call.

## The long-video path (clip proxy)

A long MinIO/R2 source is far bigger than the ~20 MB Vertex inline cap, so it can't be analyzed directly. The fix is a small, **time-preserving** proxy built on the render worker (the only box with ffmpeg):

- **`fps=1` preserves wall-clock time** — proxy second N is source second N — so the timestamps Gemini returns on the proxy map 1:1 back onto the original. No rescaling anywhere.
- Visual detail is intentionally lossy (default long-edge height 480, x264 `veryfast` CRF 30): the model only needs to *locate* moments, not see them sharp.
- A low-bitrate **mono audio track is kept** (default 24 kbps AAC) because multimodal detection uses reactions — laughs, shouts, applause — to spot highlights.

### App side: `src/autoclips-proxy.js`

`processClipProxy({sourceUrl, workspaceId, fps, height})` POSTs to `${CLIP_PROXY_URL}/clip-proxy` with the `x-render-token: RENDER_SHARED_SECRET` header (the worker fetches the source server-side, so an anonymous endpoint would be an SSRF/DoS proxy). Behavior by config:

- `CLIP_PROXY_MOCK=true` → deterministic passthrough `{ url: sourceUrl, key: null, passthrough: true, mock: true }` so the prepare→detect loop is e2e-testable without the worker (fine for short sources that already inline).
- `CLIP_PROXY_URL` unset (and no mock) → a clear, actionable error; the route turns this into an honest 503 *before* metering.
- Timeout: `CLIP_PROXY_TIMEOUT_MS`, default 30 minutes.

### Worker side: `render-worker/clip-proxy.js` + `clip-proxy-service.js`

- `ffmpegProxyArgs(input, output, opts, env)` — pure argv builder: `-vf fps=<fps>,scale=-2:<height>` (even width for yuv420p), `libx264 -preset veryfast -crf <CLIP_PROXY_CRF>`, mono AAC or `-an`, `+faststart`. Clamps: fps 1–8 (default 1), height 120–720 (default 480), audio 0–128 kbps (default 24), CRF 18–40 (default 30).
- `proxyStorageKey({sourceUrl, fps, height})` — content-addressed sha1 (16 hex chars) over the canonical `(sourceUrl, fps, height)` tuple, so the same proxy is reused, never collides across sources, and re-renders if the fps/height policy changes.
- `processClipProxy` — fetches the source with `redirect: 'manual'` (the app validated only the first hop, so a 3xx bounce to an internal host is refused), enforces `CLIP_PROXY_MAX_BYTES` (default 512 MiB) via both the declared Content-Length *and* a streaming byte cap (a lying/missing header can't OOM the box), ffprobes dims + duration, runs ffmpeg (hard-killed at `CLIP_PROXY_FFMPEG_TIMEOUT_MS`, default 10 min), and uploads to `clip-proxy/<workspaceId>/<hash>.mp4` in MinIO. Returns `{ url, key, byteSize, width, height, durationSeconds, fps }`.
- The proxy is an **ephemeral analysis artifact — no assets row is written** (it must not clutter the Media browser). A storage sweep can prune the `clip-proxy/` prefix later (none exists yet — see Gotchas).
- `createClipProxyServer({secret, storageConfig})` — HTTP service: `GET /health`, `POST /clip-proxy` gated by a timing-safe `x-render-token` compare, 1 MiB JSON body cap. It **fails closed**: it refuses to start without a shared secret.

### Wiring, gates, and the worker-image COPY requirement

- `render-worker/worker.js` starts the clip-proxy server only when `CLIP_PROXY_PORT` is set (opt-in, same pattern as the upscale/audio-fx/remove-bg/vectorize endpoints), sharing `RENDER_SHARED_SECRET`.
- The app is gated by `CLIP_PROXY_URL` (or `CLIP_PROXY_MOCK`); unset → the route returns 503 and never charges. In production today the service runs as a dedicated `corereflex-clip-proxy` container — see [auto-clips-infra.md](auto-clips-infra.md).
- **Worker-image COPY requirement**: `render-worker/Dockerfile` must `COPY` both `clip-proxy.js` and `clip-proxy-service.js` (it does, on the same line as the other service modules). A worker module statically importing a sibling the Dockerfile omits crashes the image on boot with `ERR_MODULE_NOT_FOUND`; `render-worker/dockerfile-copy.test.js` statically resolves the relative-import graph of every COPYied file and asserts each imported sibling is also COPYied — keep it green when adding modules.

### Route: `POST /api/highlights/proxy`

- `requireApiUser`, rate-limited 6/min. Body field is **`sourceUrl`** (not `videoUrl` — see Gotchas), guarded by `safePublicUrl` (https only; `gs://` is not accepted here because the worker itself must fetch it).
- Honesty gate: 503 with a clear message when `CLIP_PROXY_URL` is unset and mock is off, *before* metering.
- Meters `usage:content`, then wraps the result in `proxiedDerivative` (`src/storage.js`): the worker built its `url` from fleet-internal MinIO env a browser can never reach, so the descriptor's URL is re-pointed at the signed asset proxy (`/api/assets/<key>?t=…`). Passthrough (mock) results keep the original URL untouched.

Client flow for a long source: `POST /api/highlights/proxy` (long source → small proxy URL + real dims/duration) → `POST /api/highlights/detect` with the proxy URL (now small enough to inline).

*Known limit (AC1.5b, not built):* an hour-plus source can produce a proxy that still exceeds the inline cap; the planned fix is GCS staging so the proxy rides as a `gs://` fileUri.

## Cut and reframe

### `createAutoClips(user, input, options)` (src/auto-clips.js)

The AC2 orchestrator: raw source + detected highlight windows → N enqueued vertical-short renders. It is a **pure manifest transform + enqueue** — no ffmpeg on the API box, no transcription, no probing.

Input:

```js
{
  source: { url, width, height, durationSeconds?, fps?, hasAudioTrack?, sizeBytes? },
  highlights: [{ startMs, endMs, title?, type?, score?, captions?, voiceover? }],
  aspect?,   // '9:16' (default) | '1:1' | '16:9' | '4:5'
  codec?,    // 'h264' (default) | 'vp8'
  duckDb?,   // voiceover ducking override, default DEFAULT_DUCK_DB
}
```

- `source.url` must be `https://` or `gs://` (module-level re-check — the render worker fetches it server-side, so it never relies on the route having pre-validated).
- `source.width`/`height` are required (400 otherwise); `fps` defaults to 30.
- Windows are validated: `endMs` clamped to the source duration ("never read past the source"), sub-second windows dropped (`MIN_CLIP_MS` = 1000).
- The batch is capped at `AUTO_CLIPS_MAX` (default 10, `AUTOCLIPS_BATCH_MAX` overridable up to 50) — each clip is a real render cost, bounded exactly like the Director's `MAX_SHOTS`.

Per window it: builds the manifest (`buildClipManifest`), inserts a composition (`insertComposition` — the same lazy new-project insert `createShort` uses: a `campaigns` row named `Auto-clip · <title>` plus a `compositions` row, in one transaction), and enqueues an ordinary render (`enqueueRender`). Returns `{ clips: [{jobId, compositionId, startMs, endMs, title, type, score, aspect}], count, aspect }` — the caller polls each `GET /api/render/:jobId`. On success the render worker inserts the assets row, so every short lands in storage + the Media browser automatically.

### `buildClipManifest` (pure, injectable)

Builds one render-safe short manifest:

1. **Trim.** A full-frame `videoAsset` + `videoItem` where the *asset* duration is the SOURCE length (so the editor's trim bounds stay honest and fully contain the out-point) and the *item* duration is the CLIP length, with `videoStartFromInSeconds = startMs/1000` as the in-point. This asset/item duration independence is why the manifest is hand-built here instead of using `buildManifest` (which ties them together).
2. **Reframe.** `reframeManifest(base, target)` from `src/shorts.js` — the deterministic center-cover affine the shipped Shorts feature uses: `scale = max(W1/W0, H1/H0)` so the source fully covers the target canvas (center-crop on the long axis); each item is scaled about the source center and re-centered; absolute-pixel style fields (fontSize, strokeWidth, …) scale with the zoom. Target dims come from `aspectToDims` (`9:16` → 1080×1920, etc.).
3. **Voiceover + ducking** (see [Voiceover and ducking](#voiceover-and-ducking)) — applied when `voiceover.audioUrl` is present.
4. **Caption burn-in** (AC3b) — applied when `captions` is a non-empty array. Captions are attached **after** reframe so they are laid out in TARGET dims (a bottom-center caption box), not cover-scaled with the video. They ride their own track pushed last (last track renders on top).

### Editor shapes: `captionAsset` / `captionsItem` (src/manifest-builder.js)

`manifest-builder.js` mirrors the editor's real types (`editor/src/editor/state/types.ts` etc.) so a server-built composition loads and renders identically to a browser-authored one.

- `captionAsset({captions})` — an asset of `type: 'caption'` holding the caption array coerced to the `@remotion/captions` `Caption` shape: `{ text, startMs, endMs, timestampMs: midpoint, confidence: null }`, sorted by start; invalid/empty entries dropped; text whitespace-collapsed and capped at 200 chars.
- `captionsItem({assetId, durationInFrames, compositionWidth, compositionHeight, fontSize?})` — an item of `type: 'captions'` carrying the styling, with the editor's json-api defaults verbatim: TikTok Sans weight 600, fontSize 80, white text with green highlight `#39E508`, black stroke width 4, `pageDurationInMilliseconds: 2000`, `maxLines: 2`. The box is bottom-center: width `min(compositionWidth, 900) − 40`, top `compositionHeight/2 + 150`.

In `buildClipManifest`, incoming captions are **CLIP-relative** (0-based) `{startMs, endMs, text}` — the same shape both `/api/captions/generate` and `/api/captions` (transcription) emit — and are clamped to the clip length before the asset is built.

## Captions

Two sources, one shape. Talking footage should transcribe via `POST /api/captions` (Google STT v2, word-level); silent/gameplay/b-roll footage should *generate* captions via `POST /api/captions/generate`. Both yield `{startMs, endMs, text}` segments, so either attaches identically to a render (AC3b).

### Personas: `src/caption-styles.js`

`CAPTION_STYLES` holds 9 personas, each a natural-language style guide that seeds the generator prompt:

| Persona | Character |
|---|---|
| `gaming` | short punchy creator captions, 1–5 words, CAPS on action beats |
| `dramatic` | cinematic, short and impactful |
| `funny` | meme-style, self-aware, slightly chaotic |
| `minimal` | understated, max 3 words |
| `genz` | internet-culture slang, 1–6 words |
| `story_news` | broadcaster narration, 2–3 sentences per caption, fewer captions |
| `story_roast` | sarcastic playful roasting, 2–3 sentences |
| `story_creepypasta` | horror tension, ellipses, slow reveals |
| `story_dramatic` | epic movie-trailer narration |

- `STORY_STYLES` / `isStoryStyle` mark the four `story_*` personas (longer, fewer captions; also the only valid story-mode voices).
- `TYPE_TO_STYLE` is the `auto` map from a detected moment type to the best-fit persona, covering both taxonomies (e.g. `insight → story_news`, `clutch → dramatic`, `epic_fail → funny`).
- `resolveCaptionStyle(style, type)` — a concrete requested style wins; `auto`/unknown falls through the type map; final fallback `gaming`.

### Generation: `generateStyledCaptions` (src/caption-generate.js)

Gemini watches the clip (same media seam as detection) and invents persona-styled captions timed across it:

- Cadence mirrors the source workflow: ~1 caption per 2.5 s (or per 5.5 s for story personas); `maxCaptions` is derived from `durationSeconds` and clamped 1–40, caller-overridable.
- Prompt rules: space captions throughout (never bunch at the start), 1–3 s each (4–8 s for story), no overlaps, exact start/end seconds, enhance the strongest beats — don't narrate the obvious. `language` other than `en` switches the output language. Structured output via `CAPTIONS_SCHEMA`, temperature 0.6.
- Emoji policy: default on (`emojis: false` to disable). The model returns at most one emoji in a *separate* field; `normalizeCaptions` appends it to the text only if it's a real pictographic char and the text doesn't already contain one.
- `normalizeCaptions` (pure): validates/clamps timestamps (accepts seconds or `MM:SS` strings), collapses whitespace, caps text at 140 chars, sorts by start, drops overlapping captions (keeps the earlier one), caps the count. Output: `[{startMs, endMs, text, style}]`.

Returns `{ captions, style, count, model }`.

### Route: `POST /api/captions/generate`

Same URL-only body + SSRF stance + workspace binding as `/api/highlights/detect`; rate-limited 12/min; meters `usage:content` before the Vertex call.

## Voiceover and ducking

### Presets: `src/voice-presets.js`

Each caption persona has a matching voice preset — a plain-language `description` (voice brief) plus a `speakingRate`. No vendor voice id is hard-coded anywhere: the brief is mapped onto a concrete Chirp HD voice by the first-party voice-design seam (`src/voice-design.js`, `POST /api/voice/design`), keeping the persona → voice mapping in one deterministic place.

- `resolveVoicePreset(style, type)` **reuses `resolveCaptionStyle`** — the same persona resolver captions use — so voice and captions always agree on the persona. Returns `{ style, description, speakingRate, duckDb }`.
- `DEFAULT_DUCK_DB = -10.5` dB ≈ 0.30 linear — how far the source ("bed") audio is pulled down while the voiceover plays. Flat, whole-clip ducking; sidechain/keyframed ducking is a listed later refinement (AC4b). `duckDbForLinear(linear)` converts a 0–1 target level to dB.

### Synthesis: `synthesizeClipVoiceover` (src/auto-clips.js) + `POST /api/auto-clips/voiceover`

Resolves the persona preset and speaks the given text (whitespace-collapsed, capped at 900 chars) via `designVoice(user, {description, text})`. Returns `{ audioUrl, style }` for the caller to attach as `highlight.voiceover`. Rate-limited 30/min; meters `usage:voice`. API callers can equally synthesize via `POST /api/voice` (named voice + `speakingRate`) or `POST /api/voice/design` directly — `buildClipManifest` accepts any `{audioUrl, durationSeconds?}`.

### Manifest wiring (in `buildClipManifest`)

When `voiceover.audioUrl` is present:

1. A full-volume `audioAsset`/`audioItem` (voiceover.mp3, `decibelAdjustment: 0`) is added on its own track.
2. The source video item's `decibelAdjustment` is lowered by `duckDb` (caller override, else `DEFAULT_DUCK_DB`) — the bed duck.
3. **TTS-first timing**: if the voiceover runs longer than the clip, the clip is *extended* (plays more source footage) so the VO is never cut — but never past the end of the source footage. The extension is clamped to the frames actually available after the in-point, and the asset's declared duration is raised to match the clamped out-point so the manifest stays self-consistent (otherwise the item would claim frames the asset doesn't have — a silently-wrong frozen/black tail).

## Story mode

`src/story-narrator.js` (AC5) generates ONE coherent narrative arc across all N selected clips instead of independent per-clip captions: part 1 sets up, the middle escalates, the final part concludes, each segment continuing from the last.

- `generateStory({clips, style, language}, {generate})` — text-only Gemini over the highlights' *metadata* (`title`/`hook`/`type`/`durationSeconds` — the `rankHighlights` pattern; no video upload is needed for a coherent arc). Needs ≥2 clips, caps at 20; temperature 0.8. Returns `{ style, narrativeArc, segments: [{clipIndex, narration}] }` — segments re-ordered by `clipIndex` and padded to the clip count, narration capped at 600 chars.
- `resolveStoryStyle(style)` coerces any style to a `story_*` persona (default `story_news`) — story mode only makes sense with a story voice.
- `narrationToCaptions(narration, {clipDurationSeconds, maxCaptions?})` — pure: splits the narration into sentences and spreads them across the clip proportional to sentence length (min 600 ms each); the last caption always ends exactly at the clip end. Emits the same `{startMs, endMs, text}` shape AC3b burns.

The narration then feeds the existing plumbing twice: as the clip's burned captions (`narrationToCaptions` → `highlight.captions`) and as the clip's voiceover text (client → `/api/auto-clips/voiceover` with the story persona → AC4).

Route: `POST /api/auto-clips/story` — body carries only clip metadata (no storage access → no IDOR surface), rate-limited 12/min, meters `usage:content` before the model call.

## Publish hook

`scheduleClipsToSocial` (src/auto-clips.js) + `POST /api/auto-clips/publish` distribute finished shorts to the workspace's connected social accounts by reusing the shipped `social.publish` relay (which meters `usage:publish` per clip and audits each post — no extra metering in this route).

- Input: `{ clips: [{mediaUrl|outputUrl, caption|title, compositionId?}], accountIds: [], startAt? (ISO), everyHours? }`. Clips without a usable media URL are dropped; captions capped at 2,200 chars; batch capped at `AUTOCLIPS_PUBLISH_MAX` (default 20, max 50).
- Cadence: `everyHours > 0` → clip *i* is scheduled `i × everyHours` after the base (`startAt` or now); `everyHours = 0` with `startAt` → all at `startAt`; neither → publish now.
- One clip's relay failure never aborts the rest — each result carries `{ ok, mediaUrl, scheduleDate, post | error }`; the response summarizes `{ scheduled, failed, results }`.
- Rate-limited 6/min.

## Studio page architecture

`public/auto-clips.html`, served at `/auto-clips` (`src/server.js`), is a single self-contained page (inline CSS + one `<script type="module">`, the `voice.html`/`book.html` pattern) that runs the whole pipeline **client-side** — the Director-UI rule that no browser surface blocks on a long server-side loop:

1. **Source pick** — an in-app library grid (`GET /api/files?type=video&limit=48&sort=newest`, soft-401: an unauthenticated load shows a sign-in prompt instead of redirecting) or a pasted public https URL. Options: clip count, min/max seconds, aspect, moment preset (General/Gaming), caption persona (or None/Auto), voiceover on/off, story mode on/off.
2. **Prepare** — `POST /api/highlights/proxy`. On success the proxy URL plus real `width`/`height`/`durationSeconds` populate the source state. On failure (worker not deployed, or the source too big) the page falls back honestly: it reveals manual width/height/duration inputs and analyzes the raw URL directly ("small clips only").
3. **Detect** — `POST /api/highlights/detect` with the (proxy or raw) URL and options; results render as a checkbox review list (title, time range, type chip, score).
4. **Story arc** (story mode on, ≥2 picks) — `POST /api/auto-clips/story`; per-clip narration is stored by `clipIndex`. The page splits each narration into captions with `splitNarration`, a client-side mirror of the server's `narrationToCaptions`.
5. **Captions** (persona chosen, story off) — `POST /api/captions/generate` **once on the analyzed source**, then `sliceCaptions` cuts the source-relative captions per clip and rebases them to clip-relative 0-based times.
6. **Voiceover** (on) — per selected clip, `POST /api/auto-clips/voiceover` with the caption/narration text (falling back to `title. hook`); the returned `audioUrl` becomes `highlight.voiceover`. A per-clip VO failure is silently skipped.
7. **Generate** — `POST /api/auto-clips` with the source (real or manually-entered dims), aspect, and the enriched highlights. Returns the job list.
8. **Poll** — `GET /api/render/:jobId` every 4 s per unfinished clip; cards flip queued → rendering → ready (inline `<video>` + open/download link) or failed (error text). Finished shorts also land in the Media browser via the ordinary render path.
9. **Distribute** — once at least one short succeeds, a step-4 panel loads `GET /api/social/accounts` (soft-fetch; a missing relay shows a dashboard link, never an error page) and `POST /api/auto-clips/publish` schedules the finished shorts with the chosen start/cadence.

Implementation notes: all user strings are escaped through a local `esc()` before `innerHTML`; a global `[hidden]{display:none!important}` rule beats the author display rules on `.grid`/`.panel` (a real bug fixed during browser verification); a 401 on an action redirects to `/sign-in`.

## API surface

All POST bodies are JSON; all routes need an authenticated session/API user. Representative request/response pairs (shapes grounded in the handlers and modules above):

### `POST /api/highlights/proxy` — prepare a long source

```json
{ "sourceUrl": "https://corereflex.com/api/assets/uploads/<ws>/talk.mp4?t=...", "fps": 1, "height": 480 }
```

```json
{
  "url": "https://corereflex.com/api/assets/clip-proxy/<ws>/<hash16>.mp4?t=...",
  "key": "clip-proxy/<ws>/<hash16>.mp4",
  "byteSize": 9400000, "width": 1920, "height": 1080,
  "durationSeconds": 1841.3, "fps": 1
}
```

503 (with a clear message) when the worker isn't deployed; 400 for a non-public/non-https source.

### `POST /api/highlights/detect` — find the moments

```json
{ "videoUrl": "https://.../<proxy>.mp4?t=...", "count": 6, "minSeconds": 12,
  "maxSeconds": 60, "preset": "general", "durationSeconds": 1841 }
```

```json
{ "highlights": [
    { "startMs": 754000, "endMs": 791000, "type": "insight", "score": 92,
      "title": "The one metric everyone ignores", "hook": "You're measuring the wrong thing.",
      "reason": "Complete, surprising claim with a clean payoff" }
  ],
  "count": 6, "model": "gemini-..." }
```

### `POST /api/captions/generate` — persona captions for one clip/source

```json
{ "videoUrl": "https://...", "style": "genz", "type": "funny",
  "durationSeconds": 37, "maxCaptions": 12, "language": "en", "emojis": true }
```

```json
{ "captions": [ { "startMs": 1200, "endMs": 3400, "text": "no cap this is insane", "style": "genz" } ],
  "style": "genz", "count": 12, "model": "gemini-..." }
```

Same `{startMs, endMs, text}` segment shape as `POST /api/captions` (transcription) — either source attaches to a render.

### `POST /api/auto-clips/story` — one arc across N clips

```json
{ "style": "story_news", "language": "en",
  "clips": [ { "title": "...", "hook": "...", "type": "insight", "durationSeconds": 32 },
             { "title": "...", "type": "payoff", "durationSeconds": 28 } ] }
```

```json
{ "style": "story_news", "narrativeArc": "From a quiet claim to the proof nobody expected",
  "segments": [ { "clipIndex": 0, "narration": "..." }, { "clipIndex": 1, "narration": "..." } ] }
```

### `POST /api/auto-clips/voiceover` — persona TTS for one clip

```json
{ "style": "auto", "type": "insight", "text": "And we're witnessing something special here..." }
```

```json
{ "audioUrl": "https://.../voice/<ws>/....mp3?t=...", "style": "story_news" }
```

### `POST /api/auto-clips` — cut, reframe, render

```json
{
  "source": { "url": "https://.../talk.mp4?t=...", "width": 1920, "height": 1080,
              "durationSeconds": 1841, "fps": 30 },
  "aspect": "9:16", "codec": "h264", "duckDb": -10.5,
  "highlights": [
    { "startMs": 754000, "endMs": 791000, "title": "The one metric everyone ignores",
      "type": "insight", "score": 92,
      "captions": [ { "startMs": 0, "endMs": 2100, "text": "wait for it..." } ],
      "voiceover": { "audioUrl": "https://...", "durationSeconds": 21.4 } }
  ]
}
```

`202`:

```json
{ "clips": [ { "jobId": "...", "compositionId": "...", "startMs": 754000, "endMs": 791000,
               "title": "The one metric everyone ignores", "type": "insight",
               "score": 92, "aspect": "9:16" } ],
  "count": 1, "aspect": "9:16" }
```

Then poll `GET /api/render/:jobId` → `{ status: "queued"|"running"|"succeeded"|"failed", outputUrl?, error? }`.

### `POST /api/auto-clips/publish` — schedule to socials

```json
{ "accountIds": ["acct_1"], "everyHours": 24, "startAt": "2026-07-07T09:00:00Z",
  "clips": [ { "mediaUrl": "https://.../render.mp4?t=...", "caption": "The one metric everyone ignores" } ] }
```

```json
{ "scheduled": 3, "failed": 0,
  "results": [ { "ok": true, "mediaUrl": "...", "scheduleDate": "2026-07-07T09:00:00.000Z", "post": { } } ] }
```

### Adjacent routes the pipeline leans on

`POST /api/captions` (STT transcription, key-bound to the workspace), `POST /api/captions/highlights` (transcript ranker), `POST /api/voice` / `POST /api/voice/design` (direct TTS), `GET /api/files` (library picker), `GET /api/social/accounts` (relay accounts), `POST /api/shorts` (single reframe of a finished film — the pre-Auto-Clips seam `createAutoClips` generalizes).

## Metering and rate limits

Every model/render cost is gated and spent **before** the vendor call ("gate + spend BEFORE"), and deploy-gated routes return an honest 503 *before* metering so a user is never charged for an unavailable feature.

| Route | Meter key | Rate limit (per user/min) |
|---|---|---|
| `POST /api/highlights/proxy` | `usage:content` | 6 |
| `POST /api/highlights/detect` | `usage:content` | 12 |
| `POST /api/captions/highlights` | `usage:content` | — |
| `POST /api/captions/generate` | `usage:content` | 12 |
| `POST /api/auto-clips` | `usage:render` × min(highlights.length, `AUTO_CLIPS_MAX`) | 8 |
| `POST /api/auto-clips/story` | `usage:content` | 12 |
| `POST /api/auto-clips/voiceover` | `usage:voice` | 30 |
| `POST /api/auto-clips/publish` | none here — `social.publish` meters `usage:publish` per clip | 6 |

The per-clip up-front multiplier on `/api/auto-clips` mirrors `/api/director/produce`'s per-shot metering.

## Configuration and env vars

App side (`src/`):

| Var | Default | Effect |
|---|---|---|
| `AUTOCLIPS_TIMEOUT_MS` | 120000 | Vertex call timeout for detect **and** caption generation |
| `AUTOCLIPS_MAX_INLINE_BYTES` | 20 MiB | inline-media byte cap for https videos sent to Gemini |
| `AUTOCLIPS_MIN_SECONDS` / `AUTOCLIPS_MAX_SECONDS` | 12 / 60 | default clip-length bounds for detection |
| `AUTOCLIPS_MIN_GAP_MS` | 500 | minimum gap between selected highlights |
| `AUTOCLIPS_BATCH_MAX` | 10 (cap 50) | `AUTO_CLIPS_MAX` — clips per `/api/auto-clips` call |
| `AUTOCLIPS_PUBLISH_MAX` | 20 (cap 50) | clips per publish call |
| `CLIP_PROXY_URL` | unset | worker `/clip-proxy` base URL; unset → honest 503 on prepare |
| `CLIP_PROXY_MOCK` | unset | `true` → passthrough prepare (keeps the source URL) |
| `CLIP_PROXY_TIMEOUT_MS` | 30 min | app-side prepare timeout |
| `AUTOCLIPS_SUBJECT_AWARE` | unset/false | `true` → one hard-bounded focal detection per batch via clip-proxy `/frame` + vision; keep subject in 9:16 crop. Failures degrade to center-crop. **Prod: true** since NW-S3 (2026-07-29) after `/frame` smoke. |
| `CLIP_FRAME_URL` | falls back to `CLIP_PROXY_URL` | optional override for `grabSourceFrame` only |
| `RENDER_SHARED_SECRET` | — | `x-render-token` for the worker endpoints (must match the worker) |

Worker side (`render-worker/`):

| Var | Default | Effect |
|---|---|---|
| `CLIP_PROXY_PORT` | unset | opt-in: start the clip-proxy HTTP service on this port |
| `CLIP_PROXY_FPS` | 1 (1–8) | proxy frame rate (1 = time-preserving baseline) |
| `CLIP_PROXY_HEIGHT` | 480 (120–720) | proxy long-edge height |
| `CLIP_PROXY_AUDIO_K` | 24 (0–128) | mono audio bitrate kbps; 0 drops audio |
| `CLIP_PROXY_CRF` | 30 (18–40) | x264 quality |
| `CLIP_PROXY_KEY_PREFIX` | `clip-proxy/` | storage prefix for proxy objects |
| `CLIP_PROXY_MAX_BYTES` | 512 MiB | source download cap (Content-Length + streaming) |
| `CLIP_PROXY_FFMPEG_TIMEOUT_MS` | 10 min | hard ffmpeg kill |
| `CLIP_PROXY_FETCH_TIMEOUT_MS` | 120 s | source fetch timeout |

Detection/captioning reuse the platform's Vertex config (`GEMINI_VERTEX_MODEL` via `models().gemini`, service-account auth) — see [generation-and-models.md](generation-and-models.md). Production values and the two must-stay-set gates (`RENDER_ENABLED`, `CLIP_PROXY_URL`) are in [auto-clips-infra.md](auto-clips-infra.md).

## Testing

From the repo root, `npm test` runs `node --test`, which discovers both the app suites in `test/` and the worker suites in `render-worker/` (node_modules excluded). The pipeline's suites:

```bash
node --test test/highlights.test.js test/highlights-video.test.js   # ranker + video detector + selection
node --test test/auto-clips.test.js                                 # buildClipManifest + createAutoClips (captions, VO, ducking, extension)
node --test test/autoclips-proxy.test.js                            # app-side proxy adapter (mock, gate, headers)
node --test test/caption-styles.test.js test/caption-generate.test.js test/caption-manifest.test.js
node --test test/voice-presets.test.js test/story-narrator.test.js
node --test render-worker/clip-proxy.test.js render-worker/clip-proxy-service.test.js
node --test render-worker/dockerfile-copy.test.js                   # the COPY-graph lock
```

All model/DB/render effects are injectable (`analyze`, `generate`, `withClient`, `enqueue`, `fetchImpl`, `spawnImpl`, `uploadBufferImpl`, …), so every suite runs without Vertex, Postgres, ffmpeg, or MinIO. `npm run lint` runs the syntax checker.

## Gotchas

- **Field-name split: `sourceUrl` vs `videoUrl`.** `/api/highlights/proxy` reads `body.sourceUrl` (mirroring `/api/upscale`); `/api/highlights/detect` and `/api/captions/generate` read `body.videoUrl`. As of this writing `public/auto-clips.html` sends `{ videoUrl }` to the *proxy* route, which therefore 400s and drops the page into its manual-dims fallback — the studio's long-video prepare path is broken until that field is renamed (the API path works; the tests use `sourceUrl`).
- **Metering happens before window validation on `/api/auto-clips`.** The route meters `usage:render × min(highlights.length, AUTO_CLIPS_MAX)`, then `createAutoClips` drops invalid windows (non-finite times, sub-1 s clips). A request with junk windows is charged for renders that never enqueue.
- **Captions must be CLIP-relative.** `buildClipManifest` clamps `{startMs,endMs}` to `[0, clipMs]` — source-relative captions attached without rebasing are silently clipped away. The studio's `sliceCaptions` does the rebase; API callers must too.
- **Captions attach AFTER reframe, on their own last track.** They lay out in target dims (a bottom-center box) instead of being cover-scaled with the video, and the last track renders on top. Don't "simplify" by adding them to the base manifest before `reframeManifest`.
- **Asset vs item duration are deliberately independent.** The video asset declares the SOURCE length; the item declares the CLIP length with `videoStartFromInSeconds` as in-point. `buildManifest` ties the two together, which is exactly why `buildClipManifest` hand-builds — don't refactor it onto `buildManifest`.
- **Voiceover extension is clamped to real footage.** A long VO extends the clip but never past the source end (and raises the asset's declared duration to match); removing the clamp reintroduces a silently-wrong frozen/black tail.
- **`speakingRate` is resolved but not spoken.** `resolveVoicePreset` returns each persona's `speakingRate`, but `synthesizeClipVoiceover` passes only `description` + `text` to `designVoice` — pacing currently rides on the description wording alone.
- **`gs://` deliberately bypasses `safePublicUrl`.** A `gs://` URI is read by Vertex (or the worker) directly, never fetched by the API box, so the SSRF guard doesn't apply — but the workspace-binding check (`assetUrlInWorkspace`) still applies to our own asset-proxy URLs. Don't "harden" gs:// through `safePublicUrl` (it would always fail — not https) and don't drop the workspace check.
- **The 1-fps proxy is the timestamp contract.** `fps=1` keeps wall-clock time, so detector timestamps on the proxy map 1:1 onto the original; the proxy also keeps mono audio because multimodal detection listens for reactions. Raising `CLIP_PROXY_FPS` is fine (still time-preserving); stripping audio (`CLIP_PROXY_AUDIO_K=0`) degrades detection quality.
- **Worker-image COPY requirement.** Any new `render-worker/*.js` module imported by a COPYied file must be added to the Dockerfile COPY line; `dockerfile-copy.test.js` fails the build otherwise (this class of miss has shipped a boot-crashing image twice before).
- **Proxy objects are ephemeral but never pruned.** By design no assets row is written for `clip-proxy/<ws>/<hash>.mp4` objects, and no storage sweep exists yet — the prefix grows until one is built.
- **`proxiedDerivative` is load-bearing.** Worker responses carry fleet-internal MinIO URLs a browser can't reach; the route re-points them at the signed asset proxy. Mock/passthrough results (no `key`) are intentionally left untouched.
- **`AUTOCLIPS_TIMEOUT_MS` is shared.** One knob times out both highlight detection and caption generation Vertex calls.
- **Long moments are trimmed, not dropped.** `selectDiverseHighlights` cuts an over-long candidate to `startMs + maxMs` ("keep the hook"); only sub-half-minimum moments are discarded.

## Provenance

Auto-Clips is a clean-room, white-label re-implementation of the *workflow* of an MIT-licensed long-form→shorts pipeline: we took the ideas (semantic moment taxonomy, caption personas, style-matched voice presets, bed ducking, diversity selection) and re-expressed them entirely first-party on our Vertex + Remotion stack with zero new dependencies. The original's GPU heuristic stack (per-frame motion/audio scoring, scene-cut detection), local STT/TTS models, and caption burn-in tooling were all replaced by seams we already own: one Gemini video-understanding call, Google STT v2, Chirp HD voice design, and Remotion caption items rendered by our own worker. Taxonomy, prompts, persona names, and file/route names are ours; no source-project branding appears anywhere in code, UI, or docs. Full seam map and phase history: `planning/AUTO_CLIPS_PILLAR.md`.

## Related docs

- [auto-clips-infra.md](auto-clips-infra.md) — production containers, the two env gates, monitoring, and recovery
- [render-worker.md](render-worker.md) — the render pipeline the clips ride on
- [generation-and-models.md](generation-and-models.md) — Vertex model configuration and seams
- [api-reference.md](api-reference.md) — the wider API surface
- [security.md](security.md) — SSRF guard, asset tokens, workspace binding
- [configuration.md](configuration.md) — platform-wide env reference