# Auto-Clips — production infrastructure & recovery runbook

The Auto-Clips pipeline (long video → detect → cut → render → captions/voice → library)
is live in production. This documents the fleet state, the render + long-video-prepare
services, and how to recover them. Verified 2026-07-06.

## Where it runs

Everything is on the **App box `31.220.104.207`** (Coolify), sharing the `coolify`
docker network. Storage = MinIO `http://10.10.0.2:9000` bucket `corereflex-assets`.

| Piece | Container | How it's deployed | Notes |
|---|---|---|---|
| API (corereflex.com) | Coolify app 15 (`mss09tlxgb755mif9f3r9sgw`) | Coolify, git `main` auto-deploy | `RENDER_ENABLED=true` durable in Coolify env store |
| Render worker | `corereflex-render-worker` | **manual `docker run`** (image `corereflex-render-worker:latest`) | polls `render_jobs`, bundles Remotion, ffmpeg → MinIO. `--restart unless-stopped`. |
| Long-video prepare | `corereflex-clip-proxy` | **manual `docker run`** (reuses the worker image) | the AC1.5 clip-proxy HTTP service on `:9099`; zero-dep; code in `/opt/clip-proxy`. |

## Two gates that must stay set

1. `RENDER_ENABLED=true` on the API app (Coolify env store) → `enqueueRender` works.
   Without it `/api/auto-clips` returns an honest 503 and clips never render.
2. `CLIP_PROXY_URL=http://corereflex-clip-proxy:9099` on the API app → `/api/highlights/proxy`
   works (long-video prepare). The API's `RENDER_SHARED_SECRET` must match the value in
   `/opt/clip-proxy.env` (both are the same `9d54…` today).

To edit these you need a Coolify API token. There's none stored on-box — mint one via
`docker exec coolify php artisan tinker` (Coolify requires `team_id` on the PAT; use
`$u->teams()->first()->id`), use the REST API on `localhost:8000/api/v1`, then **revoke it**.

## The clip-proxy container (long-video prepare)

Code lives in `/opt/clip-proxy` (the 3 zero-dep files from `render-worker/` —
`clip-proxy.js`, `clip-proxy-service.js`, `storage.js` — plus `entry.mjs` +
`package.json{"type":"module"}`). Env in `/opt/clip-proxy.env` (ASSET_* copied from the
worker + `CLIP_PROXY_PORT=9099` + `RENDER_SHARED_SECRET` = the API's value).

- **Recreate:** `/opt/clip-proxy/run.sh` (one command; recreates the container).
- **Update code:** re-sync the changed file from the repo's `render-worker/` into
  `/opt/clip-proxy/`, then `docker restart corereflex-clip-proxy` (it mounts the dir).
- **Health:** from any coolify-net container: `fetch('http://corereflex-clip-proxy:9099/health')`
  → `{ok:true,service:"clip-proxy"}`. POST `/clip-proxy` requires `x-render-token`.

### Page-URL ingestion (YouTube etc.) — yt-dlp

A watch-page URL is HTML, not media, so ffprobe rejects it with `Invalid data found
when processing input` (this is what a pasted YouTube link produced on 2026-07-26).
`clip-proxy/extractor.js` routes **allowlisted** platform hosts through `yt-dlp`
instead of the direct fetch; everything else keeps the original redirect-refusing,
byte-capped fetch.

- **The allowlist is the SSRF containment.** yt-dlp follows redirects and pulls from
  CDN hosts we never validate, which bypasses the `redirect: 'manual'` guard on the
  direct path. It only ever runs for hosts in `EXTRACTOR_HOSTS`, so it can't be aimed
  at a metadata/link-local address. **Never widen that list to a wildcard.**
- **Install (no image rebuild):** `clip-proxy/install-ytdlp.sh` — downloads the
  standalone Linux binary to `/opt/clip-proxy/bin/yt-dlp` (checksum-verified against
  the release `SHA2-256SUMS`), appends `YTDLP_PATH` to `/opt/clip-proxy.env`, then
  `run.sh` to recreate. Idempotent.
- **Why not just rebuild:** the clip-proxy runs the *render-worker* image, which also
  serves live renders. `render-worker/Dockerfile` now installs yt-dlp so future builds
  carry it, but the bind mount is the safe channel for this service.
- **If yt-dlp is missing** the service says so by name (`yt-dlp not found — …`) rather
  than leaking `ENOENT`; direct media URLs are unaffected.
- **Download size is bounded** by `--max-filesize` and a `height<=` format cap.

### Two artifacts, two quality rules

House rule is **4K target, 1080p floor, nothing sub-floor user-visible**. That applies
to *one* of the two things the clip-proxy produces, and conflating them is a live trap:

| Artifact | Route | Quality rule |
|---|---|---|
| **Ingested master** — what shorts are actually cut from, stored under `sources/<ws>/` | `POST /ingest` | **Obeys the floor.** Pulls best available up to 4K (`CLIP_PROXY_INGEST_MAX_HEIGHT`). |
| **Analysis proxy** — 1-fps, 480p, ephemeral, no assets row, only the detector sees it | `POST /clip-proxy` | **Exempt by design.** Pulling 4K to build a 480p proxy is pure waste. |

For an uploaded library video the master already exists, so only the proxy is built.
For a **page URL there is no other copy** — so ingest runs first and its output becomes
`source.url` for detect *and* render. Without that step the render worker would be
handed an HTML page as a video asset (the same failure as before, deferred to bake
time), and whatever we grabbed for analysis would silently become the master.

- `belowFloor` in the ingest response is decided on **measured** pixels (ffprobe), never
  on the requested cap. The studio raises a low-quality alert and offers the existing
  4K upscaler (`POST /api/upscale`) as an opt-in remedy; the upscaled asset then becomes
  the master. A caller cannot request a sub-floor master — `maxHeight` clamps up to the
  floor.
- Ingest has a much larger byte cap than the proxy (`CLIP_PROXY_INGEST_MAX_BYTES`,
  default 8 GiB vs the proxy's 512 MiB) and **streams** disk→storage via `uploadFile`
  rather than buffering, because .223 is memory-critical and an OOM here takes out other
  tenants.
- App route `POST /api/highlights/ingest` — rate-limited tighter than prepare (3/min:
  each call can pull gigabytes and hold a worker slot).
- **Studio** (`public/auto-clips.html`) and the **editor Auto Clips panel**
  (`editor/src/editor/auto-clips/`) both call ingest before proxy/detect/bake when
  the source is a watch page. Never hand a page URL to Remotion or ffprobe — that
  is the "render hole" (HTML as a video asset).

## Self-healing + monitoring

`/opt/clip-proxy/monitor.sh` (cron `*/5`) recreates a DOWN clip-proxy and restarts a
HUNG one (3s `/health` timeout), and logs render-worker outages to
`/var/log/autoclips-monitor.log`. Both containers are `--restart unless-stopped`, so
crashes + reboots self-heal; the monitor catches silent hangs. *Follow-up:* wire an
email alert (the box runs `usermails-mta`/postfix) on repeated failures.

## Known issue — the broken Coolify worker app

Coolify **app 19** (`zhuolcizgvp0gc6i47uzz5ax`, "corereflex-render-worker") is
`exited:unhealthy` because its `ports_exposes=3000` makes Coolify health-check a port
the *poller* doesn't serve, so Coolify kills it — which is why the worker runs as a
manual container instead. To make the worker Coolify-managed again: fix app 19's
config (clear `ports_exposes` / disable the HTTP healthcheck so a background poller
isn't killed), then cut over from the manual container (stop it to avoid the name
clash) and redeploy app 19 from `main`. Do this in a maintenance window — the manual
worker is the *proven-working* render path; don't disrupt it casually. The same
applies to folding the clip-proxy into the worker (it's `CLIP_PROXY_PORT`-gated there).

## Recovery quick-reference

| Symptom | Fix |
|---|---|
| Renders stuck `queued` | worker down → `docker start`/`run` it; confirm `RENDER_ENABLED=true` on the API |
| `/api/auto-clips` 503 | `RENDER_ENABLED` not set on the API Coolify env → set it + restart app 15 |
| `/api/highlights/proxy` 503 | `CLIP_PROXY_URL` not set on the API, or clip-proxy down → `/opt/clip-proxy/run.sh` |
| clip-proxy 401 for the API | secret mismatch → make `/opt/clip-proxy.env` `RENDER_SHARED_SECRET` == the API's |
| box rebuilt, clip-proxy gone | restore `/opt/clip-proxy/` from `render-worker/` + `run.sh` |