# Deploy & Operations

How CoreReflex's four deployables get to production, the env gates that keep them
honest, how to verify a deploy actually landed, and the order-of-operations rules
that prevent deploy-skew breakage.

**On this page**

- [The four deployables](#the-four-deployables)
- [How each one deploys](#how-each-one-deploys)
- [The two env gates](#the-two-env-gates)
- [Editing Coolify env (token minting)](#editing-coolify-env-token-minting)
- [Self-healing and monitoring](#self-healing-and-monitoring)
- [Deploy verification](#deploy-verification)
- [Verification gotchas (test/build gates)](#verification-gotchas-testbuild-gates)
- [Order-of-operations rules](#order-of-operations-rules)
- [Applying migrations](#applying-migrations)
- [Launch operations](#launch-operations)
- [Owner approval gates](#owner-approval-gates)
- [Assets & rendering](#assets--rendering)

CoreReflex runs self-hosted on the Girard Media fleet. The API and editor are
Coolify-managed Docker apps; the render worker and clip proxy are manual Docker
containers on the App box. Shared infrastructure: **Postgres** and **MinIO / S3
object storage** on the Data box, reachable only over the private mesh (or
Cloudflare R2 — see [Configuration](configuration.md)). Exact credentials live in
DevOps, not here.

## The four deployables

| # | Deployable | What it serves | Source | How it's deployed |
|---|------------|----------------|--------|-------------------|
| 1 | **API / app** | `corereflex.com` — Node server, dashboard, marketing pages, all `/api/*` | [`Dockerfile`](../../Dockerfile), `src/` | Coolify app (id 15), **auto-deploys on push to `main`** |
| 2 | **Editor studio** | `editor.corereflex.com` — the Remotion SPA (static build served by nginx) | [`editor/Dockerfile`](../../editor/Dockerfile), `editor/` | Coolify app, auto-deploys on push (separate, slower Vite build) |
| 3 | **Render worker** | drains `render_jobs` → renders MP4s with `@remotion/renderer` → MinIO | [`render-worker/Dockerfile`](../../render-worker/Dockerfile), `render-worker/` + `editor/src` | **Manual `docker run`** on the App box (`corereflex-render-worker`, `--restart unless-stopped`). Coolify app 19 exists but is **broken** — see below. |
| 4 | **Clip proxy** | `POST /clip-proxy` on `:9099` — the long-video prepare service for Auto-Clips | `render-worker/clip-proxy{.js,-service.js}` + `storage.js` (zero-dep) | **Manual `docker run`** on the App box (`corereflex-clip-proxy`); code lives on-box in `/opt/clip-proxy` |

> **Why the worker is manual:** the render worker is a queue poller with **no HTTP
> port**. Coolify app 19 was configured with `ports_exposes=3000`, so Coolify
> health-checks a port the poller doesn't serve and kills the container
> (`exited:unhealthy`). Until app 19's healthcheck is fixed in a maintenance
> window, the manual container is the *proven-working* render path — don't disrupt
> it casually. Fleet-level detail (container names, on-box paths, recovery
> quick-reference) lives in [auto-clips-infra.md](auto-clips-infra.md).

Two render-worker implementations exist in the repo. **Prod runs the standalone
`render-worker/` container** (`render-worker/worker.js` polls `render_jobs` and
renders in-process with `@remotion/renderer`). The alternative in-process drain
loop (`npm run worker` = `src/render-worker.js` + the `RENDER_WORKER_URL` engine
adapter) is documented in [render-worker.md](render-worker.md); it shares the same
`render_jobs` queue contract.

## How each one deploys

### API app and editor (Coolify, automatic)

Push to `main` → Coolify builds and deploys both apps from their Dockerfiles.

- **API app**: `node:20-alpine`, `npm ci --omit=dev`, `npm start`
  (`node src/server.js`), container healthcheck hits `/healthz`. Typically live
  in ~30s–4min.
- **Editor**: two-stage build — `node:24-alpine` runs `npm run build`
  (React Router v8 / Vite; needs Node ≥ 22 at build time), then `nginx:alpine`
  serves `build/client` with the SPA fallback and security headers from
  [`editor/nginx.conf`](../../editor/nginx.conf). The API base URL is inlined at
  build time via the `VITE_CORE_API_BASE_URL` build arg (default
  `https://corereflex.com`).

Rapid back-to-back pushes queue in Coolify — be patient; it's usually not
stalled. For a *true* stall, see [Deploy verification](#deploy-verification).

### Render worker (manual container)

The image bundles the **same** editor Remotion composition the browser renders
(`editor/src/remotion/index.ts`, comp id `Main`), so the build context must be
the **repo root**:

```
docker build -f render-worker/Dockerfile -t corereflex-render-worker .
```

Then recreate the container on the App box on the `coolify` Docker network with
`--restart unless-stopped` and the worker env (`DATABASE_URL`, the `ASSET_*`
storage vars pointing at fleet MinIO, `RENDER_SHARED_SECRET`). It is CPU-bound:
give it several vCPUs and ≥ 4 GB RAM; GPU/NVENC is optional. The worker builds
the Remotion bundle **once per process** and reuses it across jobs — which is
exactly why it must be redeployed after editor changes (see
[Order-of-operations rules](#order-of-operations-rules)).

The same image also hosts the opt-in sidecar HTTP services (`AUDIO_FX_PORT`,
`UPSCALE_PORT`, `REMOVE_BG_PORT`, `VECTORIZE_PORT`, `CLIP_PROXY_PORT`), all gated
by the same `RENDER_SHARED_SECRET`.

### Clip proxy (manual container)

The fourth deployable is a dedicated zero-dependency container running the
clip-proxy HTTP service on `:9099` (it builds a time-preserving 1-fps low-res
proxy so long videos fit the model's inline size cap for highlight detection).
Its code is the three zero-dep files from `render-worker/` synced to
`/opt/clip-proxy` on the App box; `/opt/clip-proxy/run.sh` recreates it, and code
updates are "re-sync the changed file + `docker restart corereflex-clip-proxy`".
Full runbook: [auto-clips-infra.md](auto-clips-infra.md).

## The two env gates

Two env vars on the **API app** (Coolify env store — durable across redeploys)
gate the render pipeline. Both fail *honestly* (a clear 503, never a silent
hang) when unset:

1. **`RENDER_ENABLED=true`** — required for `enqueueRender` to accept jobs.
   Without it, `POST /api/render` and `POST /api/auto-clips` return 503 and
   nothing is ever queued that no worker can drain. This is durable in the
   Coolify env store; it must stay set now that the worker is live.
2. **`CLIP_PROXY_URL=http://corereflex-clip-proxy:9099`** — required for
   `POST /api/highlights/proxy` (long-video prepare). Without it the route 503s
   with an actionable message (`src/autoclips-proxy.js`).

Coupled to gate 2: the API's **`RENDER_SHARED_SECRET` must match** the value in
the clip-proxy's env (`/opt/clip-proxy.env` on the box). A mismatch shows up as
the clip proxy returning 401 to the API. The clip-proxy service **refuses to
start** without a secret — it would otherwise be an unauthenticated server-side
fetcher.

Symptom → fix mapping for both gates is in the
[recovery quick-reference](auto-clips-infra.md#recovery-quick-reference).

## Editing Coolify env (token minting)

There is deliberately **no Coolify API token stored on-box**. To change API-app
env (the gates above, `BUILD_VERSION`, etc.) via the REST API, mint a short-lived
token inside the Coolify container, use it against `localhost:8000/api/v1`, then
**revoke it**. The exact `php artisan tinker` procedure (including the required
`team_id` on the PAT) is documented in
[auto-clips-infra.md](auto-clips-infra.md#two-gates-that-must-stay-set).

## Self-healing and monitoring

- Both manual containers run with `--restart unless-stopped`, so crashes and box
  reboots self-heal.
- **`/opt/clip-proxy/monitor.sh`** runs from cron every 5 minutes on the App
  box: it **recreates** a DOWN clip-proxy, **restarts** a HUNG one (3-second
  `/health` timeout), and logs render-worker outages to
  `/var/log/autoclips-monitor.log`. The monitor is what catches silent hangs the
  Docker restart policy can't see.
- Follow-up (not built): email alerting on repeated monitor failures.

## Deploy verification

Never assume a push deployed. Prove it:

```
curl -s https://corereflex.com/healthz
curl -s https://corereflex.com/api/health
```

- **`/healthz`** proves the Node process is serving.
- **`/api/health`** is the readiness gate: it reports `version` plus database
  connectivity, storage reachability, and auth/JWT configuration, and only
  returns `ok: true` when all three pass.

### `version` is env-first

`BUILD_VERSION` in `src/api.js` is
`process.env.BUILD_VERSION || '<fallback constant>'` — **the env var wins**. Set
`BUILD_VERSION` in Coolify to the deploy's git SHA so `/api/health` reflects the
real build automatically; the in-code constant is only the fallback when no
stamp is provided. If `version` still shows the old value after a deploy, the
deploy hasn't rolled.

The editor has no version endpoint; its Vite **manifest hash flips on rebuild**,
which is the equivalent proof.

### Empty-commit re-trigger

Coolify's build queue can genuinely stall on rapid back-to-back pushes (stuck on
an old commit). The reliable unstick:

```
git commit --allow-empty -m "chore: retrigger deploy" && git push
```

## Verification gotchas (test/build gates)

These have each masked a real regression at least once — treat them as law:

- **Run `npm test`, never `node --test test/`.** Under Node 25,
  `node --test test/` silently imports the directory and reports a bogus
  "1 test passed". `npm test` (`node --test` auto-discovery) is the gate. The
  render worker has its own suite: `npm --prefix render-worker test`.
- **The editor has no test runner.** Its gate is *both*:
  `npm --prefix editor run typecheck` **and** `npm --prefix editor run build`.
  The editor build requires Node ≥ 22.
- **Prod DB is only reachable via the SSH tunnel.** Postgres, PgBouncer, and
  MinIO live on the fleet Data box behind a private mesh — never exposed
  publicly. Run [`scripts/db-tunnel.sh`](../../scripts/db-tunnel.sh) (maps
  `localhost:55432` → direct Postgres, `:56432` → PgBouncer TLS, `:59000` →
  MinIO) and keep it open while working. From *on the fleet*, an alternative is
  a `node --input-type=module` one-liner inside the worker container — strip
  `sslmode` from `DATABASE_URL` and pass `ssl:{rejectUnauthorized:false}`.

## Order-of-operations rules

Standing rules that prevent deploy-skew incidents. All three have bitten in prod.

1. **Redeploy the render worker after ANY `editor/src` change.** The worker
   image copies `editor/src` and bundles the composition at startup, so editor
   changes don't reach renders until the image is rebuilt and the container
   recreated — otherwise the exported MP4 stops matching the in-editor preview.
2. **Worker before editor** for manifest-shape changes. The audio-session
   manifest split ([`src/audio-sessions.js`](../../src/audio-sessions.js),
   mirrored in `render-worker/props-from-manifest.js`) was shipped with the
   worker-side session filter deployed **before** any editor build that can save
   audio sessions. The at-rest split keeps even a stale worker safe by
   construction, but the belt-and-suspenders filters must land worker-first.
   Apply the same order to any future change in what the editor persists into
   `manifest`.
3. **Remotion lockstep.** `editor/package.json` and `render-worker/package.json`
   must pin the **exact same** Remotion version (currently `4.0.483`). A
   4.0.463 ≠ 4.0.483 drift broke every video render containing an Interactive
   export (bundle failure) in the 2026-07-02 incident. Check both files before
   rebuilding the worker image; `render-worker/dockerfile-copy.test.js` guards
   the file list, not the version.

Because the worker is currently a manual container (not Coolify-managed), rules
1 and 3 are **operator discipline** — worker deploys do not track pushes. That
is the main argument for fixing Coolify app 19.

## Applying migrations

Migrations are migrator-owned and applied out-of-band, never on app boot:

```
bash scripts/db-tunnel.sh     # open the SSH tunnel to prod Postgres (background it)
npm run db:migrate            # scripts/migrate.mjs — applies pending 00N_*.sql
```

`migrate.mjs` connects with `DATABASE_MIGRATION_URL` (the direct-Postgres tunnel
port, not PgBouncer), records applied files in `schema_migrations` (keyed on
filename sans `.sql`), wraps each file in `begin/commit`, and is idempotent. See
[Data Model](data-model.md) for the migration rules.

**Confirm the next free migration number before authoring one** — a concurrent
build loop can also author migrations on the same tree, and a number collision
blocks the runner.

## Launch operations

Selected rollout shape:

| Area | Decision |
|------|----------|
| Deployment target | Coolify Docker apps + manual worker containers on the private fleet |
| App domains | `corereflex.com`, `editor.corereflex.com` |
| Database | Postgres 18; app via PgBouncer TLS; migrations via direct Postgres URL |
| Storage | MinIO `corereflex-assets` by default; Cloudflare R2 supported |
| Queueing | `render_jobs` table drained by the `corereflex-render-worker` container |
| Observability | `/healthz`, `/api/health`, God Mode metrics/audit log, worker logs, `monitor.sh` log |
| Cost controls | Billing inert by default; credit metering only with `BILLING_ENABLED=true` |
| Asset URL strategy | signed app proxy unless a public asset domain is approved |

God Mode is the admin-only launch dashboard: profitability estimate, credit
usage, analytics, queued render cost, operational health links, support
impersonation, and approval gates are not exposed to Production or Beta Preview
users.

## Owner approval gates

These items are tracked but not enabled without owner approval:

- MinIO backup schedule and restore drill.
- Postgres PITR/WAL archiving.
- `pg_stat_statements` installation and retention policy.
- Public asset domain for `ASSET_PUBLIC_BASE_URL`.
- Any change that exposes database or object storage outside the private fleet.

## Assets & rendering

Generated footage and rendered masters land on MinIO/R2 and are served back per
[`../ASSET-SERVING.md`](../ASSET-SERVING.md). The render worker polls
`render_jobs`, renders with Remotion (+ system ffmpeg for the audio-FX pass),
uploads the master, inserts the workspace-scoped `assets` row, and marks the job
done. The Auto-Clips pipeline (detect → cut → render → captions/voice → library)
rides this same queue; its infra runbook is
[auto-clips-infra.md](auto-clips-infra.md).

## HostSSH `.223` — what is bind-mounted vs image-baked

Live production is the **hostssh** fleet on `.223` (not Coolify — see
[`planning/HOSTSSH_MIGRATION.md`](../../planning/HOSTSSH_MIGRATION.md)). Deploy
shape differs by container; getting this wrong is how a "recreate" silently
ships stale code:

| Container | Code path | Update |
|---|---|---|
| **App** (`corereflex-app`) | Image-baked from repo via `/opt/corereflex/deploy.sh` | Rebuild + health-gated swap (`deploy.sh <sha>`) |
| **Editor** (`editor-corereflex`) | Image-baked (`editor/Dockerfile`) | `scripts/hostssh-deploy-editor.sh` (app deploy excludes `editor/`) |
| **Clip-proxy** | **Bind-mounted** from `/opt/clip-proxy` | Copy files → `docker restart` (or `run.sh`). Backup first: `/opt/clip-proxy-backup-<stamp>` |
| **Inference gateway** (`corereflex-inference`) | **Image-baked** from `inference-worker/` | Rebuild image (`hostssh/corereflex-inference:<sha>`), verify the layer carries the change, then swap with health poll + rollback tag. A recreate of the *old* image does nothing. |
| **Engine wrappers** (qwen, whisper, …) | Image-baked | Rebuild that wrapper; see `scripts/hostssh-recreate-qwen.sh` for the CPU-quota pattern |

Gateway rollback retained after the `ed13e1f` cutover:
`hostssh/corereflex-inference:fe4edb9`.

## Operational gotchas (summary)

- **Coolify can stall** on rapid pushes — empty-commit re-trigger.
- **`BUILD_VERSION` / `SOURCE_COMMIT` env stamp** on every API deploy, or the
  health check isn't a real signal (on `.223` verify inside the container when
  the stamp is hand-set).
- **Redeploy the worker** after editor changes; **worker-before-editor** for
  manifest-shape changes; **Remotion lockstep** across both package.json files.
- **`RENDER_ENABLED` + `CLIP_PROXY_URL` + matching `RENDER_SHARED_SECRET`** must
  stay set on the API app or the pipeline 503s.
- **Gateway ≠ clip-proxy:** gateway source is baked into the image; clip-proxy
  is bind-mounted. Do not assume recreate-only updates either.
- **Migration runner state** must be reconciled with prod before applying new
  migrations (a missing row blocks the runner).
- Keep GitHub Actions removed unless the owner explicitly reverses the local
  cost-control policy.

Related: [Architecture](architecture.md) · [Configuration](configuration.md) ·
[Render Worker](render-worker.md) · [Auto-Clips infra](auto-clips-infra.md) ·
[Data Model](data-model.md).