# Own The Stack — Self-Hosted, Open-Source, 4K Persona Engine

**Goal:** beat ElevenLabs (voice) and HeyGen (avatar) on *our own infrastructure* with
the best open-source models — highest quality out of the box, 4K-capable, no per-minute
vendor tax, and biometrics that never leave the fleet. Proprietary APIs (Chirp 3, ElevenLabs)
are **optional adapters behind the same seams**, never the foundation.

> This is the engine-selection + architecture spec. The code seams already exist:
> [`src/voice.js`](../../src/voice.js) (`synthesizeVoice`) and the lip-sync provider in
> [`src/personas.js`](../../src/personas.js) (`PERSONA_LIPSYNC_URL`). The render-engine
> adapter ([`src/render-engine.js`](../../src/render-engine.js)) POSTs each job to a
> self-hosted `@remotion/renderer` service and streams back the final 4K master. Every
> engine below plugs into one of those seams — swap an env var, not the code.

> **Live vs. GPU-gated — read this first (updated 2026-07-11).** Every seam and adapter
> below is shipped in code, but only **one** self-hosted lane is *running in production
> today*: **CoreReflex Voice** — Kokoro narration + NeuTTS zero-shot cloning — on the
> CPU box (`VOICE_ENGINE_URL` + `VOICE_KOKORO_URL` are set; because `VOICE_ENGINE` is
> unset, default narration is still paid Chirp — pin `model:'kokoro'` to hit the
> self-hosted engine). Everything else here — the avatar/lip-sync engines, the two
> self-hosted **CoreReflex** video lanes ([generation-and-models.md](generation-and-models.md#the-multi-model-capability-registry-and-self-hosted-lanes-srcmodels-registryjs)),
> self-hosted image, music, and LLM — is `configured:false` and waits on a single
> 24–48 GB **GPU box** (`OPEN-009` in [`planning/roadmap/OPEN_MODELS.md`](../../planning/roadmap/OPEN_MODELS.md)).
> To go live: deploy the inference-worker GPU compose profile + weights, then set the
> `*_URL` / `*_ENABLED` env vars — no app code changes.

## The seam contracts (any engine that speaks these plugs in)

| Stage | Env | HTTP contract | Returns |
|-------|-----|---------------|---------|
| **Voice synth/clone** | `VOICE_ENGINE_URL` | `POST {text, referenceAudio?, speakingRate, languageCode}` | audio bytes (WAV/MP3) |
| **Lip-sync / avatar** | `PERSONA_LIPSYNC_URL` | `POST {imageUrl, audioUrl}` | `{videoUrl, durationInSeconds}` |
| **Render to 4K master** | `RENDER_WORKER_URL` (+ `RENDER_ENABLED=true`) | `POST {compositionId, manifest, hyperframesHtml, codec}` | rendered video bytes |
| **Upscale to 4K** | `UPSCALE_ENGINE_URL` | `POST {videoUrl, scale}` | `{videoUrl}` |

Self-hosted engines run on the GPU fleet behind Cloudflare Tunnel + Access (see
`as-devops-self-host-infra`), exactly like the render adapter's `renderFn`.

**Every URL above is fetched server-side from a user-influenced flow, so the two that
take a user-supplied reference URL (voice + lip-sync) run it through the
[SSRF guard](security.md#ssrf-guard-net-guardjs) (`safePublicUrl()`) first** — a
reference that isn't a public https URL is rejected with a `400`. The render adapter
([`src/render-engine.js`](../../src/render-engine.js)) is the honest export gate: with
no `RENDER_WORKER_URL` it throws a clear, actionable error rather than silently
hanging or faking a master. See [Render Worker](render-worker.md).

## Engine picks (June 2026, license-checked)

### Voice — clone a voice, beat ElevenLabs
| Engine | Quality | License | Use as |
|--------|---------|---------|--------|
| **Fish Speech / Fish Audio S2** | **#1 open** — beats ElevenLabs/Google/OpenAI on EmergentTTS-Eval (81.88%) + Audio Turing Test; 80+ langs; short-sample clone | open weights (verify S2 commercial terms) | **hero quality** |
| **XTTS-v2 (Coqui)** | excellent; 6s zero-shot clone; multilingual | **Coqui Public Model License — commercial OK** | **safe default** |
| **Chatterbox** | beats ElevenLabs 65.3% on listening tests | permissive (MIT-class) | strong alt / fast |
| **VoxCPM (OpenBMB)** | tokenizer-free diffusion-AR TTS on MiniCPM-4; **voice *design* from a text description** + controllable cloning; multilingual | **Apache-2.0 — code + weights, commercial OK** (attribution required; acceptable-use bars impersonation/fraud) | **voice-design lane** — set `VOICE_DESIGN=true` |
| ~~F5-TTS~~ | gorgeous, 3s clone | **CC-BY-NC — NON-commercial** ❌ | **do not ship** (eval only) |

> **Disclosure note (updated 2026-07-09 — `planning/MODEL_DISCLOSURE_PROVENANCE.md`).**
> Self-hosted open models (VoxCPM / XTTS / Fish / Chatterbox / Kokoro …) are presented to
> users as **CoreReflex** custom models — the underlying engine name is not surfaced. Their
> permissive-license attribution is retained as an **INTERNAL NOTICE** (compliance record),
> since Apache/MIT attribution triggers on distributing weights, not on serving inference.
> Paid vendor APIs stay disclosed. What *is* unambiguously ours is the orchestration:
> [`src/voice-design.js`](../../src/voice-design.js) (describe-a-voice → traits → engine
> routing / nearest-named-voice mapping) and the engine-agnostic `synthesizeVoice` seam.

**Decision:** default **XTTS-v2** (commercial-clean, zero-shot from a 6s sample → no per-persona
training step), with **Fish S2** as the quality-max lane once its commercial terms are confirmed.
Zero-shot means "cloning" = storing the persona's 6–10s **reference audio**; no key/training — the
reference rides with each synthesis. (Chirp 3 Instant Custom Voice mints a *key* instead — supported
as an adapter, but it's Google-allow-list gated, so it is never the default.)

> **What actually shipped (and what's live).** The engine picks above are the
> selection rationale; the runtime is a **voice-model registry** ([`src/voice-engines.js`](../../src/voice-engines.js))
> with one row per selectable engine — `corereflex-voice` (`VOICE_ENGINE_URL`, the
> clone/design flagship), `kokoro`, `piper`, `neutts` (clone-only), `chatterbox`,
> `qwen3-tts`, and vendor `chirp-hd`. A request pins one via `model`; self-hosted rows
> bill `usage:voice-own` and are white-labeled to "CoreReflex Voice" by
> `publicVoiceResult` ([`src/model-disclosure.js`](../../src/model-disclosure.js)).
> **Live in prod today:** Kokoro (narration) + NeuTTS (zero-shot cloning) on the CPU
> box. The GPU-class voices (Chatterbox, Qwen3-TTS) and a GPU XTTS/Fish deployment at
> `VOICE_ENGINE_URL` wait on the GPU box. See
> [generation-and-models.md](generation-and-models.md#voice--chirp-3-hd-tts-and-the-corereflex-voice-registry-srcvoicejs).

### Lip-sync / avatar — talk any face, beat HeyGen

> **Full schematic + pipeline:** [avatar-lipsync-engines.md](avatar-lipsync-engines.md) — the two seams, all four wrappers, the gateway topology, the `renderPersonaClip` flow, the consent gate, the mock smoke path, and the bake-off. The summary below is the decision table.

Two distinct seams: **lip-sync** re-lips an existing face (`PERSONA_LIPSYNC_URL`), while a full **avatar** engine drives head motion, expression, and body from a still or a text description (`AVATAR_ENGINE_URL` → the gateway `/avatar` endpoint).

| Engine | Seam | Strength | Use as |
|--------|------|----------|--------|
| **MuseTalk** | lip-sync | near-photorealistic, **real-time 30+ FPS** | **primary** (`PERSONA_LIPSYNC_URL`) |
| **LatentSync** | lip-sync | latent-space, best identity preservation | **hero / high-fidelity** lane |
| Wav2Lip | lip-sync | most accurate raw sync, but low-res | fallback only |
| **LongCat-Avatar** | avatar | full-body, long-form, multi-character, AT2V | **avatar flagship** (`AVATAR_ENGINE=longcat-avatar`) |
| EchoMimic V3 | avatar | lightweight talking-head | avatar default (`echomimic-v3`) |

**Decision:** **MuseTalk** default for lip-sync speed+quality, hero personas → **LatentSync**; for the full-motion avatar seam, **LongCat-Avatar** is the flagship over the default EchoMimic V3.

**The avatar seam upgrade (LongCat-Avatar).** Setting `AVATAR_ENGINE=longcat-avatar` (both engines share `AVATAR_BACKEND_URL` at the gateway) unlocks capabilities validated at the gateway's `/avatar` endpoint ([`inference-worker/gateway.js`](../../inference-worker/gateway.js)):

- **AT2V** — pass a text `prompt` instead of `imageUrl` and the model *invents* the character (no reference still needed).
- **Long-form `segments`** (1–20) — segment-chained generation with reference-frame identity anchoring across segments.
- **Multi-character `characters[]`** (1–4, each its own `imageUrl` + `audioUrl`), with `audioMode` `merge` (overlap) or `concat` (sequence).
- **Tiers** — `distill` (fewer sampler steps) + `quantization:'int8'`, and `resolution` `480p` / `720p`.

The app forwards the long-form knobs (`resolution` / `segments` / `distill` / `quantization`) through `renderPersonaClip` ([`src/personas.js`](../../src/personas.js)); the engine catalog (`AVATAR_ENGINE_CATALOG`) keeps the LongCat/EchoMimic attribution as an **internal NOTICE** only. **All four talking-head engines now ship first-party GPU-only FastAPI wrappers** under [`inference-worker/wrappers/`](../../inference-worker/wrappers/): `musetalk/` + `latentsync/` (lip-sync seam), `echomimic/` + `longcat-avatar/` (avatar seam) — each a `POST /generate` speaking the canonical body, with weights pulled by `scripts/download-models.sh` (`DOWNLOAD_MUSETALK` / `DOWNLOAD_LATENTSYNC` / `DOWNLOAD_ECHOMIMIC` / `DOWNLOAD_LONGCAT_AVATAR`) and real `build:` stanzas in `docker-compose.yml`. Full request contract: [`docs/INFERENCE-GATEWAY.md`](../INFERENCE-GATEWAY.md); the pipeline schematic: [avatar-lipsync-engines.md](avatar-lipsync-engines.md).

### 4K — native where possible, upscale always available
- **Generate:** Kling = native-4K; Veo = native-1080p (see [`generation-and-models.md`](generation-and-models.md)
  for the honest native-vs-upscaled stamping).
- **Save masters at 1080p**, then **upscale to 4K on demand** with **Real-ESRGAN** (fast, proven) or
  **SeedVR2** (SOTA detail) — the governed "Master/Enhance" stage already specced in
  [`../UPSCALING-8K.md`](../UPSCALING-8K.md). Talking-head clips: MuseTalk renders ~1080p → upscale to 4K.
- Provenance must stamp `upscaled-4k` vs `native-4k` — never mislabel (it's the honesty rule).

## The marriage — full persona clip pipeline
```
Persona Studio (consent):  record 6–10s reference audio + spoken consent
   → store reference on the persona (zero-shot)  |  or mint a Chirp key (adapter)
renderPersonaClip(text):
   → synthesizeVoice(text, voiceClone)            → cloned audio   (XTTS / Fish / Chirp)
   → lipsync(primaryReferenceImage, clonedAudio)  → 1080p talking head (MuseTalk / LatentSync)
   → [optional] upscale to 4K                     → 4K master (Real-ESRGAN / SeedVR2)
   → land as persona_clip → render worker masters/encodes
```
Every arrow is an existing seam. The persona's hard **consent gate** stays in front of all
of it — `renderPersonaClip` and `cloneVoice` both `403` until `consent_status === 'granted'`,
and Chirp key-minting requires a spoken consent recording. See
[Consent gates](security.md#consent-gates-biometrics).

## Why this beats the incumbents
- **Cost:** open-weight self-hosting is **50–200× cheaper per minute** than ElevenLabs/HeyGen.
- **Quality:** Fish S2 already wins blind evals vs ElevenLabs; MuseTalk/LatentSync match commercial lip-sync.
- **Ownership & trust:** voice keys + face data never leave our fleet — the white-label / consent-first thesis.
- **Composability:** the same cloned persona narrates a **VSL**, hosts an **explainer**, fronts a **reel/short**,
  or narrates a **documentary** — because voice + avatar are just two more nodes on the existing timeline.

## Build order
1. **Voice seam** — `synthesizeVoice` routes to a self-hosted engine (`VOICE_ENGINE_URL`, default open
   XTTS) **or** the Chirp custom-voice adapter, by config. Persona stores reference audio (or Chirp key). ✅ *shipped*
2. **Render-engine adapter** — `render-engine.js` (`renderFn`) POSTs jobs to `RENDER_WORKER_URL` and
   streams back the master; honest `RENDER_ENABLED` gate. ✅ *shipped* (deploy of the GPU service pending)
3. **Stand up XTTS-v2 + MuseTalk + the Remotion renderer** on the GPU fleet behind their URLs (fleet session, GPU box).
4. **Upscale stage** — `UPSCALE_ENGINE_URL` (Real-ESRGAN) wired into the render worker's success path.
5. **Persona Studio capture** — record reference + consent audio in-browser; call the consent-gated clone endpoint.
6. Confirm `native-4k` vs `upscaled-4k` provenance end-to-end.

## Roadmap / what's next

All the seams are code-complete; the remaining work is **infra, not app code**. The
single unlock for every self-hosted *generation* lane (avatar, the two CoreReflex
video lanes, image, music, LLM) is renting the **24–48 GB GPU box** — tracked as
**`OPEN-009`** in [`planning/roadmap/OPEN_MODELS.md`](../../planning/roadmap/OPEN_MODELS.md).
CoreReflex Voice (Kokoro/NeuTTS) is already live on the CPU box. Once the GPU box is
up: deploy the inference-worker GPU compose profile + weights (~27 GB for the LongCat
pair), then set `SELFHOST_VIDEO_URL`, `AVATAR_ENGINE_URL` / `AVATAR_BACKEND_URL`,
`LONGCAT_VIDEO_ENABLED=true`, and the remaining `VOICE_*_URL` / `UPSCALE_URL` vars.
For the video generation router and the two self-hosted lanes, see
[generation-and-models.md](generation-and-models.md#the-multi-model-capability-registry-and-self-hosted-lanes-srcmodels-registryjs).