The pipeline at a glance
One generation run maps to the roles of a real production studio. Each step has a clear owner, a clear input, and a clear output — and crucially each step happens once, in order.
Prompt
→
Creative Director
→
Storyboard Builder
→
Media
→
Codegen
→
Critique
The video model is codegen. A scene is one generated component
(
scene_scene_NNN) of HTML + CSS + GSAP. Captions, cards, and buttons
are just markup inside that source — there are no data-driven "component properties" to edit. Library
components can be embedded via <component> tags, but the scene owns its own composition.
Generation pipeline — src/llm/pipeline.ts → runUnifiedPipeline
From a prompt to live, playable scenes. Rendering to a final MP4 is a separate step (the render tool).
0
Prompt you
What you want — usually just a prompt, optionally with audience, tone, key points,
target duration, do-not-include, and available assets.
input: a single
prompt string — say what you want; the Creative Director fills the rest.1
Creative Director director creative-director.ts
Reads the raw prompt and, as the expert, commits to one idea. On a thin prompt
("a launch video for this feature") it fills the gaps with expertise; on a detailed prompt it honors the
specifics. It also decides the scene count. There is no separate "expand/develop" pre-step — the
director sees your actual words.
in: prompt out:
Treatment — concept, storytelling
pattern, visual through-line, emotional arc, color/type/motion commitments, scene count.2
Storyboard Builder storyboard artist storyboard-builder.ts
Turns the treatment into a scene-by-scene breakdown. The treatment is injected into its
prompt, so every scene serves the one idea.
in: prompt + treatment out:
Storyboard — narrative +
StoryboardScene[] (per scene: purpose, visual notes, voiceover script, suggested
components, b-roll query or hero-image prompt, duration, transition).3
Media art dept media-enrichment.ts · image-enrichment.ts
Fetches/generates the imagery the storyboard calls for: AI hero images, and cinematic
stock b-roll (a scene's
broll_query → Pexels). Runs before codegen so the animator
can place real media.in: storyboard out: assets on disk + URLs handed to codegen.
4
Codegen animator / the shoot scene-generator.ts · agentic-codegen.ts
Turns each storyboard scene into real HTML/CSS/GSAP (a
scene_scene_NNN
component). buildCodegenSpec surfaces the scene's Purpose and
Visual Notes verbatim, plus voiceover and any library components to embed.in: a
StoryboardScene + media URLs out: a live, animated
scene with a __MP_TIMELINE and __MP_READY handshake.5
Critique editor multi-pass-critiquer.ts · editorial-vision.ts
A vision pass over a tiled storyboard of rendered frames: did each scene actually deliver
its intended outcome? Scenes that clearly miss get re-shot (re-codegen). Fast deterministic gates
(runtime + legibility) also run.
in: rendered frames + the storyboard's intent out: fixes for failing scenes.
6
Audio & Save audio/scene-voiceover.ts · persistence/project.ts
Generates per-scene voiceover (TTS) and background music when requested, then persists the
full project (prompt, treatment, storyboard, scenes, assets, audio).
Single pass, in order: Prompt → Director → Storyboard → Media → Codegen → Critique. Media runs
before codegen (so codegen can place the b-roll/hero). The only loop is Critique → re-codegen a
failing scene. Rendering to MP4 is the separate
render tool.
Generate modes & project lifecycle
The generate tool has two modes (rendering is always the separate render tool).
| Mode | What it does |
|---|---|
storyboard | Run the Creative Director + Storyboard Builder, save the storyboard, and stop for review. |
full (default) | Produce the scenes. Smart about the storyboard: if the project already has one (you reviewed/edited it), build from that; otherwise create the storyboard first, then build. |
Lifecycle (ProjectStatus)
draft→ storyboard→ generated→ rendering→ rendered (failed on error)
- draft — created, nothing produced.
- storyboard — storyboard exists, scenes not built (the review gate).
- generated — scenes (code/copy) built; no final video yet.
- rendering / rendered — MP4 in progress / done.
Studio — /studio · src/preview-app/preview-app.ts
The direct-manipulation editor. Loads a project's scenes into a live composite preview (one document, all scenes, a master GSAP timeline) and lets you fix things by pointing at them.
Revise surgical
Pause, click an element (or pick "Whole scene"), say what to change. A SEARCH/REPLACE
patch (
reviseComponent) edits the scene's source and hot-swaps it in place. Fast.scope: This element · Whole scene api:
POST /api/reviseRegenerate rebuild
Rebuild a scene from scratch to fulfill its storyboard entry — the generate + critique
path run as an async job with live progress. For scenes a surgical revise can't fix.
api:
POST /api/regenerate (job) uses the saved storyboard verbatim.Storyboard editor
The left panel edits the scene's StoryboardScene directly
(Purpose, Script, Visual notes, Duration, B-roll, Hero image, Components) — saved via
POST /api/storyboard-scene. Edit → Save → Regenerate rebuilds from your edits, verbatim
(no re-interpretation). Fast gates (runtime + legibility) run on every revise/regenerate.
Playground (
/playground) is a separate tool for authoring reusable
library components — distinct from Studio, which edits a generated video project.Render pipeline — src/core/render.ts
Scenes → MP4. Driven by the render tool as an async job.
- Scene assembly scene-assembler.ts — resolve components + assets into one HTML doc per scene.
- Frame capture capture.ts · capture-worker.ts — a pooled headless Chromium seeks the GSAP timeline and grabs frames (waits on
__MP_READY+ fonts + media). - Encode encode.ts — frames → per-scene video (ffmpeg).
- Transitions transitions.ts — cross-scene transitions.
- Concatenate — stitch scenes into one track.
- Audio mix audio/mixer.ts — voiceover + music with ducking.
Data model — src/core/types.ts
A Project carries the whole lifecycle. The three creative artifacts are distinct:
the Prompt (the ask) → the Treatment (the vision) → the Storyboard (the scene breakdown) →
the realized Scenes (codegen).
| Type | What it is |
|---|---|
prompt | The input — what you want. Stored flat as project.prompt. |
Treatment | The Creative Director's vision: concept, pattern, through-line, emotional arc, visual style, scene count. project.treatment |
Storyboard | The blueprint: narrative + StoryboardScene[] + audio direction. project.storyboard |
StoryboardScene | One scene's blueprint: purpose, visual_notes, voiceover_text, components, broll_query, hero_image, duration, template, assets. |
Scene | The realized scene: the scene_scene_NNN codegen component(s) that actually play. project.scenes[] |
AssetRequirement | An asset a scene needs (status: needed / provided / generated / fallback). |
BrandKit | Colors, fonts, logos, assets, motion style — injected into every creative step. |
Naming note: the project's
storyboard field (the data) is separate
from the storyboard status value (the review-gate state).MCP tools — src/server.ts
| Tool | Purpose |
|---|---|
create | Create a project. |
get / list | Read a project / brand kit / scene / job; list projects or jobs. |
generate | Make the video. mode: "storyboard" | "full". |
update | Edit a project, scene, component, or the storyboard directly. |
revise | Surgical natural-language edit of a scene (Studio's revise primitive). |
render | Render a project to MP4, or a single scene to a preview image. |
add / reorder | Add or reorder scenes. |
brand / upload | Manage the brand kit; upload assets/logos. |
audio | Voiceover / music configuration. |
capture | Capture a frame/preview. |
website_to_video | One-shot: a URL → brand-extracted, rendered launch video. |
regenerate_asset | Re-run a single generated asset in place. |
job | Poll / wait / list async jobs (generate, render). |
Vocabulary
One consistent language end to end (the same words in code, data, prompts, and the tools).
- Prompt — the ask: what you want. The Creative Director turns it into a vision.
- Creative Director — the step/role that turns a prompt into a vision.
- Treatment — the Creative Director's output: the one unifying concept + look/feel.
- Storyboard / Storyboard Builder — the scene-by-scene blueprint, and the role that writes it.
- Codegen — turning a storyboard scene into real HTML/CSS/GSAP.
- Critique — the vision-in-the-loop review that re-shoots scenes that miss.
- Studio — the editor: point at an element to revise it, or regenerate a scene from its storyboard.