garrettroi/ascii-video

ASCII Video Production Pipeline

Full production pipeline for rendering any content as colored ASCII character video.

Modes

| Mode | Input | Output | Read | |------|-------|--------|------| | Video-to-ASCII | Video file | ASCII recreation of source footage | references/inputs.md § Video Sampling | | Audio-reactive | Audio file | Generative visuals driven by audio features | references/inputs.md § Audio Analysis | | Generative | None (or seed params) | Procedural ASCII animation | references/effects.md | | Hybrid | Video + audio | ASCII video with audio-reactive overlays | Both input refs | | Lyrics/text | Audio + text/SRT | Timed text with visual effects | references/inputs.md § Text/Lyrics | | TTS narration | Text quotes + TTS API | Narrated testimonial/quote video with typed text | references/inputs.md § TTS Integration |

Stack

Single self-contained Python script per project. No GPU.

| Layer | Tool | Purpose | |-------|------|---------| | Core | Python 3.10+, NumPy | Math, array ops, vectorized effects | | Signal | SciPy | FFT, peak detection (audio modes only) | | Imaging | Pillow (PIL) | Font rasterization, video frame decoding, image I/O | | Video I/O | ffmpeg (CLI) | Decode input, encode output segments, mux audio, mix tracks | | Parallel | concurrent.futures / multiprocessing | N workers for batch/clip rendering | | TTS | ElevenLabs API (or similar) | Generate narration clips for quote/testimonial videos | | Optional | OpenCV | Video frame sampling, edge detection, optical flow |

Pipeline Architecture (v2)

Every mode follows the same 6-stage pipeline. See references/architecture.md for implementation details, references/scenes.md for scene protocol, and references/composition.md for multi-grid composition and tonemap.

┌─────────┐   ┌──────────┐   ┌───────────┐   ┌──────────┐   ┌─────────┐   ┌────────┐
│ 1.INPUT  │→│ 2.ANALYZE │→│ 3.SCENE_FN │→│ 4.TONEMAP │→│ 5.SHADE  │→│ 6.ENCODE│
│ load src │  │ features  │  │ → canvas   │  │ normalize │  │ post-fx  │  │ → video │
└─────────┘   └──────────┘   └───────────┘   └──────────┘   └─────────┘   └────────┘

1. INPUT — Load/decode source material (video frames, audio samples, images, or nothing)
2. ANALYZE — Extract per-frame features (audio bands, video luminance/edges, motion vectors)
3. SCENE_FN — Scene function renders directly to pixel canvas (uint8 H,W,3). May internally compose multiple character grids via _render_vf() + pixel blend modes. See references/composition.md
4. TONEMAP — Percentile-based adaptive brightness normalization with per-scene gamma. Replaces linear brightness multipliers. See references/composition.md § Adaptive Tonemap
5. SHADE — Apply post-processing ShaderChain + FeedbackBuffer. See references/shaders.md
6. ENCODE — Pipe raw RGB frames to ffmpeg for H.264/GIF encoding

Creative Direction

Every project should look and feel different. The references provide a vocabulary of building blocks — don't copy them verbatim. Combine, modify, and invent.

Aesthetic Dimensions to Vary

| Dimension | Options | Reference | |-----------|---------|-----------| | Character palette | Density ramps, block elements, symbols, scripts (katakana, Greek, runes, braille), dots, project-specific | architecture.md § Character Palettes | | Color strategy | HSV (angle/distance/time/value mapped), discrete RGB palettes, monochrome, complementary, triadic, temperature | architecture.md § Color System | | Color tint | Warm, cool, amber, matrix green, neon pink, sepia, ice, blood, void, sunset | shaders.md § Color Grade | | Background texture | Sine fields, noise, smooth noise, cellular/voronoi, video source | effects.md § Background Fills | | Primary effects | Rings, spirals, tunnel, vortex, waves, interference, aurora, ripple, fire | effects.md § Radial / Wave / Fire | | Particles | Energy sparks, snow, rain, bubbles, runes, binary data, orbits, gravity wells | effects.md § Particle Systems | | Shader mood | Retro CRT, clean modern, glitch art, cinematic, dreamy, harsh industrial, psychedelic | shaders.md § Design Philosophy | | Grid density | xs(8px) through xxl(40px), mixed per layer | architecture.md § Grid System | | Font | Menlo, Monaco, Courier, SF Mono, JetBrains Mono, Fira Code, IBM Plex | architecture.md § Font Selection | | Mirror mode | None, horizontal, vertical, quad, diagonal, kaleidoscope | shaders.md § Mirror Effects | | Transition style | Crossfade, wipe (directional/radial), dissolve, glitch cut | shaders.md § Transitions |

Per-Section Variation

Never use the same config for the entire video. For each section/scene/quote:

• Choose a different background effect (or compose 2-3)
• Choose a different character palette (match the mood)
• Choose a different color strategy (or at minimum a different hue)
• Vary shader intensity (more bloom during peaks, more grain during quiet)
• Use different particle types if particles are active

Project-Specific Invention

For every project, invent at least one of:

• A custom character palette matching the theme
• A custom background effect (combine/modify existing ones)
• A custom color palette (discrete RGB set matching the brand/mood)
• A custom particle character set

Workflow

Step 1: Determine Mode and Gather Requirements

Establish with user:

• Input source — file path, format, duration
• Mode — which of the 6 modes above
• Sections — time-mapped style changes (timestamps → effect names)
• Resolution — default 1920x1080 @ 24fps; GIFs typically 640x360 @ 15fps
• Style direction — dense/sparse, bright/dark, chaotic/minimal, color palette
• Text/branding — easter eggs, overlays, credits, themed character sets
• Output format — MP4 (default), GIF, PNG sequence

Step 2: Detect Hardware and Set Quality

Before building the script, detect the user's hardware and set appropriate defaults. See references/optimization.md § Hardware Detection.

hw = detect_hardware()
profile = quality_profile(hw, target_duration, user_quality_pref)
log(f"Hardware: {hw['cpu_count']} cores, {hw['mem_gb']:.1f}GB RAM")
log(f"Render: {profile['vw']}x{profile['vh']} @{profile['fps']}fps, {profile['workers']} workers")

Never hardcode worker counts, resolution, or CRF. Always detect and adapt.

Step 3: Build the Script

Write as a single Python file. Major components:

1. Hardware detection + quality profile — see references/optimization.md
2. Input loader — mode-dependent; see references/inputs.md
3. Feature analyzer — audio FFT, video luminance, or pass-through
4. Grid + renderer — multi-density character grids with bitmap cache; _render_vf() helper for value/hue field → canvas
5. Character palettes — multiple palettes chosen per project theme; see references/architecture.md
6. Color system — HSV + discrete RGB palettes as needed; see references/architecture.md
7. Scene functions — each returns canvas (uint8 H,W,3) directly. May compose multiple grids internally via pixel blend modes. See references/scenes.md + references/composition.md
8. Tonemap — adaptive brightness normalization with per-scene gamma; see references/composition.md
9. Shader pipeline — ShaderChain + FeedbackBuffer per-section config; see references/shaders.md
10. Scene table + dispatcher — maps time ranges to scene functions + shader/feedback configs; see references/scenes.md
11. Parallel encoder — N-worker batch clip rendering with ffmpeg pipes
12. Main — orchestrate full pipeline

Step 4: Handle Critical Bugs

Font Cell Height (macOS Pillow)

textbbox() returns wrong height. Use font.getmetrics():

ascent, descent = font.getmetrics()
cell_height = ascent + descent  # correct

ffmpeg Pipe Deadlock

Never use stderr=subprocess.PIPE with long-running ffmpeg. Redirect to file:

stderr_fh = open(err_path, "w")
pipe = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.DEVNULL, stderr=stderr_fh)

Brightness — Use tonemap(), Not Linear Multipliers

ASCII on black is inherently dark. This is the #1 visual issue. Do NOT use linear * N brightness multipliers — they clip highlights and wash out the image. Instead, use the adaptive tonemap function from references/composition.md:

def tonemap(canvas, gamma=0.75):
    """Percentile-based adaptive normalization + gamma. Replaces all brightness multipliers."""
    f = canvas.astype(np.float32)
    lo = np.percentile(f, 1)          # black point (1st percentile)
    hi = np.percentile(f, 99.5)       # white point (99.5th percentile)
    if hi - lo < 1: hi = lo + 1
    f = (f - lo) / (hi - lo)
    f = np.clip(f, 0, 1) ** gamma     # gamma < 1 = brighter mids
    return (f * 255).astype(np.uint8)

Pipeline ordering: scene_fn() → tonemap() → FeedbackBuffer → ShaderChain → ffmpeg

Per-scene gamma overrides for destructive effects:

• Default: gamma=0.75
• Solarize scenes: gamma=0.55 (solarize darkens above-threshold pixels)
• Posterize scenes: gamma=0.50 (quantization loses brightness range)
• Already-bright scenes: gamma=0.85

Additional brightness best practices:

• Dense animated backgrounds — never flat black, always fill the grid
• Vignette minimum clamped to 0.15 (not 0.12)
• Bloom threshold lowered to 130 (not 170) so more pixels contribute to glow
• Use screen blend mode (not overlay) when compositing dark ASCII layers — overlay squares dark values: 2 * 0.12 * 0.12 = 0.03

Font Compatibility

Not all Unicode characters render in all fonts. Validate palettes at init:

for c in palette:
    img = Image.new("L", (20, 20), 0)
    ImageDraw.Draw(img).text((0, 0), c, fill=255, font=font)
    if np.array(img).max() == 0:
        log(f"WARNING: char '{c}' (U+{ord(c):04X}) not in font, removing from palette")

Step 4b: Per-Clip Architecture (for segmented videos)

When the video has discrete segments (quotes, scenes, chapters), render each as a separate clip file. This enables:

• Re-rendering individual clips without touching the rest (--clip q05)
• Faster iteration on specific sections
• Easy reordering or trimming in post

segments = [
    {"id": "intro", "start": 0.0, "end": 5.0, "type": "intro"},
    {"id": "q00", "start": 5.0, "end": 12.0, "type": "quote", "qi": 0, ...},
    {"id": "t00", "start": 12.0, "end": 13.5, "type": "transition", ...},
    {"id": "outro", "start": 208.0, "end": 211.6, "type": "outro"},
]

from concurrent.futures import ProcessPoolExecutor, as_completed
with ProcessPoolExecutor(max_workers=hw["workers"]) as pool:
    futures = {pool.submit(render_clip, seg, features, path): seg["id"]
               for seg, path in clip_args}
    for fut in as_completed(futures):
        fut.result()

CLI: --clip q00 t00 q01 to re-render specific clips, --list to show segments, --skip-render to re-stitch only.

Step 5: Render and Iterate

Performance targets per frame:

| Component | Budget | |-----------|--------| | Feature extraction | 1-5ms | | Effect function | 2-15ms | | Character render | 80-150ms (bottleneck) | | Shader pipeline | 5-25ms | | Total | ~100-200ms/frame |

Fast iteration: render single test frames to check brightness/layout before full render:

canvas = render_single_frame(frame_index, features, renderer)
Image.fromarray(canvas).save("test.png")

Brightness verification: sample 5-10 frames across video, check mean > 8 for ASCII content.

References

| File | Contents | |------|----------| | references/architecture.md | Grid system, font selection, character palettes (library of 20+), color system (HSV + discrete RGB), _render_vf() helper, compositing, v2 effect function contract | | references/inputs.md | All input sources: audio analysis, video sampling, image conversion, text/lyrics, TTS integration (ElevenLabs, voice assignment, audio mixing) | | references/effects.md | Effect building blocks: 12 value field generators (vf_sinefield through vf_noise_static), 8 hue field generators (hf_fixed through hf_plasma), radial/wave/fire effects, particles, composing guide | | references/shaders.md | 38 shader implementations (geometry, channel, color, glow, noise, pattern, tone, glitch, mirror), ShaderChain class, full _apply_shader_step() dispatch, audio-reactive scaling, transitions, tint presets | | references/composition.md | v2 core: pixel blend modes (20 modes with implementations), multi-grid composition, _render_vf() helper, adaptive tonemap(), per-scene gamma, FeedbackBuffer with spatial transforms, PixelBlendStack | | references/scenes.md | v2 scene protocol: scene function contract, Renderer class, SCENES table structure, render_clip() loop, beat-synced cutting, parallel rendering + pickling constraints, 4 complete scene examples, scene design checklist | | references/troubleshooting.md | NumPy broadcasting traps, blend mode pitfalls, multiprocessing/pickling issues, brightness diagnostics, ffmpeg deadlocks, font issues, performance bottlenecks, common mistakes | | references/optimization.md | Hardware detection, adaptive quality profiles (draft/preview/production/max), CLI integration, vectorized effect patterns, parallel rendering, memory management |