oaustegard/drawing-video

drawing-video

Footage in, narrated comic strip out. Five stages: probe → ground → narrate → draw → QA/compose. The hard part is not the drawing; it is not lying about what the footage shows. One stage is a gate.

Requires: processing-video (ffmpeg, present), invoking-gemini (image + video models), proxy.env (CF gateway creds — auto-read by the gemini client).

---

Stage 0 — Probe & extract frames

ffprobe -v quiet -show_entries format=duration -of csv=p=0 in.mov
mkdir -p frames
ffmpeg -v error -i in.mov -vf "fps=1,scale=640:-1" frames/f_%03d.png   # 1/sec, downscaled

Keep full-resolution stills too — you will feed them to the image model as reference. For a subject hidden in clutter, pull a few full-res frames at specific timestamps: ffmpeg -v error -ss 11 -i in.mov -frames:v 1 hr.png.

---

Stage 1 — GROUND THE STORYLINE (the gate)

Never narrate a scene whose subjects and action you have not confirmed from the footage. Diagnosed failure 2026-07-18: built an entire "indoor cyclist in a garage" storyline off thumbnail-sized frames of a video that was actually a leashed dog watching a deer — no bicycle in any frame, the leash visible in four. Then fed the fiction to the image model, which dutifully drew it. A wrong storyline launders a hallucination into finished art. The absence of a clear read is not license to invent a vivid one.

Two ways to ground. Prefer Mode B — it is the more reliable and reads the whole clip, not sampled stills.

Mode B (recommended): native video understanding via gemini-3.5-flash

gemini-3.5-flash parses video directly. Compress first, send as inline data.

ffmpeg -v error -i in.mov -vf "scale=480:-2,fps=8" -c:v libx264 -crf 30 -preset veryfast -an small.mp4

import sys, base64; sys.path.append('/mnt/skills/user/invoking-gemini/scripts')
from gemini_client import get_cf_credentials, _cf_request
from pathlib import Path
c = get_cf_credentials()
b = base64.b64encode(Path("small.mp4").read_bytes()).decode()
prompt = ("Watch this video and describe the actual storyline in 3-4 sentences: which "
          "subjects appear, what happens, the arc. Be literal — only what is visibly present.")
contents = [{"parts": [{"inlineData": {"mimeType": "video/mp4", "data": b}}, {"text": prompt}]}]
resp = _cf_request("gemini-3.5-flash", contents,
                   {"temperature": 0.3, "thinkingConfig": {"thinkingLevel": "minimal"},
                    "maxOutputTokens": 600}, c)
print("".join(p.get("text","") for p in resp["candidates"][0]["content"]["parts"]))

Inline works for small files (verified: a 22s 4K clip compressed to 2.3 MB). For long clips, sample frames (below) or use the File API. gemini-2.5-flash-image is not a parser — this is gemini-3.5-flash (text/multimodal).

Mode A: read the frames yourself

Build a contact sheet (PIL grid of frames/*.png), then view enough full-resolution frames to name every subject and the action out loud before writing a word of narration. If you cannot tell what a shape is, extract a higher-res crop at that timestamp and look again. Do not proceed on a guess.

Either way, end Stage 1 with one plain paragraph: subjects, setting, arc. If the user has already told you the storyline, that paragraph is authority — use it verbatim and skip re-derivation.

---

Stage 2 — Beats & narration

Let the video decide the panel count. Break the clip into its natural beats — the distinct moments the story actually has — and make one panel per beat. Do NOT default to four or to a 2×2 grid. A 12-second perch-and-glide is three beats (sentinel → launch → glide); a busier clip may be five or six. Fewer, larger panels for a slow arc; more, tighter panels for rapid action. (Diagnosed 2026-07-18: a hardcoded 2×2 four-panel default is wrong — layout must fit the footage.)

Write one short caption per beat, in the requested voice (David Attenborough hushed-drama is the common ask; dial the stakes above what the mundane footage deserves). Captions must be short enough to render legibly in a box — one or two sentences.

Keep a character/setting bible in one line: the exact look of each recurring subject (breed/colour, leash colour, the woods, the light). You will paste it into every panel prompt so the panels stay consistent.

---

Stage 3 — Design the page, then draw the panels

Division of labour (the default): you are weak at drawing but decent at layout, so you design the page and outsource only the drawing. Design a bespoke layout that fits the story — panel count from Stage 2, panel sizes and arrangement chosen to serve the arc (a splash panel for the climax; a tall panel for a vertical action; a downward stack for a descent; growing panels to build drama). Assign each panel a target aspect ratio. Then generate each panel with Gemini at that aspect ratio, and composite the drawn panels into the layout you designed (PIL). This beats both the rigid uniform grid and the one-shot page.

• Request the aspect in the prompt ("Draw ONE comic panel, 16:7 wide landscape" / "4:3 large" / "3:4 tall"). Gemini 3 honours it closely (measured 2026-07-18: asked 16:7/16:6/4:3, got 2.33/2.72/1.34).
• Composite with cover-fit (resize to fill the box, centre-crop) so a slightly-off aspect still fills its box cleanly. Design boxes at each panel's target aspect to minimise cropping.

Two alternatives, by request:

• One-shot full page — Gemini 3 can render an entire multi-panel comic in ONE image call; describe the whole layout (panel count, arrangement, gutters, per-panel scene + caption, title) in the prompt. Use when the user wants Gemini's freeform layout and you don't need pixel control. Trade-off: no per-panel regeneration, less border control.
• Uniform grid — the simplest fallback; a fixed 2×N. Use only when the story genuinely has equal-weight beats.

Model for all paths: gemini-3-pro-image (GA; was gemini-3-pro-image-preview). There is no Gemini 3.5 image model — 3.5 is text-only; the current image line is gemini-3-pro-image and gemini-3.1-flash-image. Do not use gemini-2.5-flash-image. (gemini-3.5-flash is the video parser in Stage 1, not an image model.)

Four rules for per-panel drawing, each from a diagnosed defect on 2026-07-18:

1. Anchor every panel to real stills. Pass TWO images per panel: a single shared anchor still (canonical setting + subject) and the panel's own still. Instruct: "First reference = canonical setting+subject; second = this panel's framing; redraw as a comic panel." This is what keeps the drawn scene faithful instead of inventing.
2. State continuity explicitly. "All N panels are ONE comic: identical setting, identical <subject> design (<character bible>)." Without it the woods and the animal drift panel to panel.
3. Constrain creatures with negatives. The image model turned an adult doe into a spotted fawn. Say "adult doe, uniform tan, NO white spots, NOT a fawn" — spell out what it is not.
4. Lock the caption box. "Place ONE caption box at the top containing EXACTLY this text and NOTHING else — no extra words, no stray letters, no style notes: '<caption>'. Do not write any other text anywhere." Stray style words leaked into a caption box until this was added.

Run panels detached and adaptive-wait — bash_tool times out at ~50s and image-pro calls are slow. Launch all panels in parallel threads in one detached script writing a DONE sentinel; poll with timeout 45 sh -c 'while [ ! -f DONE ]; do sleep 3; done'. Build the request with _cf_request(model, contents, {"temperature":0.65,"responseModalities":["IMAGE","TEXT"]}, creds); extract the image from the part whose key is inlineData/inline_data.

---

Stage 4 — QA, then compose

QA before shipping — do not present unseen panels. If the local image viewer is available, look at every panel. If it is not (it went dark mid-session on 2026-07-18), QA through Gemini vision instead:

from gemini_client import invoke_gemini
invoke_gemini(prompt=("QA this comic panel. JSON only: setting, subject, deer, "
   "deer_spotted_fawn (true only if white baby spots), caption (verbatim), palette."),
   model="flash", image_path="panel_1.png", max_output_tokens=500, thinking_level="minimal")

Check per panel: subjects match the grounded storyline, setting/subject consistent across panels, captions verbatim with no stray text and no doubled words (Gemini repeated "and and" once — 2026-07-18), creatures correct (adult vs fawn). Regenerate only the panels that fail — single-panel regen is cheap. For a one-shot page, QA the whole page the same way. (Note: the vision QA reports your own title/subtitle as "stray text" — that is expected, not a defect.)

Compose with PIL into the layout you designed in Stage 3. Strip any baked border first — image-model panels arrive with inconsistent baked frames (one dark, others white), which reads as a mismatched-border defect: crop a uniform ~8–10px inset off every panel, then draw one identical border on all. Cover-fit each panel into its designed box, add a title band, save to /mnt/user-data/outputs/, then present_files. (The one-shot page needs none of this — it arrives pre-composed; just QA and present.)

---

Pitfall summary (all diagnosed 2026-07-18, one session)

• Confabulated the storyline from thumbnails → Stage 1 gate; prefer Mode B.
• Wrong prompt made the image model draw the fiction → anchor to real stills.
• Woods/subject drifted between panels → shared anchor + explicit continuity.
• Doe rendered as a spotted fawn → negative constraints on creatures.
• Style words leaked into a caption box → lock the caption text.
• Panel-1 border differed → strip baked borders, apply one uniform frame.
• Guessed "Gemini 3.5 image" → no such model; use gemini-3-pro-image, bin 2.5-flash-image.
• Local image viewer unavailable → QA via Gemini vision, never ship unseen.
• Hardcoded a 2×2 four-panel layout → derive panel count from the beats; design the layout to fit the story.
• Gemini doubled a word in a caption ("and and") → QA captions for doubled words, not just stray text.