ramonclaudio/frames

Video to Frames

ultrathink

Claude cannot view video files directly but can view images. Extract frames using ffmpeg, then read a sample. Claude supports up to 600 images per request with the 1M context window, so sampling can be generous.

Environment

• ffmpeg: !which ffmpeg 2>/dev/null && ffmpeg -version 2>&1 | head -1 || echo "NOT INSTALLED - run: brew install ffmpeg"

If ffmpeg is NOT INSTALLED, stop immediately and tell the user to install it. Do not attempt any extraction.

Workflow

Parse the video path from $ARGUMENTS.

Step 1: Copy + Probe

File paths with spaces and special characters break quoting. Copy to a clean /tmp path using glob on a unique substring (timestamp, keyword).

/bin/cp -f /path/to/dir/*UNIQUE_PART* /tmp/video.mov && \
ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,duration -of csv=p=0 /tmp/video.mov

Example for Screen Recording 2026-01-10 at 11.33.27 AM.mov:

/bin/cp -f ~/Desktop/Screen*11.33.27* /tmp/video.mov && \
ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,duration -of csv=p=0 /tmp/video.mov

Step 2: Extract frames

Select FPS based on duration from the probe output. Scale to 640px width to save context tokens.

| Duration | FPS | Rationale | |----------|-----|-----------| | <5s | native | Short clip, need full detail | | 5-30s | 2 | 10-60 frames | | 30s-2min | 1 | 30-120 frames | | >2min | 0.5 | Keep frame count manageable |

trash /tmp/video-frames 2>/dev/null; mkdir -p /tmp/video-frames && \
ffmpeg -y -v warning -i /tmp/video.mov -vf "fps=FPS,scale=640:-1" /tmp/video-frames/frame_%04d.png && \
/bin/ls /tmp/video-frames/ | wc -l

For short videos (<5s), omit the fps= filter but keep the scale:

ffmpeg -y -v warning -i /tmp/video.mov -vf "scale=640:-1" /tmp/video-frames/frame_%04d.png

To extract a specific time range, add -ss START -t DURATION before -i:

ffmpeg -y -v warning -ss 00:01:00 -t 10 -i /tmp/video.mov -vf "fps=2,scale=640:-1" /tmp/video-frames/frame_%04d.png

Step 3: Read frames

List frames, then read a sample evenly distributed across the video. Use parallel Read calls for speed.

/bin/ls /tmp/video-frames/

| Frame count | Frames to read | |-------------|----------------| | <30 | All | | 30-100 | 15-20 evenly spaced | | 100-300 | 30-40 evenly spaced | | >300 | 50-60 evenly distributed |

Pick frame numbers evenly spaced across the total. Read them with the Read tool, multiple per response.

Why the Copy Pattern?

| Problem | Solution | |---------|----------| | Spaces in filenames | /bin/cp -f with glob handles any filename | | Quoted paths fail | Glob avoids quoting issues | | cd fails (zoxide) | Never use cd, use absolute paths | | Shell aliases interfere | Use /bin/cp, /bin/ls for reliability |

Gotchas

• scale=640:-1 fails on videos with odd-height dimensions. Use scale=640:-2 if ffmpeg errors with "height not divisible by 2".
• HDR screen recordings (common on Apple XDR displays) produce washed-out frames. Check color_transfer=smpte2084 in probe output and tone-map to SDR first with avconvert.
• Long videos at native FPS can dump thousands of PNGs and eat gigabytes of /tmp disk. Always check duration before extracting and pick a conservative FPS.
• If ffprobe returns no duration (e.g., some .webm or piped streams), the FPS table breaks. Fall back to 1 FPS and count frames after extraction.
• Glob-based copy (/bin/cp -f *UNIQUE*) matches multiple files if the substring isn't unique. Verify only one file matches before proceeding.