curiositech/video-frame-extraction-analysis

Video Frame Extraction & Analysis

Extract keyframes, detect scene boundaries, and build CLIP-indexed temporal search systems for video content analysis and retrieval.

Activation Triggers

Activate on: "keyframe extraction", "scene detection", "video search", "video indexing", "find frame in video", "temporal search", "video content analysis", "scene boundary detection", "CLIP video search"

NOT for: Video editing, trimming, or rendering (video-processing-editing), video generation from text/images (ai-video-production-master), or face recognition in video (face-recognition-system-builder)

Quick Start

1. Decode video — Use ffmpeg or decord for efficient frame extraction. Never decode all frames; sample intelligently.
2. Detect scenes — PySceneDetect for cut detection, or embedding-based scene boundary detection for gradual transitions.
3. Extract keyframes — One representative frame per scene, plus additional frames at fixed intervals within long scenes.
4. Embed frames — Run CLIP/SigLIP on keyframes to generate searchable visual embeddings.
5. Index and search — Store embeddings with timestamps in a vector DB. Query with text ("a person opening a door") or image.

Core Capabilities

| Domain | Technologies | Notes | |--------|-------------|-------| | Frame Extraction | ffmpeg, decord, OpenCV, PyAV | decord is fastest for random access | | Scene Detection | PySceneDetect, TransNetV2 | Cut detection + gradual transition detection | | Visual Embedding | CLIP, SigLIP, InternVideo2 | Per-frame or pooled-scene embeddings | | Temporal Search | Vector DB + timestamp metadata | "Find the frame where X happens" | | Shot Analysis | Shot type classification, motion estimation | Wide/medium/close-up, camera movement | | OCR on Frames | PaddleOCR, EasyOCR, Tesseract | Extract text from slides, titles, signage |

Architecture Patterns

Pattern 1: Scene-Based Keyframe Pipeline

Video File ──→ [Scene Detector] ──→ [Keyframe Selector] ──→ [CLIP Embed] ──→ [Vector DB]
    │                │                      │                      │              │
  input         PySceneDetect          1 frame per scene      SigLIP-large    store with
  mp4/mkv       threshold=27           at scene midpoint      384-dim         timestamp
                detect cuts            + every 5 sec in                       + scene_id
                + transitions          long scenes (>30s)                     + metadata

# Scene detection + keyframe extraction
from scenedetect import detect, ContentDetector
import decord
import numpy as np

def extract_keyframes(video_path: str) -> list[dict]:
    """Extract one keyframe per scene with timestamps."""
    # Step 1: Detect scene boundaries
    scene_list = detect(video_path, ContentDetector(threshold=27))

    # Step 2: Extract keyframe at midpoint of each scene
    vr = decord.VideoReader(video_path)
    fps = vr.get_avg_fps()
    keyframes = []

    for i, scene in enumerate(scene_list):
        start_frame = scene[0].get_frames()
        end_frame = scene[1].get_frames()
        mid_frame = (start_frame + end_frame) // 2

        frame = vr[mid_frame].asnumpy()  # RGB numpy array
        timestamp = mid_frame / fps

        keyframes.append({
            "frame": frame,
            "frame_index": mid_frame,
            "timestamp_sec": timestamp,
            "scene_index": i,
            "scene_duration": (end_frame - start_frame) / fps,
        })

        # For long scenes (>30s), add extra keyframes every 5 sec
        scene_dur = (end_frame - start_frame) / fps
        if scene_dur > 30:
            for t in np.arange(start_frame + int(5*fps), end_frame, int(5*fps)):
                extra = vr[int(t)].asnumpy()
                keyframes.append({
                    "frame": extra,
                    "frame_index": int(t),
                    "timestamp_sec": int(t) / fps,
                    "scene_index": i,
                    "scene_duration": scene_dur,
                })

    return keyframes

Pattern 2: CLIP-Indexed Video Search

Indexing (offline):
  Video ──→ [Extract Keyframes] ──→ [SigLIP Embed] ──→ [Vector DB]
                                                            │
                                                     metadata per frame:
                                                     video_id, timestamp,
                                                     scene_id, thumbnail_path

Querying (online):
  Text: "person walking through rain" ──→ [SigLIP Text Embed] ──→ [Vector Search]
                                                                        │
                                                                  top-k frames
                                                                  with timestamps
                                                                        │
                                                                  "video_3.mp4 @ 01:23:45"
                                                                  "video_7.mp4 @ 00:45:12"

Pattern 3: Multimodal Video Index

Video ──┬──→ [Keyframes] ──→ [CLIP Embed] ──→ [Visual Index]──┐
        │                                                       │
        ├──→ [Audio Track] ──→ [Whisper] ──→ [Text Embed] ──→ [Text Index]──┤──→ [Fusion Search]
        │                                                       │
        └──→ [Frame OCR] ──→ [Text Extract] ──→ [Text Embed]──┘

Fusion search: query hits all three indexes, reciprocal rank fusion combines results
  "Explain the sales chart" →
    Visual: frame with chart → timestamp 15:30
    Audio: "our Q3 numbers show..." → timestamp 15:28
    OCR: "Q3 Revenue: $4.2M" → timestamp 15:30
    Fused result: 15:28-15:35 segment with high confidence

Anti-Patterns

1. Extracting every frame — A 1-hour video at 30fps is 108,000 frames. Embedding all of them is wasteful and slow. Use scene detection to select keyframes.
2. Fixed-interval sampling without scene awareness — Sampling every 5 seconds misses fast cuts and over-samples static scenes. Scene-based extraction is always better.
3. Ignoring temporal context — A single frame without a timestamp is useless for video search. Always store timestamps and scene boundaries as metadata.
4. Using OpenCV VideoCapture for random access — OpenCV decodes sequentially; seeking to frame 50,000 reads all prior frames. Use decord for O(1) random access.
5. No thumbnail generation — Search results need visual previews. Generate thumbnails alongside embeddings, stored as small JPEGs referenced by path in metadata.

Quality Checklist

• [ ] Scene detection threshold tuned for content type (27-35 for cuts, lower for gradual)
• [ ] Keyframes extracted at scene midpoints (not boundaries, which may be transitional)
• [ ] Long scenes (>30s) have additional keyframes at regular intervals
• [ ] CLIP/SigLIP embeddings normalized before storage
• [ ] Timestamps stored as metadata on every vector (video_id + seconds)
• [ ] Thumbnails generated and stored alongside embeddings
• [ ] Text search via transcription aligned with visual frames
• [ ] Random access via decord or PyAV (not sequential OpenCV reads)
• [ ] Search results return video_id + timestamp + thumbnail + confidence score
• [ ] Pipeline handles multiple video formats (MP4, MKV, WebM, MOV)