flejz/gst-caps-negotiation

GStreamer Caps Negotiation

Resolve caps (capabilities) negotiation failures and control media format flow between GStreamer elements.

What Are Caps?

Caps describe the media format flowing between elements. Each pad has a set of caps it supports. Two linked pads must agree on a compatible format during negotiation.

# Caps structure
media-type, field1=value1, field2=value2, ...

# Examples
video/x-raw, format=I420, width=1920, height=1080, framerate=30/1
audio/x-raw, format=S16LE, rate=44100, channels=2, layout=interleaved
video/x-h264, stream-format=avc, alignment=au, profile=high, level=(string)4.1
image/jpeg, width=1280, height=720

Common Video Formats

| Format | Description | Typical Use | |--------|-------------|-------------| | video/x-raw | Uncompressed video | Between processing elements | | video/x-h264 | H.264 encoded | Streaming, recording | | video/x-h265 | H.265/HEVC encoded | High-efficiency recording | | video/x-vp8 | VP8 encoded | WebM, WebRTC | | video/x-vp9 | VP9 encoded | WebM, YouTube | | video/x-av1 | AV1 encoded | Next-gen streaming | | image/jpeg | JPEG frames | MJPEG cameras |

Common Audio Formats

| Format | Description | Typical Use | |--------|-------------|-------------| | audio/x-raw | Uncompressed audio | Between processing elements | | audio/mpeg | AAC/MP3 | Streaming, recording | | audio/x-opus | Opus encoded | WebRTC, VoIP | | audio/x-vorbis | Vorbis encoded | Ogg containers | | audio/x-flac | FLAC lossless | Archival |

Raw Video Format Field Values

format:    I420, NV12, YUY2, UYVY, BGRA, RGBA, BGRx, RGBx, BGR, RGB, GRAY8, ...
width:     integer (e.g., 1920)
height:    integer (e.g., 1080)
framerate: fraction (e.g., 30/1, 25/1, 60/1, 0/1 for variable)
interlace-mode: progressive, interleaved, mixed
pixel-aspect-ratio: fraction (e.g., 1/1)
colorimetry: bt709, bt601, smpte240m
chroma-site: mpeg2, jpeg, none

Negotiation Failure Patterns and Fixes

Pattern 1: Encoder expects specific raw format

# FAILS: x264enc only accepts I420, YV12, NV12
videotestsrc ! x264enc ! ...

# FIX: Insert videoconvert
videotestsrc ! videoconvert ! x264enc ! ...

# BEST: Explicit caps for predictability
videotestsrc ! videoconvert ! video/x-raw,format=I420 ! x264enc ! ...

Pattern 2: Resolution mismatch

# FAILS: Sink expects different size than source
src ! sink expecting 1280x720

# FIX: Insert videoscale + caps filter
src ! videoscale ! video/x-raw,width=1280,height=720 ! sink

Pattern 3: Framerate mismatch

# FAILS: Muxer expects constant framerate, source is variable
camera ! muxer

# FIX: Insert videorate
camera ! videorate ! video/x-raw,framerate=30/1 ! muxer

Pattern 4: Audio sample rate mismatch

# FAILS: Encoder expects 44100 Hz, source produces 48000 Hz
src ! encoder

# FIX: Insert audioresample
src ! audioconvert ! audioresample ! audio/x-raw,rate=44100 ! encoder

Pattern 5: Channel count mismatch

# FAILS: Mono source, stereo sink
src ! audio/x-raw,channels=1 ! stereo_sink

# FIX: Insert audioconvert
src ! audioconvert ! audio/x-raw,channels=2 ! stereo_sink

Conversion Element Reference

| Problem | Solution Element | Purpose | |---------|-----------------|---------| | Wrong pixel format | videoconvert | Converts between video color formats | | Wrong resolution | videoscale | Scales video to target resolution | | Wrong framerate | videorate | Adjusts framerate by duplicating/dropping | | Wrong audio format | audioconvert | Converts between audio sample formats | | Wrong sample rate | audioresample | Resamples audio to target rate | | Need parsed stream | h264parse, mpegaudioparse | Parses encoded bitstream |

Inspecting Caps

# View element pad templates (what formats it accepts/produces)
gst-inspect-1.0 x264enc | grep -A 20 "Pad Templates"

# See negotiated caps at runtime
GST_DEBUG=GST_CAPS:4 gst-launch-1.0 ...

# Use identity to print caps flowing through
... ! identity silent=false ! ...

# Use dot graph to see caps on every link
GST_DEBUG_DUMP_DOT_DIR=/tmp gst-launch-1.0 ...

Caps Filter Syntax in gst-launch

# Inline caps (shorthand)
... ! video/x-raw,width=1280,height=720 ! ...

# capsfilter element (explicit)
... ! capsfilter caps="video/x-raw,width=1280,height=720" ! ...

# Multiple caps alternatives (element negotiates best match)
... ! "video/x-raw,format=I420; video/x-raw,format=NV12" ! ...

# Range values
... ! video/x-raw,width=[640,1920],height=[480,1080] ! ...

# List values
... ! video/x-raw,format={I420,NV12,YV12} ! ...

The Safe Conversion Chain

When unsure about format compatibility, this chain handles most conversions:

# Video: handles format, size, and rate
... ! videoconvert ! videoscale ! videorate ! capsfilter caps="TARGET_CAPS" ! ...

# Audio: handles format, rate, and channels
... ! audioconvert ! audioresample ! capsfilter caps="TARGET_CAPS" ! ...

Guidelines

• videoconvert is nearly zero-cost when input and output formats already match - it is safe to insert liberally
• Always check pad templates with gst-inspect-1.0 to understand what an element can accept
• Prefer explicit caps filters over relying on auto-negotiation in production pipelines
• When debugging caps, GST_DEBUG=GST_CAPS:4 shows the negotiation process step by step
• Parsers (h264parse, aacparse, etc.) are often needed between decoders and muxers to ensure proper stream framing
• decodebin3 auto-inserts decoders and converters, but for custom pipelines you often need manual conversion elements