hsliuustc0106/vllm-omni-multimodal

vLLM-Omni Multimodal (Omni-Modality Models)

Overview

Omni-modality models accept multiple input types (text, image, audio, video) and produce multiple output types (text, audio) in a single model. vLLM-Omni currently supports the Qwen-Omni family for this capability.

Supported Omni Models

| Model | HF ID | Inputs | Outputs | Min VRAM | |-------|-------|--------|---------|----------| | Qwen2.5-Omni-7B | Qwen/Qwen2.5-Omni-7B | Text, image, audio, video | Text, audio | 24 GB | | Qwen2.5-Omni-3B | Qwen/Qwen2.5-Omni-3B | Text, image, audio, video | Text, audio | 12 GB | | Qwen3-Omni-30B-A3B | Qwen/Qwen3-Omni-30B-A3B-Instruct | Text, image, audio, video | Text, audio | 48 GB |

Quick Start

Offline: Text Conversation

from vllm_omni.entrypoints.omni import Omni

omni = Omni(model="Qwen/Qwen2.5-Omni-7B")
outputs = omni.generate("What is the capital of France?")
print(outputs[0].request_output[0].text)

Online: Start Server

vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091

Input Validation Workflow

Validate media inputs before sending to avoid OOM errors and processing failures:

import os
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8091/v1", api_key="unused")

MAX_IMAGE_MB = 20
MAX_AUDIO_MB = 50
MAX_VIDEO_MB = 100
SUPPORTED_IMAGE = {".jpg", ".jpeg", ".png", ".webp"}
SUPPORTED_AUDIO = {".wav", ".mp3", ".flac"}
SUPPORTED_VIDEO = {".mp4", ".webm"}

def validate_and_encode(path: str, max_mb: float, supported_exts: set) -> str:
    ext = os.path.splitext(path)[1].lower()
    assert ext in supported_exts, f"Unsupported format: {ext}"
    size_mb = os.path.getsize(path) / (1024 * 1024)
    assert size_mb <= max_mb, f"File too large: {size_mb:.1f}MB > {max_mb}MB"
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode()

Multi-Modal Input Patterns

Image Understanding

img_b64 = validate_and_encode("photo.jpg", MAX_IMAGE_MB, SUPPORTED_IMAGE)
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
            {"type": "text", "text": "What do you see in this image?"},
        ],
    }],
)
print(response.choices[0].message.content)

Audio Understanding (Speech-to-Text)

audio_b64 = validate_and_encode("recording.wav", MAX_AUDIO_MB, SUPPORTED_AUDIO)
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
            {"type": "text", "text": "Transcribe this audio."},
        ],
    }],
)

Video Understanding

video_b64 = validate_and_encode("clip.mp4", MAX_VIDEO_MB, SUPPORTED_VIDEO)
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_b64}"}},
            {"type": "text", "text": "Describe what happens in this video."},
        ],
    }],
)

Combined Inputs

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
            {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
            {"type": "text", "text": "Does the audio describe what's in the image?"},
        ],
    }],
)

Audio Output (Voice Synthesis)

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B",
    messages=[{"role": "user", "content": "Say hello in English and Chinese."}],
    extra_body={"output_modalities": ["text", "audio"]},
)

Multi-Turn Conversations

messages = [
    {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
        {"type": "text", "text": "What's in this image?"},
    ]},
]
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B", messages=messages
)
messages.append({"role": "assistant", "content": response.choices[0].message.content})
messages.append({"role": "user", "content": "What colors are dominant?"})
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Omni-7B", messages=messages
)

Qwen3-Omni (MoE)

Qwen3-Omni uses a Mixture-of-Experts architecture (30B total, 3B active). Requires multi-GPU:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
  --tensor-parallel-size 2 --port 8091

Qwen3-Omni is compatible with the v2 model runner (vllm 0.19). Uses native launch_core_engines instead of custom spawning. add_streaming_update API removed; audio output tensors are explicitly converted to float. CUDAGraph supports tuple-returning thinker model. Fixed in #2522.

Troubleshooting

Slow with video input: Video processing requires extracting and encoding frames. Shorter clips process faster.

Audio output garbled: Ensure the client correctly handles the audio response format (base64 encoded WAV).

Out of memory with multi-modal input: Large images/videos consume significant memory. Use the validation workflow above to check file sizes before sending.

Qwen3-Omni performance: The multi-stage pipeline optimizes CPU hidden-state copying — only copies to CPU when downstream stages need payloads. Text-only inference (without --omni) is supported for benchmarking via use_omni: false. Fixed in #3203.

References

• For Qwen-Omni architecture and advanced config, see references/qwen-omni.md