hsliuustc0106/vllm-omni-hardware

vLLM-Omni Hardware Configuration

Overview

vLLM-Omni supports four hardware backends: NVIDIA CUDA (default), AMD ROCm, Huawei NPU (Ascend), and Intel XPU. Each backend has specific installation steps and configuration options.

Supported Backends

| Backend | Accelerators | Install Method | Maturity | |---------|-------------|----------------|----------| | CUDA | NVIDIA A100/H100/L40/RTX | uv pip install vllm==$VLLM_VERSION | Production | | ROCm | AMD MI300X/MI250X | uv pip install vllm==$VLLM_VERSION --extra-index-url ... | Production | | NPU | Huawei Ascend 910B | Source build with CANN | Supported | | XPU | Intel Data Center GPU Max | Source build with oneAPI | Experimental |

Backend Selection Workflow

Step 1: Identify Hardware

# NVIDIA GPU
nvidia-smi

# AMD GPU
rocm-smi

# Huawei NPU
npu-smi info

# Intel XPU
xpu-smi discovery

Step 2: Install for Backend

CUDA (NVIDIA):

uv pip install vllm==$VLLM_VERSION --torch-backend=auto

ROCm (AMD):

uv pip install vllm==$VLLM_VERSION --extra-index-url https://wheels.vllm.ai/rocm/$VLLM_VERSION/rocm722

NPU (Huawei):

# Requires CANN toolkit pre-installed
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
pip install -e ".[npu]"

XPU (Intel):

# Requires oneAPI toolkit pre-installed
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
pip install -e ".[xpu]"

Step 3: Verify Backend

import torch

# CUDA
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU count: {torch.cuda.device_count()}")

# ROCm (same API as CUDA via HIP)
print(f"ROCm/HIP available: {torch.cuda.is_available()}")

# XPU
print(f"XPU available: {torch.xpu.is_available()}")

Device Selection

Control which devices vLLM-Omni uses:

# CUDA: Select specific GPUs
CUDA_VISIBLE_DEVICES=0,1 vllm serve <model> --omni

# ROCm: Select specific GPUs
HIP_VISIBLE_DEVICES=0,1 vllm serve <model> --omni

# NPU: Select specific devices
ASCEND_RT_VISIBLE_DEVICES=0,1 vllm serve <model> --omni

Model Support by Backend

Not all models are supported on every backend. Check the support matrix:

| Model | CUDA | ROCm | NPU | XPU | |-------|------|------|-----|-----| | Qwen3-Omni | Yes | Yes | Yes | No | | Qwen2.5-Omni | Yes | Yes | Yes | No | | Qwen-Image | Yes | Yes | Yes | No | | Z-Image | Yes | Yes | Yes | No | | BAGEL | Yes | Yes | No | No | | Wan2.2 | Yes | Yes | Yes | No | | FLUX | Yes | Yes | Yes | No | | Qwen3-TTS | Yes | Yes | Yes | No | | Stable-Diffusion-3 | Yes | Yes | No | No | | Stable-Audio | Yes | No | No | No |

Troubleshooting

CUDA out of memory: Reduce --gpu-memory-utilization or use tensor parallelism across multiple GPUs.

ROCm kernel compilation slow: First launch compiles kernels for your GPU. Subsequent launches reuse cached kernels. Set MIOPEN_USER_DB_PATH for persistent kernel cache.

ROCm Wan2.2 RoPE error (too many values to unpack): Fixed in #3463. Added dedicated forward_hip() RoPE path for ROCm using flash attention rotary embedding.

NPU operator not supported: Some operations fall back to CPU on NPU. Check logs for fallback warnings and update CANN to the latest version.

Out-of-Tree Hardware Backends

Out-of-tree hardware backends can customize the diffusion engine without forking core code by subclassing OmniPlatform and overriding extension points:

| Method | Returns | Purpose | |--------|---------|---------| | get_diffusion_worker_cls() | Fully qualified class path | Custom diffusion worker | | get_diffusion_model_runner_cls() | Fully qualified class path | Custom model runner |

Use OmniPlatformEnum.OOT with is_out_of_tree() for platform detection. Register custom diffusion pipelines via register_diffusion_model(model_arch, module_name, class_name) in vllm_omni.diffusion.registry. Defaults preserve existing behavior for CUDA, ROCm, NPU, XPU, and MUSA. NPU LaserAttention unsupported error: On Ascend NPU with mindiesd, selecting FLASH_ATTN as the diffusion attention backend (--diffusion-attn-backend FLASH_ATTN) auto-imports mindiesd to configure ASCEND_CUSTOM_OPP_PATH. The internal environment variable MINDIE_SD_FA_TYPE is set to ascend_laser_attention automatically. Fixed in #2674.

References

• For CUDA-specific optimization, see references/cuda.md
• For ROCm setup details, see references/rocm.md
• For NPU configuration, see references/npu.md
• For XPU setup, see references/xpu.md