ADu2021/dcm-dual-expert-consistency

DCM: Dual-Expert Consistency Model for Efficient and High-Quality Video Generation

Core Concept

Consistency distillation for video synthesis faces a fundamental optimization conflict: early diffusion steps require rapid semantic changes (layout, motion) while later steps need gradual detail refinement. A single student model cannot simultaneously optimize for both dynamics, creating conflicting gradients. DCM solves this by employing two specialized expert denoisers—one for semantic/motion reasoning, one for detail synthesis—enabling efficient 4-step video generation while maintaining quality near 50-step original models.

The approach maintains parameter efficiency through LoRA adapters and frozen semantic experts, achieving VBench scores of 83.83 (vs. 80.33 baseline) with 92% latency reduction.

Architecture Overview

• Semantic Expert (SemE): Operates on high-noise samples (timesteps 1-25), captures layout and motion dynamics
• Detail Expert (DetE): Operates on low-noise samples (timesteps 26-50), refines fine details through adversarial training
• Parameter Efficiency: Freezes semantic expert, adds LoRA + timestep embeddings to detail expert only
• Specialized Loss Functions: Temporal coherence for motion consistency, GAN + feature matching for detail quality
• Inference Pipeline: Routes timesteps to appropriate expert during 4-step generation

Implementation

1. Expert Specialization: Train denoisers on distinct noise ranges

# Expert training specification
def create_dual_expert_curriculum(diffusion_model, train_data):
    """
    Partition training into high-noise (semantic) and low-noise (detail) phases.
    Each expert specializes in different denoising dynamics.
    """
    semantic_expert = copy_model(diffusion_model)  # Will be frozen
    detail_expert = copy_model(diffusion_model)   # Will be adapted with LoRA

    # High-noise timesteps for semantic learning (t: 1-25, high sigma)
    semantic_batch = filter_timesteps(train_data, timestep_range=(1, 25))

    # Low-noise timesteps for detail learning (t: 26-50, low sigma)
    detail_batch = filter_timesteps(train_data, timestep_range=(26, 50))

    return {
        'semantic_expert': semantic_expert,
        'detail_expert': detail_expert,
        'semantic_data': semantic_batch,
        'detail_data': detail_batch
    }

2. Temporal Coherence Loss: Preserve motion patterns across frames in semantic expert

def temporal_coherence_loss(semantic_expert, batch_frames, target_frames):
    """
    Loss for semantic expert to maintain consistent motion across frames.
    Ensures optical flow consistency and frame alignment.
    """
    # Predict denoised output
    denoised = semantic_expert(batch_frames, timestep=high_noise)

    # Compute optical flow between consecutive frames
    flow_target = compute_optical_flow(target_frames)
    flow_denoised = compute_optical_flow(denoised)

    # L2 loss on flow consistency
    flow_loss = torch.nn.functional.mse_loss(flow_denoised, flow_target)

    # Frame alignment loss for temporal coherence
    alignment_loss = 0.0
    for i in range(len(denoised) - 1):
        # Warp frame[i+1] using flow[i] and compare to frame[i]
        warped = warp_frame(denoised[i+1], flow_denoised[i])
        alignment_loss += torch.nn.functional.mse_loss(warped, denoised[i])

    total_coherence_loss = 0.7 * flow_loss + 0.3 * alignment_loss
    return total_coherence_loss

3. Adversarial Detail Refinement: GAN-based training for detail expert

def detail_expert_loss(detail_expert, detail_discriminator, batch_frames, target_frames):
    """
    Adversarial loss for detail expert to synthesize high-frequency details.
    Combines reconstruction fidelity with feature matching.
    """
    # Forward through detail expert
    denoised = detail_expert(batch_frames, timestep=low_noise)

    # Reconstruction loss: match target quality
    recon_loss = torch.nn.functional.mse_loss(denoised, target_frames)

    # Adversarial loss: fool discriminator
    fake_logits = detail_discriminator(denoised)
    adversarial_loss = -torch.log(fake_logits + 1e-8).mean()

    # Feature matching: align intermediate representations with target
    # Extract features from intermediate layers of discriminator
    fake_features = extract_features(detail_discriminator, denoised)
    real_features = extract_features(detail_discriminator, target_frames)

    feature_loss = 0.0
    for fake_feat, real_feat in zip(fake_features, real_features):
        feature_loss += torch.nn.functional.l1_loss(fake_feat, real_feat)

    # Combined objective: reconstruction + adversarial + features
    total_loss = 0.5 * recon_loss + 0.3 * adversarial_loss + 0.2 * feature_loss
    return total_loss

4. LoRA-based Parameter Efficiency: Add lightweight adapters to detail expert only

def apply_lora_to_detail_expert(detail_expert, lora_rank=8):
    """
    Add LoRA adapters to detail expert while keeping semantic expert frozen.
    Maintains parameter efficiency—only ~1-5% additional parameters.
    """
    for name, module in detail_expert.named_modules():
        if isinstance(module, torch.nn.Linear) and 'attn' in name:
            # Add LoRA matrices to attention layers
            module.lora_A = torch.nn.Linear(module.in_features, lora_rank, bias=False)
            module.lora_B = torch.nn.Linear(lora_rank, module.out_features, bias=False)

            # Original forward: out = W @ x
            # LoRA forward: out = W @ x + (B @ A @ x) * scale
            original_forward = module.forward

            def forward_with_lora(x):
                out = original_forward(x)
                lora_out = module.lora_B(module.lora_A(x))
                return out + lora_out * (2.0 / lora_rank)  # Scaling factor

            module.forward = forward_with_lora

    # Freeze semantic expert parameters
    for param in detail_expert.parameters():
        if 'lora' not in param.__dict__:
            param.requires_grad = False

5. Inference Routing: Direct timesteps to appropriate experts

def dual_expert_denoising_step(semantic_expert, detail_expert,
                               noisy_frames, timestep, context):
    """
    Route denoising to appropriate expert based on noise level.
    Semantic expert: high noise (early steps)
    Detail expert: low noise (late steps)
    """
    sigma = get_sigma_from_timestep(timestep)

    if sigma > threshold_sigma:  # High noise → semantic expert
        denoised = semantic_expert(noisy_frames, timestep, context)
    else:  # Low noise → detail expert
        denoised = detail_expert(noisy_frames, timestep, context)

    return denoised

Practical Guidance

When to Apply:

• Video diffusion model distillation from 50+ steps to 4-8 steps
• Quality maintained while reducing latency critical
• Computational budget for dual expert training available

Setup Requirements:

• Pre-trained video diffusion model (HunyuanVideo, CogVideoX, Wan)
• High-quality video dataset with diverse motion types
• Discriminator network for adversarial detail training
• Multiple GPUs (recommend 8× A100 for 4-step distillation)

Performance Expectations:

• 4-step inference: 92% latency reduction vs. 50-step original
• VBench scores: 83.83 (dual-expert) vs. 80.33 (single LCM baseline)
• Human preference: 82.67% over competing distillation methods
• Quality nearly matches original 50-step model

Key Tuning Parameters:

• Semantic/detail split point: Typically σ = 0.5 works well; adjust based on dataset
• LoRA rank: 8 typically sufficient; increase to 16 for more expressivity
• Temporal coherence weight: 0.7 for 24fps, 0.5 for 30fps content
• Adversarial loss weight: Start at 0.3, decrease if training becomes unstable

Training Strategy:

1. First train semantic expert with temporal coherence loss (2-3 epochs)
2. Freeze semantic expert parameters
3. Train detail expert with GAN + feature matching (3-5 epochs)
4. Fine-tune jointly with reduced learning rate for 1-2 epochs

Common Pitfalls:

• Insufficient temporal coherence training causes flickering artifacts
• Adversarial training instability—use spectral normalization in discriminator
• Detail expert overfitting to training distribution—validate on diverse videos
• LoRA scaling factor impact—test 1.0, 2.0, 4.0 values

Reference

Implemented on HunyuanVideo base model. Achieves VBench 83.83 with 4-step inference. Training uses 8 A100 GPUs with mixed precision. Validated through human evaluation (82.67% preference) and automated metrics (PSNR, temporal consistency scores).