wenmin-wu/cv-classifier-free-guidance-diffusion

Classifier-Free Guidance Diffusion

Overview

Classifier-free guidance (CFG) steers diffusion model generation toward a text prompt without a separate classifier. At each denoising step, the model predicts noise twice — once conditioned on the prompt, once unconditionally — and the final noise is an extrapolation: noise = uncond + scale * (cond - uncond). Higher guidance scales produce images more faithful to the prompt at the cost of diversity.

Quick Start

import torch
from torch.cuda.amp import autocast

uncond_input = tokenizer([""], padding="max_length", max_length=77, return_tensors="pt")
text_input = tokenizer([prompt], padding="max_length", max_length=77, truncation=True, return_tensors="pt")
with torch.no_grad():
    uncond_emb = text_encoder(uncond_input.input_ids.cuda())[0]
    text_emb = text_encoder(text_input.input_ids.cuda())[0]
text_embeddings = torch.cat([uncond_emb, text_emb])

latents = torch.randn(1, 4, 64, 64).cuda() * scheduler.init_noise_sigma
for i, t in enumerate(scheduler.timesteps):
    inp = torch.cat([latents] * 2)
    inp = scheduler.scale_model_input(inp, t)
    with torch.no_grad():
        noise_pred = unet(inp, t, encoder_hidden_states=text_embeddings).sample
    uncond, cond = noise_pred.chunk(2)
    noise_pred = uncond + 7.5 * (cond - uncond)
    latents = scheduler.step(noise_pred, t, latents).prev_sample

Workflow

1. Encode empty string and target prompt into text embeddings, concatenate
2. Initialize random latents scaled by the scheduler's initial noise sigma
3. For each timestep: duplicate latents, run UNet to get both unconditional and conditional noise predictions
4. Interpolate: noise = uncond + guidance_scale * (cond - uncond)
5. Step the scheduler to update latents
6. Decode final latents through VAE (scale by 1/0.18215 first)

Key Decisions

• Guidance scale: 7.5 is the standard default; 3-5 for more diversity, 10-15 for strict prompt adherence
• Scheduler: LMS, DDIM, or Euler-A — LMS is smooth, DDIM allows fewer steps, Euler-A adds stochasticity
• Inference steps: 20-50; diminishing returns beyond 50 for most schedulers
• VAE scaling: always divide latents by 0.18215 before decoding (SD v1.x convention)

References

• Text to Image generation (Stable Diffusion)