harsh040506/generative-models

Generative Models — Diffusion, GANs, and Transformer LMs

Provides architecture patterns, loss function derivations, training stability techniques, and evaluation protocols for diffusion models, GANs, and large-scale language models.

Diffusion Models

Mathematical Foundation:

• Forward process (noising): q(x_t | x_{t-1}) = N(x_t; √(1−β_t)·x_{t-1}, β_t·I)
• Closed-form marginal: q(x_t | x_0) = N(x_t; √ᾱ_t·x_0, (1−ᾱ_t)·I)
• Reverse process (denoising): learn p_θ(x_{t-1} | x_t) by predicting ε_θ(x_t, t) ≈ ε

Training Objective (simplified ELBO): L_simple = E_{t, x_0, ε}[||ε − ε_θ(√ᾱ_t·x_0 + √(1−ᾱ_t)·ε, t)||²]

Architecture: U-Net with:

• Residual blocks at each resolution
• Self-attention at low resolutions (32×32 and 16×16)
• Sinusoidal time-step embedding conditioning (d = 256)
• GroupNorm (32 groups) for training stability

Noise Schedules: linear (DDPM), cosine (improved DDPM), or sigmoid (more uniform SNR)

Sampling Acceleration: DDIM (50 steps), DPM-Solver (10–20 steps) vs. DDPM (1000 steps)

See references/diffusion-models.md for full derivation, implementation, and latent diffusion (LDM) patterns.

Generative Adversarial Networks

Minimax Objective: min_G max_D V(D,G) = E_{x~p_data}[log D(x)] + E_{z~p_z}[log(1 − D(G(z)))]

Critical Stabilization Techniques (apply all for stable training):

1. Spectral normalization on D: constrains Lipschitz constant, prevents discriminator from becoming too powerful
2. Label smoothing: real labels = 0.9 (not 1.0), fake labels = 0.1 (not 0.0)
3. WGAN-GP gradient penalty: λ·E[(||∇D(x̂)||₂ − 1)²], λ = 10, eliminates mode collapse risk
4. TTUR: use LR_D = 4e-4, LR_G = 1e-4 (different timescales stabilize training)
5. MiniBatch discrimination: prevents G from always generating the same output

Evaluation Metric: FID (Fréchet Inception Distance) — lower is better; FID < 10 is state-of-the-art for faces.

See references/gans.md for architecture variants (DCGAN → StyleGAN2 → ProjectedGAN) and CTGAN for tabular data.

Transformer Language Models

Core Architecture (decoder-only, GPT-style):

• Token embedding (vocab_size → d_model) + positional encoding (RoPE preferred)
• N × blocks: [LayerNorm → MHA → residual] → [LayerNorm → FFN(4×d) → residual]
• Attention: Attention(Q,K,V) = softmax(QK^T / √d_k)·V
• Output: linear projection to vocab_size → softmax → token probabilities

Scaling Guidance (Chinchilla-optimal, Hoffmann et al. 2022):

• Optimal tokens T for N parameters: T_opt = 20·N
• Equal scaling of model size and data is key — do not over- or under-train

Training Techniques for LLMs:

• Mixed precision BF16 (A100+), FP16 + GradScaler (V100)
• FlashAttention-2 for memory-efficient O(n) attention
• Gradient accumulation to achieve large effective batch sizes without large VRAM
• Learning rate: warmup to peak over 1% of steps, then cosine decay to 10% of peak

See references/transformer-lms.md for full implementation, tokenization, and RLHF fine-tuning patterns.

Quality Evaluation by Model Type

| Model Type | Primary Metric | Secondary Metrics | |---|---|---| | Diffusion (images) | FID (↓) | IS (↑), CLIP score, human preference | | GAN (images) | FID (↓) | Precision, Recall, coverage | | Language Model | Perplexity (↓) | BLEU, ROUGE, BERTScore, task benchmarks | | Conditional generation | Alignment score | Diversity, fidelity vs. reference |