ADu2021/distribution-matching-vae

Overview

DMVAE generalizes beyond conventional Gaussian priors by explicitly aligning the encoder's latent distribution with an arbitrary reference distribution. The framework enables matching with distributions derived from self-supervised learning, diffusion noise, or other priors—moving beyond rigid architectural constraints.

When to Use

• Image generation models where latent distribution affects quality
• Scenarios exploring optimal latent structures for specific tasks
• Applications where self-supervised learning distributions outperform Gaussians
• Models needing flexibility in distribution choice
• Seeking efficient training (64 epochs on ImageNet)

When NOT to Use

• Models already achieving satisfactory results with Gaussian priors
• Fixed, rigid latent assumptions are acceptable
• Scenarios where distribution flexibility adds complexity without benefit

Core Technique

Explicit distribution matching for flexible latent space design:

# Distribution Matching VAE
class DistributionMatchingVAE:
    def __init__(self, encoder, decoder, reference_distribution=None):
        self.encoder = encoder
        self.decoder = decoder
        self.reference_dist = reference_distribution or self.default_gaussian()

    def forward(self, x):
        """
        Encode to latent space and reconstruct.
        Explicitly match latent distribution to reference.
        """
        # Encode
        latent, encoder_params = self.encoder(x)

        # Reconstruction
        recon = self.decoder(latent)

        return recon, latent, encoder_params

    def compute_distribution_matching_loss(self, latent_samples):
        """
        Explicitly align encoder latent distribution with reference.
        This replaces implicit KL divergence from fixed Gaussian prior.
        """
        # Measure distribution discrepancy
        # e.g., Wasserstein distance, Maximum Mean Discrepancy, etc.
        matching_loss = self.compute_distribution_distance(
            latent_samples,
            self.reference_dist
        )

        return matching_loss

    def train_step(self, batch):
        """
        DMVAE training combines reconstruction with distribution matching.
        """
        x = batch

        # Forward pass
        recon, latent, encoder_params = self.forward(x)

        # Reconstruction loss (pixel-space quality)
        recon_loss = torch.nn.functional.mse_loss(recon, x)

        # Distribution matching loss (latent structure)
        dist_matching_loss = self.compute_distribution_matching_loss(latent)

        # Combined objective
        total_loss = recon_loss + self.beta * dist_matching_loss

        return total_loss

    def use_ssl_derived_distribution(self, ssl_model, dataset):
        """
        Match encoder to self-supervised learning distribution.
        SSL distributions balance reconstruction fidelity and efficiency.
        """
        # Extract SSL features as reference distribution
        ssl_features = []
        for batch in dataset:
            features = ssl_model.encode(batch)
            ssl_features.append(features)

        ssl_features = torch.cat(ssl_features, dim=0)

        # Fit reference distribution to SSL features
        # (e.g., mixture of Gaussians, normalizing flow, etc.)
        self.reference_dist = self.fit_distribution_to_features(
            ssl_features
        )

        return self.reference_dist

    def use_diffusion_noise_distribution(self, diffusion_model):
        """
        Match encoder to diffusion model noise distribution.
        Enables alignment with denoising training paradigms.
        """
        # Diffusion models implicitly define noise distributions
        # at different timesteps
        noise_samples = diffusion_model.sample_noise_distribution()
        self.reference_dist = self.fit_distribution_to_features(
            noise_samples
        )

        return self.reference_dist

    def compute_distribution_distance(self, samples, reference):
        """
        Measure discrepancy between sample distribution and reference.
        Supports multiple distance metrics for flexibility.
        """
        if self.metric == 'wasserstein':
            # Wasserstein distance via optimal transport
            distance = self.wasserstein_distance(samples, reference)
        elif self.metric == 'mmd':
            # Maximum Mean Discrepancy
            distance = self.maximum_mean_discrepancy(samples, reference)
        elif self.metric == 'ks':
            # Kolmogorov-Smirnov test
            distance = self.ks_distance(samples, reference)
        else:
            raise ValueError(f"Unknown metric: {self.metric}")

        return distance

The framework systematically investigates which latent distributions are optimal for image synthesis. SSL-derived distributions provide superior performance for bridging high-fidelity synthesis with computational efficiency (gFID 3.2).

Key Results

• gFID 3.2 on ImageNet with 64 training epochs
• Self-supervised learning distributions outperform Gaussian priors
• Competitive with standard VAE approaches
• Flexible distribution choice enables optimization
• Reduced training time while maintaining quality

Implementation Notes

• Reference distribution can be arbitrary (SSL features, diffusion, etc.)
• Matching constraint replaces fixed KL divergence
• Multiple distance metrics supported (Wasserstein, MMD, KS)
• Training converges in significantly fewer epochs
• Can compare different distribution choices systematically

References

• Original paper: https://arxiv.org/abs/2512.07778
• Focus: Latent distribution design in VAEs
• Domain: Generative models, image synthesis