ADu2021/adaptive-data-refinement-vlm-long-tail

Core Concept

Large Vision-Language Models (LVLMs) suffer from long-tail distribution problems where common concepts are overrepresented while rare concepts are underrepresented in training data. The ADR (Adaptive Data Refinement) framework addresses this by: (1) analyzing entity distributions across token, object, co-occurrence, and query perspectives; (2) adaptively rebalancing redundant head-category data; and (3) synthesizing new samples for underrepresented tail categories using diffusion models.

Architecture Overview

ADR consists of two sequential stages:

• Data Rebalancing (DR) Stage: Analyzes entity distributions and removes redundant samples from over-represented classes while retaining diverse examples
• Data Synthesis (DS) Stage: Uses Denoising Diffusion Probabilistic Models (DDPMs) to generate new images for tail entities paired with LLM-synthesized text descriptions
• Entity Distribution Analysis: Four perspectives identify imbalance: token-level vocabulary distribution, object detection distributions, co-occurrence patterns, and query-based question distributions

Implementation Steps

1. Entity Distribution Construction and Analysis

Analyze training data to identify head vs. tail entity distributions across multiple perspectives:

# Entity distribution analysis for long-tail detection
from collections import defaultdict, Counter
import numpy as np

def analyze_entity_distributions(dataset, perspectives=['token', 'object']):
    """
    Construct entity distributions across multiple perspectives.

    Args:
        dataset: list of (image, text) tuples from VLM training
        perspectives: analysis dimensions ['token', 'object', 'cooccurrence']

    Returns:
        dict with distributions and head/tail split points
    """
    distributions = {}

    if 'token' in perspectives:
        # Tokenize all captions and count vocabulary frequency
        token_freq = Counter()
        for _, caption in dataset:
            tokens = caption.lower().split()
            token_freq.update(tokens)

        # Identify head/tail split (e.g., 80/20 principle)
        freq_sorted = sorted(token_freq.values(), reverse=True)
        cumsum = np.cumsum(freq_sorted)
        total = cumsum[-1]
        head_idx = np.where(cumsum >= 0.8 * total)[0][0]
        head_tokens = set([t for t, f in token_freq.items()
                          if f >= freq_sorted[head_idx]])

        distributions['token'] = {
            'freq': token_freq,
            'head_tokens': head_tokens,
            'tail_tokens': set(token_freq.keys()) - head_tokens,
            'head_count': sum(f for t, f in token_freq.items()
                             if t in head_tokens),
        }

    if 'object' in perspectives:
        # Run object detection on images and aggregate
        object_freq = Counter()
        for image, _ in dataset:
            objects = detect_objects(image)  # Requires detector
            object_freq.update(objects)

        # Find head/tail objects
        freq_sorted = sorted(object_freq.values(), reverse=True)
        cumsum = np.cumsum(freq_sorted)
        total = cumsum[-1]
        head_idx = np.where(cumsum >= 0.8 * total)[0][0]
        head_objects = set([o for o, f in object_freq.items()
                           if f >= freq_sorted[head_idx]])

        distributions['object'] = {
            'freq': object_freq,
            'head_objects': head_objects,
            'tail_objects': set(object_freq.keys()) - head_objects,
        }

    return distributions

2. Data Rebalancing Through Probability Dictionary

Create a weighting scheme to downsample head categories and preserve tail categories:

def construct_probability_dictionary(dataset, distributions,
                                    head_downsample_ratio=0.5):
    """
    Build probability weights for each sample based on entity prevalence.

    Args:
        dataset: training data
        distributions: entity frequency distributions
        head_downsample_ratio: fraction of head samples to keep

    Returns:
        sample_weights: probability of keeping each sample
    """
    token_dist = distributions.get('token', {})
    head_tokens = token_dist.get('head_tokens', set())

    sample_weights = []

    for image, caption in dataset:
        tokens = set(caption.lower().split())

        # Count how many head tokens in this sample
        head_token_count = len(tokens & head_tokens)
        total_token_count = len(tokens)

        # Samples with mostly head content get downsampled
        if total_token_count > 0:
            head_ratio = head_token_count / total_token_count
        else:
            head_ratio = 0.0

        # Downweight head-heavy samples, preserve tail
        if head_ratio > 0.8:  # Mostly head content
            weight = head_downsample_ratio
        elif head_ratio > 0.5:  # Mixed
            weight = 0.75
        else:  # Mostly tail content
            weight = 1.0

        sample_weights.append(weight)

    # Normalize to probabilities
    sample_weights = np.array(sample_weights)
    sample_weights = sample_weights / sample_weights.sum()

    return sample_weights

3. Diffusion-Based Visual Data Synthesis for Tail Categories

Use diffusion models to generate images for underrepresented tail entities:

def synthesize_tail_data(tail_entities, num_samples_per_entity,
                        text_generator, diffusion_model):
    """
    Generate synthetic images for tail entity categories.

    Args:
        tail_entities: list of underrepresented object/concept names
        num_samples_per_entity: how many images to generate per entity
        text_generator: LLM for generating diverse captions
        diffusion_model: DDPM for image generation

    Returns:
        list of (synthetic_image, caption) tuples
    """
    synthetic_data = []

    for entity in tail_entities:
        # Generate diverse text descriptions for this entity
        prompts = []
        for i in range(num_samples_per_entity):
            # Use LLM to generate varied descriptions
            prompt = text_generator(
                f"Generate a diverse visual description of a {entity} "
                f"variation {i+1}. Include scene context, colors, pose.",
                temperature=0.8
            )
            prompts.append(prompt)

        # Use diffusion model to generate images from text
        for prompt in prompts:
            # Add entity keyword to ensure relevance
            full_prompt = f"{entity}, {prompt}"

            # Generate with diffusion model
            image = diffusion_model.generate(
                prompt=full_prompt,
                num_steps=50,
                guidance_scale=7.5,
                seed=None  # Randomize for diversity
            )

            synthetic_data.append((image, full_prompt))

    return synthetic_data

Practical Guidance

When to Use ADR

• Training vision-language models on diverse datasets with natural long-tail distributions
• Need to improve tail category performance without adding more real data
• Models like LLaVA where object/concept distribution is imbalanced
• Want to improve generalization on rare visual concepts

When NOT to Use

• Datasets already well-balanced across categories
• Where synthetic data quality is critical (medical imaging, etc.)
• When tail categories should remain underrepresented by design
• Very small datasets where rebalancing statistics are unreliable

Hyperparameters & Configuration

• Head threshold percentile: 80% (80/20 Pareto principle); adjust based on data skewness
• Head downsample ratio: 0.5 (keep 50% of head samples); balance between coverage and tail emphasis
• Synthesis samples per entity: 5-10 per tail entity; more diversity helps generalization
• Entity detection model: YOLOv8 or similar for object detection
• Diffusion steps: 50 steps for quality; 25 for speed
• Guidance scale: 7.5 for text-image alignment; higher values (10+) strengthen entity presence

Common Pitfalls

• Over-rebalancing: Completely removing head categories hurts overall accuracy; keep some head data
• Poor caption quality for synthesis: LLM-generated captions must be descriptive; use temperature 0.7-0.8
• Synthetic data mode collapse: Diffusion models may generate similar images; use seed randomization
• Entity detection errors: Object detector mistakes propagate; validate detection on sample images
• Ignoring statistical significance: Small tail categories may need higher synthesis count for reliability
• Validation set bias: Ensure test set reflects original distribution, not rebalanced distribution

Reference

• Kingma & Welling. 2014. Auto-Encoding Variational Bayes (VAE foundations).
• Ho et al. 2020. Denoising Diffusion Probabilistic Models (DDPM).
• Liu et al. 2019. Decoupling Representation and Classifier for Long-Tailed Recognition.
• LLaVA project: https://github.com/haotian-liu/LLaVA
• Project page: https://vlmlt.github.io/