ADu2021/concerto-joint-learning

Concerto: Joint 2D-3D Self-Supervised Representation Learning

Human spatial understanding emerges from multiple sensory modalities simultaneously. Concerto mirrors this by learning from paired 2D images and 3D point clouds, discovering representations richer than either modality alone.

The joint learning approach creates spatial features with superior geometric and semantic consistency, improving downstream scene understanding tasks.

Core Concept

Key insight: simultaneous 2D-3D training creates more coherent spatial concepts than training separately. Concerto uses:

• 3D intra-modal self-distillation (learning within 3D point clouds)
• 2D-3D cross-modal joint embedding (aligning image and 3D representations)
• Multi-view consistency to reinforce learned concepts

Architecture Overview

• Separate encoders for 2D and 3D modalities
• Cross-modal contrastive learning between image and 3D features
• Intra-modal self-distillation within 3D representations
• Multi-view consistency constraints across modalities

Implementation Steps

Create dual-stream encoders for 2D and 3D data. Each stream learns modality-specific representations while staying aligned through contrastive loss:

class ConcertoEncoder(nn.Module):
    def __init__(self, feature_dim=256):
        super().__init__()
        # 2D encoder (e.g., Vision Transformer)
        self.image_encoder = ViT(
            patch_size=16,
            num_layers=12,
            hidden_dim=768
        )
        # 3D encoder (e.g., PointNet++)
        self.pointcloud_encoder = PointNet(
            num_layers=4,
            feature_dim=feature_dim
        )

        # Projection heads for alignment
        self.image_proj = nn.Sequential(
            nn.Linear(768, 512),
            nn.ReLU(),
            nn.Linear(512, feature_dim)
        )
        self.pc_proj = nn.Sequential(
            nn.Linear(feature_dim, 512),
            nn.ReLU(),
            nn.Linear(512, feature_dim)
        )

    def forward(self, images, point_clouds):
        # Encode both modalities
        image_feat = self.image_encoder(images)
        pc_feat = self.pointcloud_encoder(point_clouds)

        # Project to common space
        image_proj = self.image_proj(image_feat)
        pc_proj = self.pc_proj(pc_feat)

        return image_proj, pc_proj

Implement cross-modal contrastive learning that aligns 2D and 3D representations. Instances from the same scene should have similar features:

class CrossModalContrastiveLoss(nn.Module):
    def __init__(self, temperature=0.07):
        super().__init__()
        self.temperature = temperature

    def forward(self, image_proj, pc_proj):
        """Contrastive loss between 2D and 3D projections."""
        # Normalize features
        image_proj = torch.nn.functional.normalize(image_proj, dim=-1)
        pc_proj = torch.nn.functional.normalize(pc_proj, dim=-1)

        # Compute similarity matrix
        batch_size = image_proj.shape[0]
        logits = torch.mm(image_proj, pc_proj.t()) / self.temperature

        # Labels: diagonal elements are positives (same scene)
        labels = torch.arange(batch_size, device=image_proj.device)

        # Symmetric cross-entropy loss
        loss_i2p = torch.nn.functional.cross_entropy(logits, labels)
        loss_p2i = torch.nn.functional.cross_entropy(logits.t(), labels)

        return (loss_i2p + loss_p2i) / 2

Implement 3D intra-modal self-distillation using a momentum encoder for point clouds. This captures 3D geometry better than cross-modal learning alone:

class PointCloud3DDistillation(nn.Module):
    def __init__(self, encoder, momentum=0.999):
        super().__init__()
        self.encoder = encoder
        self.momentum_encoder = copy.deepcopy(encoder)
        self.momentum = momentum

        # Freeze momentum encoder
        for param in self.momentum_encoder.parameters():
            param.requires_grad = False

    def forward(self, point_clouds, aug_point_clouds):
        """Self-distillation within 3D modality."""
        # Online encoder
        feat = self.encoder(point_clouds)

        # Momentum encoder (no gradient)
        with torch.no_grad():
            feat_momentum = self.momentum_encoder(aug_point_clouds)

        # Knowledge distillation loss
        loss = torch.nn.functional.mse_loss(feat, feat_momentum.detach())

        # Update momentum encoder
        self._update_momentum_encoder()

        return loss

    def _update_momentum_encoder(self):
        """Update momentum encoder with EMA."""
        for param, momentum_param in zip(
            self.encoder.parameters(),
            self.momentum_encoder.parameters()
        ):
            momentum_param.data = (
                self.momentum * momentum_param.data +
                (1 - self.momentum) * param.data
            )

Practical Guidance

| Parameter | Recommendation | |-----------|-----------------| | Feature dimension | 256 (balance capacity and efficiency) | | Temperature | 0.07-0.1 (standard for contrastive learning) | | Momentum coefficient | 0.999 (slow teacher updates) | | Cross-modal weight | 1.0, 3D distillation weight | 0.5 | | Batch size | 256-512 (contrastive learning needs diverse negatives) |

When to use:

• Scene understanding tasks with both RGB and 3D data
• Indoor/outdoor robotics requiring spatial reasoning
• 3D reconstruction or depth estimation
• Multi-view geometry applications

When NOT to use:

• Single modality data only (more efficient methods exist)
• Tasks with abundant labeled data (supervised learning better)
• Real-time inference with latency constraints (dual encoders add cost)

Common pitfalls:

• Imbalanced modality contributions (one dominates)
• Insufficient momentum encoder updates (stale teacher)
• Cross-modal loss weight too low (poor alignment)
• Not aligning 2D views with 3D geometrically (learns spurious correlations)

Reference: Concerto on arXiv