wenmin-wu/cv-semi-supervised-pretrained-backbone

Semi-Supervised Pretrained Backbone

Overview

Facebook's semi-supervised and semi-weakly supervised ImageNet models were trained on up to 940M unlabeled images from YFCC100M, producing significantly better representations than standard supervised pretraining. Using these as backbones (ResNet-50, ResNeXt-50) provides 1-3% accuracy improvements on downstream tasks, especially on medical imaging and domain-shifted data where standard ImageNet features are weak. Available via torch.hub with no extra dependencies.

Quick Start

import torch
import torch.nn as nn

# Load semi-weakly supervised ResNeXt-50
backbone = torch.hub.load(
    'facebookresearch/semi-supervised-ImageNet1K-models',
    'resnext50_32x4d_swsl'  # swsl = semi-weakly supervised
)

class CustomModel(nn.Module):
    def __init__(self, n_classes=6):
        super().__init__()
        self.encoder = nn.Sequential(*list(backbone.children())[:-2])
        nc = list(backbone.children())[-1].in_features  # 2048
        self.head = nn.Sequential(
            nn.AdaptiveAvgPool2d(1), nn.Flatten(),
            nn.Linear(nc, 512), nn.ReLU(),
            nn.Dropout(0.5), nn.Linear(512, n_classes))

    def forward(self, x):
        return self.head(self.encoder(x))

model = CustomModel(n_classes=6)

Workflow

1. Load pretrained model via torch.hub.load
2. Strip the final classification layer
3. Add custom pooling + classification head
4. Fine-tune end-to-end on target dataset

Key Decisions

• Model variants: resnext50_32x4d_ssl (semi-supervised), resnext50_32x4d_swsl (semi-weakly supervised — stronger)
• vs supervised: SWSL models consistently outperform supervised on transfer tasks
• vs DINO/MAE: SWSL is older but simpler to use; no special fine-tuning needed
• Available architectures: ResNet-18/50, ResNeXt-50/101 in both SSL and SWSL variants

References

• PANDA concat tile pooling starter