wenmin-wu/nlp-transformer-embedding-svr

Transformer Embedding + SVR

Overview

Instead of fine-tuning transformers end-to-end, extract mean-pooled embeddings from multiple frozen pretrained models, concatenate them into a high-dimensional feature vector, and train a lightweight SVR on top. This avoids GPU-intensive fine-tuning while achieving competitive scores.

Quick Start

import torch
from transformers import AutoTokenizer, AutoModel
from cuml.svm import SVR  # GPU-accelerated

def extract_embeddings(texts, model_name, batch_size=32):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModel.from_pretrained(model_name).eval().cuda()
    embeddings = []
    for i in range(0, len(texts), batch_size):
        batch = tokenizer(texts[i:i+batch_size], padding=True,
                         truncation=True, max_length=512, return_tensors="pt").to("cuda")
        with torch.no_grad():
            out = model(**batch).last_hidden_state
        mask = batch["attention_mask"].unsqueeze(-1).float()
        pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
        embeddings.append(pooled.cpu().numpy())
    return np.vstack(embeddings)

# Concatenate embeddings from multiple models
feats = np.hstack([extract_embeddings(texts, m) for m in model_names])
svr = SVR(C=1)
svr.fit(feats_train, y_train)

Workflow

1. Select 3-5 diverse pretrained models (e.g., DeBERTa-v3-base/large, DeBERTa-v2-xlarge)
2. Extract mean-pooled embeddings from each (frozen, no gradients)
3. Concatenate into single feature matrix (e.g., 5 models x 1024 = 5120 dims)
4. Train one SVR per target variable with GPU-accelerated cuML
5. Average predictions across K folds

Key Decisions

• Model diversity: Mix base/large/xlarge for complementary representations
• cuML SVR: 100x faster than sklearn on GPU; critical for 25-fold CV
• Per-target models: Train separate SVR per target — each target has different feature importance
• No fine-tuning: Trades marginal accuracy for massive speed and simplicity

References

• Feedback Prize - English Language Learning (Kaggle)
• Source: rapids-svr-cv-0-450-lb-0-44x