wenmin-wu/nlp-averaged-meta-embedding

Overview

Concatenating embedding matrices (GloVe + Paragram + FastText) doubles or triples the input dimension, inflating model size. An unweighted mean of the matrices — Dynamic Meta Embedding (DME) — preserves the original dimension while combining the semantic signal from each source. It's parameter-free, adds no inference cost, and often performs comparably to weighted blends.

Quick Start

import numpy as np

def load_glove(word_index, embed_size):
    # ... returns (vocab_size, embed_size)
    ...

def load_paragram(word_index, embed_size):
    # ... returns (vocab_size, embed_size)
    ...

embedding_matrix_1 = load_glove(word_index, 300)
embedding_matrix_2 = load_paragram(word_index, 300)

# Unweighted DME: element-wise mean, same shape as inputs
embedding_matrix = np.mean([embedding_matrix_1, embedding_matrix_2], axis=0)
# Shape: (vocab_size, 300) — NOT (vocab_size, 600)

Workflow

1. Build embedding matrices from each pretrained source using the same word→index mapping
2. Stack them and take np.mean(..., axis=0) — result has the same shape as each source
3. Pass the averaged matrix to your Embedding layer
4. Train the model normally — no special architecture changes required

Key Decisions

• Dim compatibility: All matrices must share the same dimension. Project mismatched sources with a linear layer or drop them.
• vs. concat: Concat doubles input dim, increases first-layer parameters. Mean keeps dim fixed, no parameter growth.
• vs. weighted mean: Simple mean is often within 0.5% of weighted blends and needs no hyperparameter search.
• Init alignment: Ensure OOV rows are handled consistently across sources before averaging, or the mean becomes noisy.

References

• Single RNN with 4 folds (CLR)