wenmin-wu/nlp-lda-topic-modeling

LDA Topic Modeling

Overview

Latent Dirichlet Allocation discovers latent topics in a text corpus. Each document is modeled as a mixture of topics, each topic as a distribution over words. Fit LDA on a CountVectorizer bag-of-words matrix to extract topic distributions per document — usable as features for downstream models or for understanding corpus structure.

Quick Start

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

cvec = CountVectorizer(min_df=4, max_features=50000,
                       ngram_range=(1, 2), stop_words='english')
bow = cvec.fit_transform(texts)

lda = LatentDirichletAllocation(
    n_components=20, learning_method='online',
    max_iter=20, random_state=42)
topic_dist = lda.fit_transform(bow)  # shape: (n_docs, n_topics)

vocab = cvec.get_feature_names_out()
for i, topic in enumerate(lda.components_):
    top_words = [vocab[j] for j in topic.argsort()[:-11:-1]]
    print(f"Topic {i}: {', '.join(top_words)}")

Workflow

1. Build bag-of-words with CountVectorizer (not TF-IDF — LDA expects raw counts)
2. Fit LDA with learning_method='online' for large corpora
3. fit_transform returns per-document topic distributions (features)
4. Inspect lda.components_ to label topics by their top words
5. Append topic distributions as features to the original dataset

Key Decisions

• n_components: 10-30 for EDA, tune via coherence score for production
• CountVectorizer not TfidfVectorizer: LDA's generative model assumes word counts
• online vs batch: online is faster for large datasets, batch is more accurate
• min_df: 3-5 removes rare terms that create noise topics

References

• Mercari Interactive EDA + Topic Modelling