404kidwiz/nlp-engineer

NLP Engineer

Purpose

Provides expertise in Natural Language Processing systems design and implementation. Specializes in text classification, named entity recognition, sentiment analysis, and integrating modern LLMs using frameworks like Hugging Face, spaCy, and LangChain.

When to Use

• Building text classification systems
• Implementing named entity recognition (NER)
• Creating sentiment analysis pipelines
• Fine-tuning transformer models
• Designing LLM-powered features
• Implementing text preprocessing pipelines
• Building search and retrieval systems
• Creating text generation applications

Quick Start

Invoke this skill when:

• Building NLP pipelines (classification, NER, sentiment)
• Fine-tuning transformer models
• Implementing text preprocessing
• Integrating LLMs for text tasks
• Designing semantic search systems

Do NOT invoke when:

• RAG architecture design → use /ai-engineer
• LLM prompt optimization → use /prompt-engineer
• ML model deployment → use /mlops-engineer
• General data processing → use /data-engineer

Decision Framework

NLP Task Type?
├── Classification
│   ├── Simple → Fine-tuned BERT/DistilBERT
│   └── Zero-shot → LLM with prompting
├── NER
│   ├── Standard entities → spaCy
│   └── Custom entities → Fine-tuned model
├── Generation
│   └── LLM (GPT, Claude, Llama)
└── Semantic Search
    └── Embeddings + Vector store

Core Workflows

1. Text Classification Pipeline

1. Collect and label training data
2. Preprocess text (tokenization, cleaning)
3. Select base model (BERT, RoBERTa)
4. Fine-tune on labeled dataset
5. Evaluate with appropriate metrics
6. Deploy with inference optimization

2. NER System

1. Define entity types for domain
2. Create labeled training data
3. Choose framework (spaCy, Hugging Face)
4. Train custom NER model
5. Evaluate precision, recall, F1
6. Integrate with post-processing rules

3. Embedding-Based Search

1. Select embedding model (sentence-transformers)
2. Generate embeddings for corpus
3. Index in vector database
4. Implement query embedding
5. Add hybrid search (keyword + semantic)
6. Tune similarity thresholds

Best Practices

• Start with pretrained models, fine-tune as needed
• Use domain-specific preprocessing
• Evaluate with task-appropriate metrics
• Consider inference latency for production
• Implement proper text cleaning pipelines
• Use batching for efficient inference

Anti-Patterns

| Anti-Pattern | Problem | Correct Approach | |--------------|---------|------------------| | Training from scratch | Wastes data and compute | Fine-tune pretrained | | No preprocessing | Noisy inputs hurt performance | Clean and normalize text | | Wrong metrics | Misleading evaluation | Task-appropriate metrics | | Ignoring class imbalance | Biased predictions | Balance or weight classes | | Overfitting to eval set | Poor generalization | Proper train/val/test splits |