curiositech/knowledge-distillation-survey

Knowledge Distillation: A Survey

Core Concept

Knowledge distillation transfers learning from teacher systems to students across capability gaps. The key insight: effective transfer requires matching knowledge type, teacher capacity, and transfer mechanism to the student's representational constraints.

DECISION POINTS

Primary Decision Tree: Choosing Transfer Approach

1. ASSESS CAPACITY GAP
   ├── Small gap (teacher 1-2x student params) → Direct offline distillation
   ├── Large gap (teacher 10x+ student params) → Progressive with teacher assistants  
   └── Similar capacity/peer learning → Online collaborative distillation

2. IDENTIFY KNOWLEDGE TYPE NEEDED
   ├── Task needs final decisions → Response-based (soft labels, logits)
   ├── Task needs pattern recognition → Feature-based (intermediate representations)
   ├── Task needs structural reasoning → Relation-based (similarity matrices, attention maps)
   └── Task needs multiple types → Multi-level distillation

3. SELECT TRANSFER MECHANISM
   ├── Proven expert + stable domain → Offline (sequential training)
   ├── Exploration needed + multiple learners → Online (mutual learning)
   ├── No external teacher available → Self-distillation (temporal/spatial)
   └── Cross-modal/domain required → Alignment-based transfer

4. HANDLE REPRESENTATION GAPS
   ├── Same modality → Direct feature matching
   ├── Different modalities → Paired sample alignment
   ├── Different architectures → Attention transfer or feature adaptation
   └── Different tasks → Structural knowledge extraction

Capacity Gap Diagnostic

| Teacher-Student Performance Ratio | Approach | Rationale | |-----------------------------------|----------|-----------| | 1.1-1.5x | Direct distillation | Student can decode teacher knowledge | | 1.5-3x | Add 1 teacher assistant | Bridge representational gap | | 3x+ | Multi-step progressive | Prevent knowledge loss in translation | | Similar performance | Collaborative learning | Mutual improvement through diversity |

FAILURE MODES

1. Capacity Mismatch Collapse

• Symptom: Student performs worse with distillation than training alone
• Detection: Student accuracy drops >2% vs independent baseline
• Diagnosis: Teacher-student gap too large; student cannot represent teacher's compressed knowledge
• Fix: Insert teacher assistant with intermediate capacity, or reduce teacher complexity

2. Knowledge Type Confusion

• Symptom: Good distillation loss but poor task performance
• Detection: Temperature scaling doesn't improve results; student mimics confidences but wrong decisions
• Diagnosis: Transferring wrong knowledge dimension (e.g., response-based when student needs feature-based)
• Fix: Switch to feature distillation or relation-based transfer matching student's architectural constraints

3. Teacher Overconfidence Amplification

• Symptom: Student more confident than teacher but less accurate
• Detection: Student entropy lower than teacher entropy on validation set
• Diagnosis: Teacher's wrong predictions transferred with high confidence
• Fix: Increase temperature, add label smoothing, or use multiple diverse teachers

4. Cross-Modal Alignment Failure

• Symptom: Same-modality distillation works, cross-modal distillation fails completely
• Detection: Cross-modal student performs at random baseline despite good teacher
• Diagnosis: Modality representations not properly aligned; no semantic correspondence
• Fix: Add alignment losses, use paired training data, or learn modality translation first

5. Progressive Learning Degradation

• Symptom: Each step in teacher assistant chain performs worse than previous
• Detection: Teacher assistant accuracy < 90% of previous teacher accuracy
• Diagnosis: Knowledge corruption cascading through chain; information bottlenecks
• Fix: Skip connections to original teacher, ensemble multiple assistants, or reduce chain length

WORKED EXAMPLES

Example 1: Large Language Model Compression

Scenario: Compress GPT-3.5 (175B params) to run on mobile (500M params max)

Decision Process:

1. Capacity gap assessment: 350x parameter difference = massive gap
2. Knowledge type: Need response-based (text generation) + relation-based (reasoning chains)
3. Progressive approach: GPT-3.5 → GPT-2 (1.5B) → GPT-small (500M)

Execution:

• Stage 1: Distill GPT-3.5 → GPT-2 using response-based (next token prediction) + attention transfer
• Stage 2: Distill GPT-2 → GPT-small using feature-based (hidden states) + relation-based (attention patterns)
• Validation: Test reasoning on held-out problems, not just perplexity

Expert catches: Temperature needs adjustment per stage; mobile model needs different attention patterns optimized for inference speed

Example 2: Cross-Modal Robotics Transfer

Scenario: Robot trained on RGB cameras must work with depth sensors only

Decision Process:

1. Cross-modal gap: RGB (3 channels) → Depth (1 channel) with different semantic info
2. Knowledge type: Feature-based (spatial understanding) + relation-based (object relationships)
3. Alignment strategy: Paired RGB-depth data during distillation

Execution:

• Collect paired RGB-depth sequences of same scenes
• Train alignment network to map depth features → RGB feature space
• Distill RGB teacher → depth student using aligned feature representations
• Validate on manipulation tasks requiring spatial reasoning

Expert catches: Depth lacks color/texture info; student needs different attention for material properties

QUALITY GATES

Task completion checklist for knowledge distillation:

• [ ] Student validation accuracy ≥ 95% of teacher accuracy (or ≥ 105% of independent baseline)
• [ ] Knowledge transfer verified: Student generalizes to held-out test set, not just mimicking training
• [ ] Capacity appropriateness: Teacher-student gap assessed; progressive learning used if gap >3x performance
• [ ] Knowledge type alignment: Response/feature/relation distillation matches student architecture capabilities
• [ ] Cross-validation performed: Student tested on different data distribution than teacher training
• [ ] Failure mode check: No overconfidence, capacity mismatch, or degradation symptoms detected
• [ ] Compression ratio documented: Inference speedup and memory reduction quantified vs. teacher
• [ ] Transfer mechanism justified: Offline/online/self-distillation choice explained with rationale
• [ ] Boundary conditions tested: Student performance verified at edge cases where teacher might fail
• [ ] Knowledge preservation verified: Critical capabilities from teacher successfully transferred and validated

NOT-FOR BOUNDARIES

Do NOT use knowledge distillation for:

• Simple rule-based systems: Use direct rule transfer instead - no learning gap to bridge
• When teacher and student solve fundamentally different problems: Use transfer-learning-frameworks for task adaptation
• Real-time systems where student must exceed teacher speed: Use model-optimization-techniques for architecture efficiency
• When student needs capabilities teacher lacks: Use curriculum-learning-design for progressive skill building
• Safety-critical systems without validation frameworks: Use robust-ai-validation for safety assurance first
• When interpretability is primary concern: Use explainable-ai-methods - distillation often reduces interpretability

Delegate to other skills:

• Model architecture optimization → neural-architecture-search
• Cross-task transfer learning → transfer-learning-frameworks
• Training data efficiency → few-shot-learning-strategies
• Model safety and robustness → robust-ai-validation
• Deployment optimization → model-optimization-techniques