tachyon-beep/using-neural-architectures

Using Neural Architectures: Architecture Selection Router

<CRITICAL_CONTEXT> Architecture selection comes BEFORE training optimization. Wrong architecture = no amount of training will fix it.

This meta-skill routes you to the right architecture guidance based on:

• Data modality (images, sequences, graphs, etc.)
• Problem type (classification, generation, regression)
• Constraints (data size, compute, latency, interpretability)

Load this skill when architecture decisions are needed. </CRITICAL_CONTEXT>

When to Use This Skill

Use this skill when:

• ✅ Selecting an architecture for a new problem
• ✅ Comparing architecture families (CNN vs Transformer, RNN vs Transformer, etc.)
• ✅ Designing custom network topology
• ✅ Troubleshooting architectural instability (deep networks, gradient issues)
• ✅ Understanding when to use specialized architectures (GNNs, generative models)

DO NOT use for:

• ❌ Training/optimization issues (use training-optimization pack)
• ❌ PyTorch implementation details (use pytorch-engineering pack)
• ❌ Production deployment (use ml-production pack)

When in doubt: If choosing WHAT architecture → this skill. If training/deploying architecture → different pack.

---

How to Access Reference Sheets

IMPORTANT: All reference sheets are located in the SAME DIRECTORY as this SKILL.md file.

When this skill is loaded from: skills/using-neural-architectures/SKILL.md

Reference sheets like cnn-families-and-selection.md are at: skills/using-neural-architectures/cnn-families-and-selection.md

NOT at: skills/cnn-families-and-selection.md ← WRONG PATH

When you see a link like [cnn-families-and-selection.md](cnn-families-and-selection.md), read the file from the same directory as this SKILL.md.

---

Core Routing Logic

Step 1: Identify Data Modality

Question to ask: "What type of data are you working with?"

| Data Type | Route To | Why | |-----------|----------|-----| | Images (photos, medical scans, etc.) | cnn-families-and-selection.md | CNNs excel at spatial hierarchies | | Sequences (time series, text, audio) | sequence-models-comparison.md | Temporal dependencies need sequential models | | Graphs (social networks, molecules) | graph-neural-networks-basics.md | Graph structure requires GNNs | | Generation task (create images, text) | generative-model-families.md | Generative models are specialized | | Multiple modalities (text + images, audio, video) | multimodal-architectures.md | Vision-language, CLIP/SigLIP, LLaVA, Flamingo, native multimodal | | Unclear / Generic | architecture-design-principles.md | Start with fundamentals |

Step 2: Check for Special Requirements

If any of these apply, address FIRST:

| Requirement | Route To | Priority | |-------------|----------|----------| | Deep network (> 20 layers) unstable | normalization-techniques.md | CRITICAL - fix before continuing | | Need attention mechanisms | attention-mechanisms-catalog.md | Specialized component | | Custom architecture design | architecture-design-principles.md | Foundation before specifics | | Transformer-specific question | transformer-architecture-deepdive.md | Specialized architecture |

Step 3: Consider Problem Characteristics

Clarify BEFORE routing:

Ask:

• "How large is your dataset?" (Small < 10k, Medium 10k-1M, Large > 1M)
• "What are your computational constraints?" (Edge device, cloud, GPU availability)
• "What are your latency requirements?" (Real-time, batch, offline)
• "Do you need interpretability?" (Clinical, research, production)

These answers determine architecture appropriateness.

---

Routing by Data Modality

Images → CNN Families

Symptoms triggering this route:

• "classify images"
• "object detection"
• "semantic segmentation"
• "medical imaging"
• "computer vision"

Route to: See cnn-families-and-selection.md for CNN architecture selection and comparison.

When to route here:

• ANY vision task (CNNs are default for spatial data)
• Even if considering Transformers, check CNN families first (often better with less data)

Clarifying questions:

• "Dataset size?" (< 10k → Start with proven CNNs, > 100k → Consider ViT)
• "Deployment target?" (Edge → EfficientNet, Cloud → Anything)
• "Task type?" (Classification → ResNet/EfficientNet, Detection → YOLO/Faster-RCNN)

---

Sequences → Sequence Models Comparison

Symptoms triggering this route:

• "time series"
• "forecasting"
• "natural language" (NLP)
• "sequential data"
• "temporal patterns"
• "RNN vs LSTM vs Transformer"

Route to: See sequence-models-comparison.md for sequential model selection (RNN, LSTM, Transformer, TCN).

When to route here:

• ANY sequential data
• When user asks "RNN vs LSTM" (skill will present modern alternatives)
• Time-dependent patterns

Clarifying questions:

• "Sequence length?" (< 100 → RNN/LSTM/TCN, 100-1000 → Transformer, > 1000 → Sparse Transformers)
• "Latency requirements?" (Real-time → TCN/LSTM, Offline → Transformer)
• "Data volume?" (Small → Simpler models, Large → Transformers)

CRITICAL: Challenge "RNN vs LSTM" premise if they ask. Modern alternatives (Transformers, TCN) often better.

---

Graphs → Graph Neural Networks

Symptoms triggering this route:

• "social network"
• "molecular structure"
• "knowledge graph"
• "graph data"
• "node classification"
• "link prediction"
• "graph embeddings"

Route to: See graph-neural-networks-basics.md for GNN architectures and graph learning.

When to route here:

• Data has explicit graph structure (nodes + edges)
• Relational information is important
• Network topology matters

Red flag: If treating graph as tabular data (extracting features and ignoring edges) → WRONG. Route to GNN skill.

---

Generation → Generative Model Families

Symptoms triggering this route:

• "generate images"
• "synthesize data"
• "GAN vs VAE vs Diffusion"
• "image-to-image translation"
• "style transfer"
• "generative modeling"

Route to: See generative-model-families.md for GANs, VAEs, and Diffusion models.

When to route here:

• Goal is to CREATE data, not classify/predict
• Need to sample from distribution
• Data augmentation through generation

Clarifying questions:

• "Use case?" (Production text-to-image → SDXL/SD3/FLUX + LoRA; real-time/edge → distilled diffusion (LCM/Turbo) or a pretrained GAN niche; cross-modal generation → native multimodal (Chameleon/Gemini-class))
• "Quality vs speed?" (Quality → DiT-class diffusion; Speed → distilled diffusion before reaching for GAN — distillation closed most of the speed gap)
• "Controllability?" (Structural control → ControlNet on top of diffusion; identity / style → IP-Adapter or LoRA fine-tune)

CRITICAL: Different generative models have VERY different trade-offs. Don't reflexively recommend GAN for "fast" — distilled diffusion is the modern fast option. Don't recommend training Stable Diffusion from scratch.

---

Routing by Architecture Component

Attention Mechanisms

Symptoms triggering this route:

• "when to use attention"
• "self-attention vs cross-attention"
• "attention in CNNs"
• "attention bottleneck"
• "multi-head attention"

Route to: See attention-mechanisms-catalog.md for attention mechanism selection and design.

When to route here:

• Designing custom architecture that might benefit from attention
• Understanding where attention helps vs hinders
• Comparing attention variants

NOT for: General Transformer questions → transformer-architecture-deepdive.md instead

---

Transformer Deep Dive

Symptoms triggering this route:

• "how do transformers work"
• "Vision Transformer (ViT)"
• "BERT architecture"
• "positional encoding"
• "transformer blocks"
• "scaling transformers"

Route to: See transformer-architecture-deepdive.md for Transformer internals and implementation.

When to route here:

• Implementing/customizing transformers
• Understanding transformer internals
• Debugging transformer-specific issues

Cross-reference:

• For sequence models generally → sequence-models-comparison.md (includes transformers in context)
• For LLMs specifically → yzmir/llm-specialist/transformer-for-llms (LLM-specific transformers)

---

Normalization Techniques

Symptoms triggering this route:

• "gradient explosion"
• "training instability in deep network"
• "BatchNorm vs LayerNorm"
• "normalization layers"
• "50+ layer network won't train"

Route to: See normalization-techniques.md for deep network stability and normalization methods.

When to route here:

• Deep networks (> 20 layers) with training instability
• Choosing between normalization methods
• Architectural stability issues

CRITICAL: This is often the ROOT CAUSE of "training won't work" - fix architecture before blaming hyperparameters.

---

Architecture Design Principles

Symptoms triggering this route:

• "how to design architecture"
• "architecture best practices"
• "when to use skip connections"
• "how deep should network be"
• "custom architecture for [novel task]"
• Unclear problem modality

Route to: See architecture-design-principles.md for custom architecture design fundamentals.

When to route here:

• Designing custom architectures
• Novel problems without established architecture
• Understanding WHY architectures work
• User is unsure what modality/problem type they have

This is the foundational skill - route here if other specific skills don't match.

---

Multi-Modal / Cross-Pack Routing

When Problem Spans Multiple Modalities

Default route: multimodal-architectures.md — covers contrastive (CLIP / SigLIP), generative bolt-on (LLaVA / BLIP-2 / Flamingo), and native multimodal (Chameleon / Gemini-style) recipes, plus audio and video extensions.

Only fall back to component sheets if the user is building from scratch without using foundation models:

1. sequence-models-comparison.md (for text)
2. cnn-families-and-selection.md (for images)
3. architecture-design-principles.md (for hand-rolled fusion strategy)

Order matters: In 2026 the default for "vision + language" is SigLIP + MLP projector + open LLM, not designing fusion from first principles. Read the multimodal sheet before reaching for design-principles.

When Architecture + Other Concerns

Example: "Select architecture AND optimize training"

Route order:

1. Architecture skill FIRST (this pack)
2. Training-optimization SECOND (after architecture chosen)

Why: Wrong architecture can't be fixed by better training.

Example: "Select architecture AND deploy efficiently"

Route order:

1. Architecture skill FIRST
2. ML-production SECOND (quantization, serving)

Deployment constraints might influence architecture choice - if so, note constraints during architecture selection.

---

Common Routing Mistakes (DON'T DO THESE)

| Symptom | Wrong Route | Correct Route | Why | |---------|-------------|---------------|-----| | "My transformer won't train" | transformer-architecture-deepdive.md | training-optimization | Training issue, not architecture understanding | | "Deploy image classifier" | cnn-families-and-selection.md | ml-production | Deployment, not selection | | "ViT vs ResNet for medical imaging" | transformer-architecture-deepdive.md | cnn-families-and-selection.md | Comparative selection, not single architecture detail | | "Implement BatchNorm in PyTorch" | normalization-techniques.md | pytorch-engineering | Implementation, not architecture concept | | "GAN won't converge" | generative-model-families.md | training-optimization | Training stability, not architecture selection | | "Which optimizer for CNN" | cnn-families-and-selection.md | training-optimization | Optimization, not architecture |

Rule: Architecture pack is for CHOOSING and DESIGNING architectures. Training/deployment/implementation are other packs.

---

Red Flags: Stop and Clarify

If query contains these patterns, ASK clarifying questions before routing:

| Pattern | Why Clarify | What to Ask | |---------|-------------|--------------| | "Best architecture for X" | "Best" depends on constraints | "What are your data size, compute, and latency constraints?" | | Generic problem description | Can't route without modality | "What type of data? (images, sequences, graphs, etc.)" | | Latest trend mentioned (ViT, Diffusion) | Recency bias risk | "Have you considered alternatives? What are your specific requirements?" | | "Should I use X or Y" | May be wrong question | "What's the underlying problem? There might be option Z." | | Very deep network (> 50 layers) | Likely needs normalization first | "Are you using normalization layers? Skip connections?" |

Never guess modality or constraints. Always clarify.

---

Recency Bias: Resistance Table

| Trendy Architecture | When NOT to Use | Better Alternative | |---------------------|------------------|-------------------| | Vision Transformers (ViT) | Small datasets (< 10k images) | CNNs (ResNet, EfficientNet) | | Vision Transformers (ViT) | Edge deployment (latency/power) | EfficientNets, MobileNets | | Transformers (general) | Very small datasets | RNNs, CNNs (less capacity, less overfit) | | Diffusion Models | Real-time generation needed | GANs (1 forward pass vs 50-1000 steps) | | Diffusion Models | Limited compute for training | VAEs (faster training) | | Graph Transformers | Small graphs (< 100 nodes) | Standard GNNs (GCN, GAT) simpler and effective | | Mamba / SSMs | Short context, strong-pretrained ecosystem matters | Transformer + GQA + FlashAttention (still dominant) | | MoE Transformers | Below ~3-5B total params, on-device inference | Dense Transformer (active params == total params) | | Native multimodal models | Below frontier scale, single-modality output | LLaVA-style bolt-on (SigLIP + MLP + LLM) | | Diffusion Transformers (DiT) | Tiny generative tasks, very small datasets | U-Net latent diffusion (cheaper to train at small scale) | | LLMs (GPT-style) for narrow classification | Tabular / closed-set tasks with clean labels | Boosted trees (XGBoost/LightGBM) or small fine-tuned encoder |

Counter-narrative: "New architecture ≠ better for your use case. Match architecture to constraints."

---

Decision Tree

Start here: What's your primary goal?

┌─ SELECT architecture for task
│  ├─ Data modality?
│  │  ├─ Images → [cnn-families-and-selection.md](cnn-families-and-selection.md)
│  │  ├─ Sequences → [sequence-models-comparison.md](sequence-models-comparison.md)
│  │  ├─ Graphs → [graph-neural-networks-basics.md](graph-neural-networks-basics.md)
│  │  ├─ Generation → [generative-model-families.md](generative-model-families.md)
│  │  ├─ Multimodal (text + image / audio / video) → [multimodal-architectures.md](multimodal-architectures.md)
│  │  └─ Unknown → [architecture-design-principles.md](architecture-design-principles.md)
│  └─ Special requirements?
│     ├─ Deep network (>20 layers) unstable → [normalization-techniques.md](normalization-techniques.md) (CRITICAL)
│     ├─ Need attention mechanism → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│     └─ None → Proceed with modality-based route
│
├─ UNDERSTAND specific architecture
│  ├─ Transformers → [transformer-architecture-deepdive.md](transformer-architecture-deepdive.md)
│  ├─ Attention → [attention-mechanisms-catalog.md](attention-mechanisms-catalog.md)
│  ├─ Normalization → [normalization-techniques.md](normalization-techniques.md)
│  └─ General principles → [architecture-design-principles.md](architecture-design-principles.md)
│
├─ DESIGN custom architecture
│  └─ [architecture-design-principles.md](architecture-design-principles.md) (start here always)
│
└─ COMPARE architectures
   ├─ CNNs (ResNet vs EfficientNet) → [cnn-families-and-selection.md](cnn-families-and-selection.md)
   ├─ Sequence models (RNN vs Transformer) → [sequence-models-comparison.md](sequence-models-comparison.md)
   ├─ Generative (GAN vs Diffusion) → [generative-model-families.md](generative-model-families.md)
   └─ General comparison → [architecture-design-principles.md](architecture-design-principles.md)

---

Workflow

1. Clarify → Data modality, task type, dataset size, compute/latency constraints
2. Route by modality → Images/Sequences/Graphs/Generation → appropriate skill
3. Check critical issues → Deep network unstable? Fix normalization FIRST
4. Apply architecture skill → Follow routed skill, consider trade-offs
5. Cross-pack → training-optimization (training) or ml-production (deployment)

---

Rationalization Table

| Rationalization | Reality | Counter | |-----------------|---------|---------| | "Transformers are SOTA, recommend them" | SOTA on benchmark ≠ best for user's constraints | "Ask about dataset size and compute first" | | "User said RNN vs LSTM, answer that" | Question premise might be outdated | "Challenge: Have you considered Transformers or TCN?" | | "Just recommend latest architecture" | Latest ≠ appropriate | "Match architecture to requirements, not trends" | | "Architecture doesn't matter, training matters" | Wrong architecture can't be fixed by training | "Architecture is foundation - get it right first" | | "They seem rushed, skip clarification" | Wrong route wastes more time than clarification | "30 seconds to clarify saves hours of wasted effort" | | "Generic architecture advice is safe" | Generic = useless for specific domains | "Route to domain-specific skill for actionable guidance" |

---

Integration with Other Packs

After Architecture Selection

Once architecture is chosen, route to:

Training the architecture: → yzmir/training-optimization/using-training-optimization

• Optimizer selection
• Learning rate schedules
• Debugging training issues

Implementing in PyTorch: → yzmir/pytorch-engineering/using-pytorch-engineering

• Module design patterns
• Performance optimization
• Custom components

Deploying to production: → yzmir/ml-production/using-ml-production

• Model serving
• Quantization
• Inference optimization

Dynamic/growing architectures: → yzmir/dynamic-architectures/using-dynamic-architectures

• Networks that grow, prune, or adapt during training
• Continual learning and catastrophic forgetting prevention
• Module lifecycle management and progressive training

Before Architecture Selection

If problem involves:

Reinforcement learning: → yzmir/deep-rl/using-deep-rl FIRST

• RL algorithms dictate architecture requirements
• Value networks vs policy networks have different needs

Large language models: → yzmir/llm-specialist/using-llm-specialist FIRST

• LLM architectures are specialized transformers
• Different considerations than general sequence models

Architecture is downstream of algorithm choice in RL and LLMs.

---

Summary

Use this meta-skill to:

• ✅ Route architecture queries to appropriate specialized skill
• ✅ Identify data modality and problem type
• ✅ Clarify constraints before recommending
• ✅ Resist recency bias (latest ≠ best)
• ✅ Recognize when architecture is the problem (vs training/implementation)

Neural Architecture Specialist Skills

After routing, load the appropriate specialist skill for detailed guidance:

1. architecture-design-principles.md - Custom design, architectural best practices, skip connections, network depth fundamentals
2. attention-mechanisms-catalog.md - Self-attention, cross-attention, multi-head attention, attention in CNNs, attention variants comparison
3. cnn-families-and-selection.md - ResNet, EfficientNet, MobileNet, YOLO, computer vision architecture selection
4. generative-model-families.md - GANs, VAEs, Diffusion models, image generation, style transfer, generative modeling trade-offs
5. graph-neural-networks-basics.md - GCN, GAT, node classification, link prediction, graph embeddings, molecular structures
6. normalization-techniques.md - BatchNorm, LayerNorm, GroupNorm, training stability for deep networks (>20 layers)
7. multimodal-architectures.md - CLIP/SigLIP, LLaVA, BLIP-2 / Q-Former, Flamingo / Idefics, native multimodal (Chameleon / Gemini-style), audio + video extensions
8. sequence-models-comparison.md - RNN, LSTM, Transformer, TCN, Mamba/SSM, RWKV, Jamba; time series, NLP, sequential data
9. transformer-architecture-deepdive.md - Transformer internals, ViT (DINOv2/MAE/SigLIP-era), MoE (Mixtral/DeepSeek-MoE/OLMoE), positional encoding (RoPE/YaRN/NoPE)

Critical principle: Architecture comes BEFORE training. Get this right first.