harsh040506/architecture-search

Architecture Search — NAS and Pareto-Optimal Design

Provides systematic methodology for neural architecture search, candidate evaluation using proxy metrics, and Pareto-front analysis to balance accuracy against deployment constraints.

Architecture Search Paradigms

Manual Design (Expert-Driven)

Use domain expertise to select from known architecture families:

• Tabular: gradient boosting → MLP → attention-based (TabNet)
• Vision: ResNet → EfficientNet → ViT (scale-dependent)
• Text: LSTM → Transformer → sparse attention (long context)

When to use: Dataset is small, domain is well-understood, latency constraint is tight.

Zero-Cost NAS (Proxy-Based)

Evaluate architectures without training using proxy signals computed in seconds:

• SNIP: gradient-based pruning score at initialization — correlates with trained accuracy
• GradNorm: magnitude of gradients at init — high GradNorm → good information flow
• SynFlow: path norm proxy — avoids "layer-collapse" in pruning-based search

When to use: Tight time budget (< 1 hour), many candidates to screen.

Differentiable NAS (DARTS-style)

Relax the discrete architecture search to a continuous optimization:

• Mixed operations at each edge with learnable architecture weights α
• Jointly optimize model weights W and architecture weights α
• After training, discretize by selecting argmax over α at each edge

When to use: Custom search space, sufficient compute (DARTS requires ~4 GPU days).

Evolutionary / RL-Based NAS

• Encode architectures as genomes; evolve a population using mutation and crossover
• Or train a controller RNN to generate architecture strings (Zoph et al.)

When to use: Largest compute budget, maximum potential for novel architectures.

Pareto-Front Analysis

For each candidate architecture, evaluate on two axes:

1. Accuracy proxy (e.g., zero-cost score or validation accuracy on small proxy dataset)
2. Efficiency proxy (e.g., latency on target hardware, FLOP count, parameter count)

A candidate is Pareto-optimal if no other candidate is strictly better on both axes simultaneously.

Selection rules:

• If latency constraint is hard: filter to candidates meeting constraint, then pick highest accuracy
• If no hard constraints: pick the "knee" of the Pareto curve (max marginal accuracy per FLOP)

See references/nas-algorithms.md for DARTS, ENAS, and evolutionary search implementation details. See references/search-spaces.md for domain-specific search space definitions and operator libraries.

FLOP and Parameter Estimation

Quick estimation formulas:

• Linear/FC layer: FLOPs = 2 · in_features · out_features (multiply-add)
• Conv2d: FLOPs = 2 · C_in · C_out · K² · H_out · W_out
• Multi-Head Attention: FLOPs ≈ 4 · seq_len · d_model² + 2 · seq_len² · d_model

Parameters:

• Linear(in, out): in × out + out (weights + bias)
• Conv2d(C_in, C_out, K): C_in × C_out × K² + C_out
• MHA(d_model, n_heads): 4 × d_model²

Use these to quickly screen candidate architectures before running any training.