harsh040506/hyperparameter-optimization

Hyperparameter Optimization — Bayesian Search and Efficient Tuning

Provides methodology for finding optimal hyperparameter configurations λ* in high-dimensional spaces using Gaussian Process surrogate modeling, acquisition function optimization, and compute-efficient pruning strategies.

Why Bayesian Optimization Over Grid/Random Search

• Grid search: exponential cost in number of hyperparameters (O(k^d) for k points per dimension)
• Random search: sample-efficient over grid but ignores past observations
• Bayesian optimization: builds a probabilistic surrogate model of the objective — uses past evaluations to guide next trial intelligently

For d ≥ 3 hyperparameters and a budget of < 200 trials, Bayesian optimization typically requires 5–10× fewer trials than random search to reach the same optimum.

Gaussian Process Surrogate

The surrogate model f̂(λ) ~ GP(μ(λ), k(λ, λ')) encodes:

• Prior belief about the objective surface (encoded in the kernel k)
• Posterior updates as trial results arrive (Bayesian updating)

Kernel selection:

• Matérn 5/2: smooth but not infinitely differentiable — best for most HPO tasks
• RBF / Squared Exponential: infinitely smooth — can oversmooth rough objective surfaces
• Warped GP: apply input transformation for log-uniform parameters (learning rate, weight decay)

After n trials, the GP posterior provides: expected value μ_n(λ) and uncertainty σ_n(λ) at any new point λ.

Acquisition Functions

See references/acquisition-functions.md for mathematical derivations.

Summary:

• EI (Expected Improvement): E[max(0, f(λ) − f(λ*))]. Exploration-exploitation balance. Default choice.
• UCB (Upper Confidence Bound): μ(λ) + κ·σ(λ). κ controls exploration. Use when variance is high early.
• PI (Probability of Improvement): P(f(λ) > f(λ*)). Conservative — only cares about beating best.
• Thompson Sampling: sample a function from GP posterior; pick its argmax. Naturally diversifies search.

Hyperband Early Stopping

Hyperband extends SHA (Successive Halving Algorithm) to handle unknown budget:

1. Divide budget B into brackets with different η (halving rate).
2. In each bracket, sample n configurations and run to r_min resources.
3. Keep top 1/η fraction; run to next rung (r_min × η).
4. Repeat until budget exhausted.

ASHA (Asynchronous SHA): runs brackets in parallel without synchronization barriers — better GPU utilization.

Hyperband reduces wasted compute by 3–10× vs. running all trials to completion.

Hyperparameter Importance

After HPO, estimate which hyperparameters most influence the objective using fANOVA:

• Fit a random forest on (trial hyperparameters → trial L_val).
• Compute variance explained by each single hyperparameter and interaction pairs.
• High importance (> 10%): tune carefully in follow-up runs.
• Low importance (< 5%): can be fixed to a reasonable default.

See references/bayesian-optimization.md for full GP formulation, implementation details, and Optuna/Ray Tune integration patterns.

Search Space Design Rules

1. Use log-uniform for scale-invariant parameters (learning rate, weight decay, regularization).
2. Use categorical for optimizer type — treat as discrete.
3. Constrain batch size to powers of 2 for GPU memory alignment.
4. Always warm up with 5–10 random trials before fitting the GP.
5. Expand if best config is at a boundary — the true optimum may be outside the current range.