harsh040506/ml-theory

ML Theory — Mathematical Foundations

Provides the rigorous mathematical theory underlying all machine learning components in this plugin. Covers loss function derivation, optimization dynamics, and statistical learning guarantees that inform architectural and training decisions.

Core Domains

Loss Function Taxonomy

Loss functions define what a model is optimizing. The choice of loss encodes all assumptions about the output distribution:

• Regression: MSE (assumes Gaussian noise), MAE (assumes Laplacian noise), Huber (robust to outliers)
• Classification: Cross-Entropy (maximizes log-likelihood of correct class), Focal Loss (down-weights easy examples)
• Generation: ELBO (variational lower bound on log p(x)), Score Matching (denoising objective)
• Ranking: Pairwise hinge loss, ListNet, LambdaRank

See references/loss-functions.md for complete derivations and PyTorch implementations.

Optimization Theory

All training is an instance of empirical risk minimization: find θ* = argmin_θ (1/n) Σ L(f_θ(xᵢ), yᵢ).

Key optimization concepts:

• Gradient Descent: θ ← θ − η·∇_θ L
• SGD with Momentum: combines gradient history to dampen oscillations
• Adam: adaptive per-parameter learning rates using first and second moment estimates
• Learning Rate Schedules: warmup + cosine annealing prevents loss spikes and aids convergence

See references/optimization-theory.md for convergence proofs, saddle point analysis, and loss landscape geometry.

Statistical Learning Theory

Theory that bounds generalization error — the gap between training loss and test loss:

• Bias-Variance Decomposition: E[MSE] = Bias² + Variance + Irreducible Noise
• VC Dimension: upper bounds on hypothesis class complexity
• PAC Learning: conditions under which learnability is guaranteed
• Double Descent: phenomenon where test error can decrease again in the overparameterized regime

See references/statistical-learning.md for formal bounds, regularization theory, and empirical verification methods.

When to Apply Theory

Apply ML theory concepts when:

1. Choosing a loss function: match the loss to the output distribution assumption
2. Diagnosing training failure: use theory to distinguish underfitting vs. overfitting vs. optimization failure
3. Justifying model complexity: use VC dimension / scaling laws to reason about needed parameter count
4. Setting regularization strength: use bias-variance analysis to choose λ in L1/L2 regularization

Notation Reference

| Symbol | Meaning | |---|---| | θ | Model parameters | | η | Learning rate | | L(·) | Loss function | | f_θ(x) | Model prediction | | μ, σ | Mean, standard deviation | | ∇_θ | Gradient with respect to θ | | ᾱ_t | Cumulative noise schedule product (diffusion) | | γ | Discount factor (RL) |