lyndonkl/ml-primitive-decoder

ML Primitive Decoder

Table of Contents

1. Workflow
2. The Primitive Catalog
3. Decoding Patterns
4. Ablation Thought Experiment
5. Common Patterns
6. Guardrails
7. Quick Reference

Most ML constructs that look opaque are actually a short stack of well-understood linear algebra primitives composed in a specific way. Attention is dot product + softmax + weighted sum. Layer norm is recenter + rescale + learnable affine. Diffusion is noise injection + denoising regression repeated. This skill names the primitives, then shows why their composition produces the behavior the construct is famous for.

The signature deliverable is: construct = primitive₁ + primitive₂ + primitive₃, and that's why it does X.

Quick example (Attention):

Construct: Self-attention layer, output = softmax(QKᵀ/√d) · V.

Primitives:

1. Linear projections (Q, K, V each = a different learnable linear layer applied to the same input).
2. Dot product (QKᵀ measures pairwise similarity between query and key vectors).
3. Scaled softmax (divides scores by √d to keep variance bounded, then row-normalizes into mixing weights).
4. Weighted sum (multiplying by V is a per-row weighted average over value vectors).

Why it works: Each token learns to advertise (K), ask (Q), and deliver (V). The dot product matches asks to advertisements; softmax turns raw matches into selection weights; the weighted sum is the selected content. The whole layer is content-addressable memory implemented in pure linear algebra.

Ablation:

• Remove softmax → linear combination, no selectivity (just averaging everything).
• Remove √d → softmax saturates as d grows, gradients vanish.
• Tie Q = K → tokens can only ask questions about themselves; no cross-token information transfer.

Workflow

Copy this checklist and track your progress:

Decoding Progress:
- [ ] Step 1: Identify the construct and write its formula
- [ ] Step 2: List the primitives it's composed of
- [ ] Step 3: Explain what each primitive contributes
- [ ] Step 4: Show how composition produces the famous behavior
- [ ] Step 5: Run the ablation thought experiment (what breaks if you remove each piece?)
- [ ] Step 6: Verify with a tiny worked example or invitation

Step 1: Identify the construct and write its formula

Pin down what you're decoding. Get the formula in front of you (and the user). If the user asks "why does attention work" without specifying, it's almost always self-attention — but check. Multi-head attention, cross-attention, and flash attention are different decompositions.

For a catalog of common ML constructs and their formulas, see resources/constructs.md.

Step 2: List the primitives it's composed of

The primitives are a small fixed set (see The Primitive Catalog below). Decompose the formula into a list of these. Most constructs are 2-5 primitives.

The discipline: name only the load-bearing primitives. Trivial reshapes, broadcasts, and identity passes don't count. If a primitive doesn't change the behavior of the construct, skip it.

Step 3: Explain what each primitive contributes

For each primitive, write one sentence saying what it does in this construct specifically — not in general. "Dot product measures alignment" is too generic. "Dot product, applied to (Q, K) pairs, scores how well each token's question matches each other token's advertisement" is specific.

Step 4: Show how composition produces the famous behavior

This is the payoff. The construct is famous for some behavior — attention does content-addressable retrieval; layer norm stabilizes training; softmax produces a sharp-or-smooth distribution. Show how the combination of primitives produces that behavior. The composition is where the magic isn't.

A good Step 4 reads like: "Primitive A produces sub-effect 1. Primitive B turns sub-effect 1 into sub-effect 2. Primitive C turns sub-effect 2 into the famous behavior. Each step is mechanical; the famous behavior emerges from the chain."

Step 5: Run the ablation thought experiment

For each primitive, ask: "what would break if we removed this?" The answers are the highest-information part of the decoding — they tell the learner why each piece is there, which is much harder to forget than the formula.

See Ablation Thought Experiment below for the structure. For ablation tables per construct, see resources/ablations.md.

Step 6: Verify with a tiny worked example or invitation

Either compute one tiny instance of the construct end-to-end (a 3-token attention, a 2-feature layer norm) showing each primitive's intermediate output — or invite the user to do so. The worked example is what makes the decomposition feel real, not just declarative.

For tiny worked examples per construct, hand off to the worked-example-walkthrough skill.

The Primitive Catalog

Almost every ML construct is built from a small set of these:

| Primitive | What it does | Famous uses | |---|---|---| | Linear projection (matrix multiply) | Maps input vector to a new space; learnable | Q/K/V projections, FFN layers, embeddings | | Dot product | Measures alignment between two vectors | Attention scores, cosine similarity, score functions | | Outer product | Forms a rank-1 matrix from two vectors | Hebbian updates, low-rank adaptations (LoRA) | | Softmax | Normalizes scores into a probability distribution; sharpens at high contrast | Attention weights, classification heads, mixture-of-experts gating | | Element-wise nonlinearity (ReLU, GELU, sigmoid, tanh) | Introduces nonlinearity, gates information | Hidden layers, gating mechanisms | | Weighted sum | Combines vectors with given weights | Attention output, mixture models, EMA | | Centering / normalization | Subtract mean, divide by std | LayerNorm, BatchNorm, GroupNorm | | Affine transform (γx + β) | Learnable shift + scale | LayerNorm tail, BatchNorm tail | | Residual / skip connection (x + f(x)) | Adds an identity path around a block | Transformers, ResNets | | Dropout / noise injection | Randomly zeros or perturbs values | Regularization, diffusion forward process | | Convolution | Weight-shared local linear map | CNNs | | Pooling (max, mean, attention) | Aggregates many features into one | Read-out layers, set/graph reductions | | Embedding lookup | Indexes into a learnable table | Token embeddings, position embeddings | | Loss (cross-entropy, MSE, contrastive, KL) | Scalar measure of wrongness | Training objective | | Gradient descent step | Subtract scaled gradient from parameters | Every training loop |

Whenever you decode a construct, the primitives list should come from this catalog. If you find yourself naming something that isn't here, either it's a more complex construct (decompose further) or it's a new primitive worth adding.

Decoding Patterns

Pattern A: The pipeline construct

The construct is a sequence of primitives applied in order. Decode as: input → P₁ → intermediate → P₂ → ... → output. Examples: attention block, transformer FFN, layer norm.

Pattern B: The decorated construct

The construct is a base primitive wrapped with normalization, residuals, or activations. Decode as: base + decorator₁ + decorator₂. Examples: a transformer block (attention + residual + LN + FFN + residual + LN), ResNet block.

Pattern C: The objective construct

The construct is a loss function. Decode as: similarity measure + normalization + reduction. Examples: cross-entropy (log + sum), contrastive loss (similarity + softmax + cross-entropy).

Pattern D: The iterative construct

The construct is a single step repeated many times. Decode the step, then explain the dynamics of repetition. Examples: gradient descent (gradient + step), diffusion (noise + denoise step), RNN (state update step).

For one full decode per pattern, see resources/constructs.md.

Ablation Thought Experiment

For every decoded construct, run an ablation table. For each primitive, answer: "what would break if you removed it (or replaced it with the simplest possible thing)?"

This is the most pedagogically valuable part of the skill. It forces the user to see why each piece is there.

Format:

| Remove | What happens | What insight it reveals | |---|---|---| | Primitive 1 | Concrete failure mode | What primitive 1 is contributing | | Primitive 2 | Concrete failure mode | What primitive 2 is contributing | | ... | ... | ... |

Example (Attention):

| Remove | What happens | What insight it reveals | |---|---|---| | Softmax | Output becomes a linear combination — every token mixed equally with every other; no selectivity | Softmax is what makes attention attention (not averaging) | | √d scaling | At high d, dot products become large, softmax saturates, gradients vanish | √d controls the variance of scores so softmax stays in its useful regime | | Separate Q vs K | Tokens can only attend based on self-similarity, not cross-similarity | Splitting Q and K is what allows asymmetric matching | | V (use K instead) | Model can only retrieve what tokens advertise about themselves; no separate "content" channel | V is the actual payload — separate from the matching key | | Multi-head | Single semantic relation per layer; loses ability to attend along multiple axes simultaneously | Heads are parallel attention computations on different subspaces |

For ablation tables for many constructs, see resources/ablations.md.

Common Patterns

When the user asks "why does X work?"

Always run the full workflow. The "why" is in Step 4 (composition) and Step 5 (ablation), not in Step 2 (primitive list).

When the user asks "what is X doing?"

Steps 1-3 may be enough. Skip ablation if the user just wants a quick read.

When the construct is opaque even to you

Don't fake it. Decode what you understand; flag what you don't. "I can decode the attention part but I'm not sure why this paper added the gating; let me look it up" is a much better response than confident-sounding nonsense.

When the construct is novel

Decompose into the catalog of primitives anyway. Truly novel ML constructs are rare; most are reshufflings of the catalog. If something genuinely doesn't fit, name it as a new primitive and explain why.

Guardrails

• Don't list primitives without saying what they do in this construct. Generic descriptions ("softmax produces a probability distribution") are worth less than construct-specific ones ("softmax converts the row of dot product scores into mixing weights for the value vectors").
• Don't skip the ablation. It's the part that builds durable understanding. The user who can answer "what would happen if you removed the softmax from attention?" understands attention; the user who can only recite the formula does not.
• Don't bury the punchline. Step 4 — "and that's why it does X" — should be loud and short. If the punchline is buried in paragraph 4, the reader missed it.
• Don't decode further than necessary. Linear projection is a primitive. You don't need to decode it into "matrix multiply, which is itself a sum of weighted columns…" unless the user asks. Decompose to the level the user needs and stop.
• Don't conflate the construct with its claimed purpose. Attention isn't "how transformers attend to relevant context" — that's its emergent behavior. The construct is softmax(QKᵀ/√d)V; the behavior emerges from training. Be precise about what the math forces vs. what learning produces.

Quick Reference

| Construct | Primitives (in order) | Famous behavior emerges because… | |---|---|---| | Self-attention | Linear proj × 3 + dot product + scaled softmax + weighted sum | Each token learns to ask, advertise, deliver; the math implements soft database lookup | | LayerNorm | Center + rescale + learnable affine | Standardizes per-example activations; γ, β preserve expressiveness | | Softmax | Exp + normalize | Positivity + sum-to-1 + amplification of contrast | | Cross-entropy | Log + weighted sum | Punishes low predicted prob on the true class | | Conv layer | Weight-shared linear projection | Translation equivariance + parameter efficiency from sharing | | ResNet block | f(x) + identity (residual) + nonlinearity | Identity path means gradients flow + small learned perturbations to the input | | Dropout | Random binary mask + rescale | Forces redundancy; prevents co-adaptation | | Diffusion (one step) | Noise injection (forward) + linear projection + nonlinearity (denoise) | Slow noise schedule lets simple regression learn complex distributions | | SGD step | Gradient + scaled subtraction | Local descent on the loss surface | | Adam step | Gradient + moving averages + per-coord rescale + scaled subtraction | Per-parameter adaptive learning rate from gradient statistics | | Embedding | Index + lookup table | Discrete tokens get learnable continuous coordinates | | Contrastive (InfoNCE) | Dot product + scaled softmax + cross-entropy | Pulls aligned pairs together, pushes others apart, all in one loss |

For the full decode of each construct, see resources/constructs.md. For ablation tables for each construct, see resources/ablations.md.