wshobson/lora-qlora-recipes

LoRA & QLoRA Recipes

This skill assumes the routing decision already happened — finetuning-method-selection should have already pointed here because the data shape is demonstrations (SFT), not preference pairs or a verifiable reward signal. What follows is the current best-practice recipe for configuring the adapter itself: which modules to target, how to size rank and alpha, what learning rate to use, and when QLoRA buys real headroom versus when it just adds risk. Dataset preparation and quality checks are a separate concern — see dataset-curation.

Input: a routing decision (SFT via LoRA/ QLoRA) plus a target size class. Output format: a validated adapter config — the kwarg values below, not free-form advice — that llm-finetuning-training-engineer consumes directly when it generates a runnable script.

The Reference Recipe

The reference recipe is "LoRA Without Regret" (Thinking Machines/Schulman, 2025-09), now the settled convention for LoRA/QLoRA SFT.

Target Modules

Target all-linear modules, not just attention:

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",   # attention
    "gate_proj", "up_proj", "down_proj",      # MLP — matters most
]

The MLP layers (gate_proj, up_proj, down_proj) matter most — attention-only targeting was the older, weaker convention. Dropping modules to save memory is a Failure Mode below, not a valid optimization.

Alpha and Learning Rate

• lora_alpha = 2 * r is the settled convention (NeurIPS 2025 "intruder dimensions" result). Don't hand-tune alpha independently of rank — derive it from rank every time.
• LoRA learning rate ≈ 10x the equivalent full-fine-tune LR. For QLoRA specifically, 2e-4 is the standard starting point. Full hyperparameter tables and worked examples: references/hyperparameters.md.

Rank by Task

Rank is task-shaped, not a single global default:

| Task | Rank | |---|---| | RL (GRPO/RLVR adapters) | 1–32 | | General default | 16–32 | | SFT at scale | up to ~256 |

Higher rank isn't automatically better — it raises capacity to memorize as fast as it raises capacity to generalize. Start at the row matching the task, and only move up a row if the lower rank measurably underfits on held-out eval, not as a default hedge.

Effective Batch Size

Keep effective batch size under 32. This recipe was validated at that scale — pushing effective batch higher is an untested extrapolation, not a free throughput win.

Unsloth Defaults

Unsloth is the reference implementation this plugin assumes as the default fast path — except for messages-shaped conversational SFT with assistant_only_loss=True, where Unsloth 2026.7.x's compiled trainer has no messages-shaped path at all and the plain-TRL escape hatch (references/unsloth-trl-mapping.md) is the default for that combination, not a rare-regression fallback. Its out-of-the-box defaults, and why each one is set that way:

• lora_dropout=0 — the optimized kernel path assumes zero dropout; setting a nonzero value forfeits the fused-kernel speedup.
• bias="none" — bias terms add adapter parameters for negligible quality gain at this rank range.
• use_gradient_checkpointing="unsloth" — Unsloth's checkpointing variant, not vanilla HF checkpointing; saves roughly 30% VRAM over no checkpointing.
• optim="adamw_8bit" — 8-bit AdamW cuts optimizer-state memory with negligible quality impact at LoRA/QLoRA adapter scale.
• random_state fixed — pins LoRA initialization for reproducibility across runs; treat it like any other seed, not a tunable.

These show up together on the get_peft_model call:

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=target_modules,
    lora_alpha=64,               # 2 * r
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

Exact kwarg names and their plain-TRL/PEFT equivalents, plus a full worked config including SFTConfig: references/unsloth-trl-mapping.md and references/hyperparameters.md.

LoRA vs QLoRA vs Full FT

| Situation | Default choice | |---|---| | Adapting behavior on demonstrations | LoRA | | Base model doesn't fit in bf16 at target rank | QLoRA | | Injecting dense new domain knowledge | Full FT (see finetuning-method-selection) | | Unsure which one | LoRA — upgrade to QLoRA only if memory forces it |

• QLoRA = NF4-quantized frozen base weights + BF16 adapters. This is what makes a 65B-class model trainable on 48GB — the quantized base is the memory win, not the adapter itself.
• Full fine-tuning is not a default. Reserve it for dense knowledge injection where the goal is changing what the model knows at the weight level, not adapting a behavior. For everything else in this skill's scope, LoRA or QLoRA is the starting assumption.
• On DGX Spark, QLoRA can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller — bitsandbytes dequantization buffers are transient CUDA-side allocations that spike during load. A QLoRA OOM is not proof the model doesn't fit; the dgx-spark-ops plugin's spark-memory-thermal-ops skill covers the full OOM remediation ladder (bf16 LoRA is the next thing to try, not a further QLoRA shrink).

Failure Modes

• fp16 divergence on non-BF16 GPUs. Training in fp16 on hardware that doesn't have solid BF16 support is a known source of loss spikes and silent divergence. Force bf16=True wherever the hardware supports it; don't fall back to fp16 as if it were equivalent. Check hardware support before picking a dtype:

  python -c "import torch; print(torch.cuda.is_bf16_supported())"
  

• Rank too high on a small dataset overfits. A rank picked for "SFT at scale" (up to ~256) on a dataset that doesn't have scale behind it memorizes rather than generalizes. Match rank to the Rank by Task table above, not to the largest number available.

• Removing target modules to save memory costs quality for negligible savings. The adapter parameters on gate_proj/up_proj/down_proj are a small fraction of total model size — cutting them barely moves memory but measurably hurts quality. If memory is tight, move to QLoRA or reduce rank/batch/pack length before trimming target modules.

All three failure modes share a pattern: they look like a training-loop bug (loss spikes, plateaus, memorization) but are actually a config choice that contradicts the reference recipe above. Check configuration against this skill before debugging the training loop itself.

References

• references/hyperparameters.md — full rank/ alpha/LR tables by task type, rsLoRA notes, batch/packing interactions, and a complete worked Unsloth config block.
• references/unsloth-trl-mapping.md — every Unsloth kwarg mapped to its TRL/PEFT equivalent, current TRL API notes, and the escape-hatch rule for when to drop back to plain TRL.

Related skills: finetuning-method-selection routes here; dataset-curation covers the data side this skill doesn't; llm-finetuning-training-engineer is the downstream consumer of the config this skill produces.