Parameter-Efficient Fine-Tuning (PEFT) — All Variants & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. What Is PEFT?
2. Foundations: Why It Works
3. LoRA: Low-Rank Adaptation
4. LoRA Variants
5. QLoRA and Quantization-Aware PEFT
6. Adapters (Houlsby, Pfeiffer)
7. Soft Prompts and Prefix Tuning
8. Selective / Sparse PEFT
9. Reparameterization Beyond LoRA
10. PEFT for Diffusion Models
11. PEFT for Vision Models and VLMs
12. PEFT for VLA / Robotics
13. Memory Engineering for PEFT
14. PEFT Frameworks and Tools
15. PEFT Recipes by Use Case
16. Multi-Adapter Serving and Composition
17. Theoretical Foundations
18. Stability, Hyperparameters, Pitfalls
19. Production Deployment
20. PEFT for RL / Alignment
21. Frontier 2025–2026
22. Production Stack 2026 Appendix A: Twenty-Five Things to Know Appendix B: Decision Tree — "Which PEFT?" Appendix C: Year-by-Year PEFT Milestones

1. What Is PEFT?

1.1 The premise

Adapting a large pretrained model to a downstream task by training only a tiny fraction of parameters. Saves memory, compute, and storage — and avoids catastrophic forgetting.

1.2 Why PEFT exploded

• Modern foundation models are huge (≥7B; up to 100B+).
• Full fine-tuning requires hundreds of GBs of optimizer state.
• Many downstream tasks need only minor capability adjustments.
• Multi-tenant scenarios need many adaptations of one base model.

1.3 The big four properties

1. Memory-efficient: optimizer states only for a few % of params.
2. Fast: shorter training time.
3. Modular / stackable: swap adapters; combine multiple.
4. Quality-preserving: matches full fine-tuning for many tasks.

1.4 The four families

1. Reparameterization: low-rank deltas (LoRA, DoRA, OFT, BOFT, VeRA).
2. Additive: insert new modules between frozen layers (Adapters, Houlsby, Pfeiffer).
3. Soft prompts: train embedding tokens / prefixes (prompt tuning, prefix tuning, P-tuning).
4. Sparse / selective: train a tiny subset of original parameters (BitFit, IA3, sparse fine-tuning).

1.5 The PEFT-vs-full-FT scaling story

Key

At ≥7B parameters, LoRA-class methods match full fine-tuning quality on most tasks at 1/100 to 1/1000 trainable parameters. Modern industry default: LoRA on attention + MLP linears, rank 8–256.

1.6 When NOT PEFT?

• Massive distribution shift (new language, modality).
• Very long-horizon training (PEFT can hit a ceiling).
• Need to update tokenizer / vocab.
• Need to change architecture.

For these: full fine-tune or continued pretraining.

2. Foundations: Why It Works

2.1 The intrinsic-dimension hypothesis

Aghajanyan et al. (2020): pretrained models have an "intrinsic rank" on task gradients. Fine-tuning fits within a low-dimensional subspace of all possible weight changes. PEFT methods exploit this directly.

2.2 Why low-rank works for adaptation

\[W' = W + \Delta W, \quad \Delta W \approx BA, \ A \in \mathbb{R}^{r \times d}, \ B \in \mathbb{R}^{d \times r}, \ r \ll d.\]

The task-relevant change is approximately rank-r. LoRA's bet: most fine-tunes don't need full-rank updates.

2.3 The forgetting argument

Full fine-tuning can destroy pretrained capabilities. Freezing the base and learning small deltas preserves generalization.

2.4 Compositionality argument

Since adapters are small, can store many of them; load on demand for different tasks. Multi-tenant serving from one base.

2.5 The matrix-rank trade-off

Higher rank = more capacity = closer to full FT. Lower rank = less memory, faster, more regularization. Most tasks: rank 8–64; some hard tasks: rank 128–512.

3. LoRA: Low-Rank Adaptation

3.1 The original (Hu et al. 2021)

For each linear layer \(W \in \mathbb{R}^{d_{out} \times d_{in}}\):

Key

\[W' = W + \alpha \cdot BA, \quad A \in \mathbb{R}^{r \times d_{in}}, \ B \in \mathbb{R}^{d_{out} \times r}.\]

A initialized Gaussian; B initialized to zero. Initial output: identity to base model.

3.2 Hyperparameters

• r (rank): 8 / 16 / 32 / 64 / 128 / 256.
• α (scaling): typically \(\alpha = r\) or \(\alpha = 2r\). Effective scale \(\alpha/r\).
• Dropout: 0.05–0.1 on A output.
• Target modules: which layers to LoRA-ize.

3.3 Where to apply LoRA

• Attention Q, K, V, O (most common).
• MLP up / down projections.
• Both: "all-linear" default in modern recipes.
• Embedding / LM head: rare, sometimes useful.

3.4 Why initialize B = 0?

At step 0: \(\Delta W = BA = 0\), so output identical to base. Avoids the early-training shock.

3.5 Memory savings

For 7B model with ~4096 hidden dim, r = 16:

• Full FT: ~14P bytes optimizer state (~100 GB for 7B).
• LoRA: trainable params ~4M; optimizer state proportional. Total: ~100 MB.
• Memory ~100× less.

3.6 Inference: merge or keep separate?

Merged: \(W' = W + BA\) computed once; no inference cost.

Unmerged: keep base + LoRA separate; useful for swapping LoRAs at inference.

3.7 LoRA at inference time

• Merge: standard for single-adapter deployment.
• Multi-LoRA serving: vLLM, S-LoRA support runtime LoRA loading.
• Stack: \(W + B_1 A_1 + B_2 A_2\) for combined adapters.

3.8 LoRA recipes by domain

• LLM SFT: rank 8–64; \(\alpha = 16\)–64; LR \(1\times10^{-4}\)–\(5\times10^{-4}\).
• LLM DPO: rank 8–64; LR \(5\times10^{-7}\)–\(5\times10^{-6}\).
• Diffusion (SDXL / FLUX): rank 8–128; LR \(1\times10^{-4}\).
• VLM: rank 16–128; LR varies.

4. LoRA Variants

4.1 LoRA+

Use different learning rates for A and B. Theoretical analysis: optimal \(\eta_B/\eta_A \sim 16\times\). Empirically improves convergence.

4.2 rsLoRA (rank-stabilized)

Scale by \(\alpha/\sqrt{r}\) instead of \(\alpha/r\). Stabilizes training as rank grows.

4.3 LoRA-FA (Frozen-A)

Freeze A at initialization; only train B. Half the trainable params; surprisingly effective.

4.4 DoRA (Weight-Decomposed Low-Rank Adaptation)

Decompose W into magnitude + direction; LoRA only the direction:

\[W' = m \cdot \frac{W + BA}{\|W + BA\|_c},\]

where m is a learnable per-column magnitude vector. Closer to full FT than LoRA in many settings.

4.5 LoRA-XS

Tiny adaptations: r as low as 1–2 with cleverly chosen scaling.

4.6 ReLoRA

Iteratively merge LoRA into base, then re-init new LoRA. Allows higher effective rank over training without ever holding full-rank optimizer state.

4.7 AdaLoRA

Adaptive rank: dynamically allocate rank budget per layer based on importance. Variable rank optimization.

4.8 LoRA-MOE

Mixture-of-LoRA: route to one of several LoRA experts per token. Increases capacity at fixed inference cost.

4.9 NoLA (None or LoRA Allocator)

Decide per-layer whether to apply LoRA. Tiny memory overhead; surprisingly competitive.

4.10 Comparison table

★ 2026 SOTA update — Gradient-optimal LoRA init - LoRA-GA: initialize A,B from the SVD of the first full-batch gradient so LoRA's step-1 gradient matches full FT; near-full-FT convergence speed. - LoRA-Pro: rescale/couple A,B gradients each step so the low-rank update mimics the full-FT optimization trajectory ('are low-rank adapters properly optimized?').

★ 2026 SOTA update — Beyond low-rank: high-rank & MoE - HiRA: delta = \(W_{\text{frozen}} \odot BA\) (Hadamard), so a low-rank mask on the base yields a HIGH-rank update; more expressive than LoRA (ICLR 2025 oral). - RandLoRA: learn only diagonal scaling of many fixed random low-rank bases to reach FULL-rank updates at LoRA-level memory; closes LoRA plateau on vision-language (ICLR 2025). - HydraLoRA: asymmetric shared-A / multiple-B architecture with a trained MoE router; splits data into intrinsic components, no domain expertise needed (NeurIPS 2024 oral).

5. QLoRA and Quantization-Aware PEFT

5.1 QLoRA (Dettmers et al. 2023)

• Base model in 4-bit (NF4 format).
• LoRA adapters in BF16 for training.
• Paged optimizer states (CPU offload on overflow).
• Double quantization (quantize the per-block scales themselves).

Result: fine-tune 65B model on a single 48 GB GPU.

5.2 NF4 (NormalFloat 4)

4-bit format optimized for normally-distributed weights. Better than uniform INT4 for typical model weights.

5.3 Standard QLoRA recipe

• Load base in NF4 with double-quant.
• Add LoRA on attention + MLP linears (rank 8–256).
• Train LoRA only; base frozen.
• Merge or keep separate at inference.
• Optional: dequantize for inference / serving.

5.4 QA-LoRA (Quantization-Aware LoRA)

Train LoRA with quantization-aware-style loss; more robust at deployment in INT4.

5.5 LoftQ (LoRA-Fine-Tuning-aware Quantization)

Initialize LoRA to compensate for quantization error; better starting point than zero.

5.6 LQ-LoRA, AQLM-LoRA

Joint LoRA + low-bit weight quantization. Pushes memory further.

5.7 Q-Galore, GaLore + Q

Memory-efficient training of full-precision base via low-rank gradient projection; combined with quantization for further savings.

5.8 Unsloth optimizations

• Custom kernels for QLoRA training; 2–5× speedup.
• Bit-packed memory layouts.
• Triton-based dispatch.
• Standard for consumer-GPU QLoRA in 2025–26.

5.9 Memory budget for QLoRA fine-tune

• 7B model: ~8 GB VRAM (4-bit base + LoRA + activations).
• 13B model: ~14 GB VRAM.
• 70B model: ~48 GB VRAM (single GPU possible).
• 405B model: ~200 GB (multi-GPU FSDP).

★ 2026 SOTA update — Sub-2-bit quantized PEFT - IR-QLoRA: information-retention calibration + elastic connection; recovers accuracy for LoRA fine-tuning at 2-4 bit (CVPR 2024). - LowRA: pushes accurate LoRA fine-tuning BELOW 2 bits/param via learned high-granularity quantization mapping, beating QLoRA/LoftQ at ultra-low bit (2025).

6. Adapters (Houlsby, Pfeiffer)

6.1 Houlsby Adapter (2019)

Insert small bottleneck modules after attention and FFN:

\[\text{Adapter}(h) = h + \text{Up}(\text{NL}(\text{Down}(h))),\]

where Down: \(\mathbb{R}^d \to \mathbb{R}^{d_b}\), Up: \(\mathbb{R}^{d_b} \to \mathbb{R}^d\), \(d_b \ll d\).

6.2 Pfeiffer Adapter

Single adapter per block (after FFN only); empirically as good as Houlsby with half the params.

6.3 AdapterFusion

Train multiple task-specific adapters, then a fusion module that combines them per-input. Multi-task learning friendly.

6.4 Compacter

Hyper-network generates adapter weights from low-rank components. Even fewer params.

6.5 Adapter ranks

Bottleneck dim typically \(d_b = 8\)–64. Smaller than corresponding LoRA-rank for similar param count, due to two matrices.

6.6 Memory profile

Similar to LoRA. Slight difference: adapters add inference cost (extra matmul); LoRA can be merged at no inference cost.

6.7 Why LoRA mostly replaced adapters

• LoRA mergeable → no inference overhead.
• LoRA simpler architecturally.
• LoRA generalizes better empirically on most tasks.
• Adapter use: still common in BERT-class encoders, multilingual.

7. Soft Prompts and Prefix Tuning

7.1 Prompt tuning (Lester et al.)

Train a small set of learnable embedding tokens prepended to the input.

Frozen base model.

Hyperparams:

number of tokens (10–100). Param count: just tokens × d.

7.2 Prefix tuning (Li & Liang)

Train per-layer learnable key/value prefixes prepended to each attention layer. More expressive than prompt tuning; more parameters.

7.3 P-tuning v2

Combines prefix tuning + per-layer reparam via a small encoder. Strong on NLU.

7.4 Soft-prompt recipes

• Tokens: 10–200.
• Initialize with pretrained word embeddings (vocab tokens).
• LR slightly higher than other PEFT.
• Sensitive to seed.

7.5 Visual prompt tuning (VPT)

Same idea for ViTs: learnable prompt tokens prepended to patch tokens. "VPT-Shallow" (only first layer) and "VPT-Deep" (every layer).

7.6 Why soft prompts decline

Empirically:

• Task-specific; doesn't transfer across tasks well.
• Inferior to LoRA for most things.
• Sensitive to initialization.

Use cases now: very specific NLU / retrieval, where LoRA overkill.

8. Selective / Sparse PEFT

8.1 BitFit

Train only the biases of the model. Surprisingly competitive on many GLUE tasks. Tiny param count (< 0.1%).

8.2 IA3 (Liu et al. 2022)

Multiply intermediate activations by learnable per-feature scalars:

\[h' = h \odot \ell_h,\]

where \(\ell_h\) is a learnable vector per layer. Three vectors per Transformer block (key, value, FFN). Even fewer params than LoRA.

8.3 (IA)3 recipe

• Per-task vectors initialized to 1.
• LR moderate.
• Strong on T5-class generation; weaker on chat models.

8.4 LayerNorm fine-tuning

Train only LayerNorm scale + shift parameters. Tiny; surprisingly effective for some adaptation scenarios.

8.5 Sparse fine-tuning (Diff Pruning, FishMask)

Identify a sparse mask of important parameters; train only those. Tens of thousands of params suffice.

8.6 Diff Pruning (Guo et al.)

\[\theta' = \theta_0 + \text{mask} \odot \delta,\]

with sparse \(\delta\). Optimize sparsity + task jointly. Compresses task adaptation to < 0.5% of params.

8.7 When sparse PEFT wins

• Storage-tight (many small adapters).
• Quick task switches.
• Interpretability research.

9. Reparameterization Beyond LoRA

9.1 OFT (Orthogonal Fine-Tuning)

Apply learnable orthogonal rotation:

\[W' = RW, \quad R \in SO(d).\]

R parameterized via Cayley transform. Preserves spectrum; identity-preserving. Useful for diffusion (DreamBooth-style identity preservation).

9.2 BOFT (Block-Diagonal OFT)

Block-diagonal R for compute savings. Standard for diffusion personalization.

9.3 VeRA (Vector-based Random Matrix Adaptation)

Shared random projection matrices across layers; train only per-layer scaling vectors. ~10× fewer trainable params than LoRA at similar quality.

9.4 LoHa, LoKr (Hadamard / Kronecker)

• LoHa: \(\Delta W = (B_1 A_1) \odot (B_2 A_2)\) via Hadamard product.
• LoKr: \(\Delta W = B \otimes A\) via Kronecker product.

Both express more complex updates with fewer parameters.

9.5 HiWi (Hidden Weight)

Lightweight reparameterization with hidden weight sharing.

9.6 Tied / shared LoRA

Share A or B across layers / modules. Even fewer params; compatible with VeRA.

9.7 Decomposition spectrum

★ 2026 SOTA update — SVD / spectral subspace PEFT - PiSSA: init B,A from top singular vectors of W, freeze residual; faster convergence, beats LoRA/QLoRA (Mistral-7B GSM8K 72.9 vs 67.7). - MiLoRA: adapt the MINOR (small) singular components, freeze principal ones to preserve pretrained knowledge (NAACL 2025). - CorDA: context-oriented SVD using activation covariance from task or world-knowledge data; knowledge-preserving or task-oriented adapters (NeurIPS 2024). - SVFT: update = sparse learned combination of outer products of W's own singular vectors; ultra-low param (0.006-0.25%). - SORSA: trainable principal singular weights + frozen residual, with orthonormal regularization on singular vectors.

10. PEFT for Diffusion Models

10.1 Why diffusion needs PEFT

• SDXL, SD3, FLUX have billions of params.
• Per-style / per-character / per-concept customization is the use case.
• Multi-LoRA serving (Civitai marketplace).

10.2 Diffusion LoRA standard recipe

• Target: cross-attention layers (text → image conditioning).
• Sometimes: also self-attention.
• Rank: 8–128 typical.
• Training: 500–5000 steps on 5–100 images.
• LR: \(1\times10^{-4}\).

10.3 Locon, LyCORIS variants

• LoCon: extend LoRA to convolutional layers (for SD U-Net).
• LoHa, LoKr in LyCORIS: alternative decompositions.
• Diag-OFT, BOFT: identity-preserving for personalization.

10.4 DreamBooth + LoRA

1. Curate ~5–20 reference images of subject.
2. Use unique token ("[V] dog").
3. Train LoRA with prior-preservation loss.
4. Merge or distribute.

10.5 IP-Adapter is not PEFT

IP-Adapter adds new modules (parallel image cross-attention); usually trained with the base frozen. Functionally similar to PEFT; technically additive.

10.6 Multi-LoRA stacking for diffusion

• Stack \(W + \sum_i \alpha_i B_i A_i\) at inference.
• Combine style + character + composition LoRAs.
• Standard in ComfyUI workflows.

10.7 Diffusion-DPO with LoRA

• LoRA on cross-attention.
• Preference-trained on aesthetic pairs.
• Light alignment; cheap.

10.8 LoRA for video diffusion

• Target temporal attention modules.
• Smaller datasets (~10s of clips).
• Stack with image LoRAs.
• Used heavily for Wan 2.x, Hunyuan Video customization.

11. PEFT for Vision Models and VLMs

11.1 ViT PEFT

• LoRA on attention layers.
• VPT (Visual Prompt Tuning).
• Adapters (CLIP-Adapter, Tip-Adapter).
• BitFit on ViT.

11.2 CLIP fine-tuning

• LoRA on text + vision encoders.
• CoOp / CoCoOp: learnable text prompts.
• Tip-Adapter / CLIP-Adapter: insertion-based.

11.3 VLM PEFT (LLaVA family)

• Standard: LoRA on LLM backbone; freeze vision encoder + projector.
• Sometimes: also LoRA on projector.
• Rank 16–128.

11.4 Foundation model PEFT (Qwen-VL / InternVL)

Similar to LLaVA. Frozen ViT + LoRA on LLM.

11.5 SAM fine-tuning

SAM-Adapter, MedSAM-style: lightweight adapters for medical / domain-specific SAM. Cross-attention to spatial features.

11.6 Visual prompt design

• Per-task token count: 10–200.
• Initialize with text tokens or random.
• Standard for dense prediction (segmentation, detection) on top of frozen backbones.

12. PEFT for VLA / Robotics

12.1 OpenVLA recipe

• LoRA on Llama 2 backbone.
• Rank 8–256 depending on task complexity.
• Per-task / per-robot LoRA.
• Stack base + LoRA at inference.

12.2 Multi-LoRA for cross-embodiment

• Per-robot LoRA on shared base.
• Per-task LoRA on top.
• Combine at deploy.

12.3 π0 adaptation

Open-source π0 recipe uses LoRA on the VLM backbone for per-task adaptation; the FM action head is fully fine-tuned (small enough).

12.4 Embodiment-specific tokens

Add per-robot embedding tokens; train alongside LoRA.

★ 2026 SOTA update — OpenVLA-OFT fine-tuning recipe - OpenVLA-OFT: LoRA on the LLM backbone (frozen vision+projector) plus parallel decoding, action chunking, continuous actions and L1 regression; LIBERO 76.5->97.1%, 26x faster action generation (2025).

13. Memory Engineering for PEFT

13.1 Optimizer state

• Adam: 8 bytes per parameter (FP32 m, v).
• AdamW: same.
• Lion: half (no second moment).
• 8-bit Adam (bitsandbytes): 2 bytes per param.
• Paged optimizer: spill to CPU on overflow (QLoRA).

13.2 Gradient checkpointing

Recompute activations on backward pass; saves memory at ~30% extra compute. Standard with PEFT for fitting longer contexts.

13.3 Mixed precision

• BF16 for forward / backward.
• FP32 for optimizer state / master weights.
• FP8 (H100) for matmul where supported.

13.4 LoRA + FlashAttention

PEFT integrates cleanly with FlashAttention. Most kernels handle LoRA via separate matmul + add.

13.5 Sequence parallelism

For long-context PEFT: split sequence across GPUs; each computes attention over local chunk. Combines with LoRA.

13.6 FSDP + PEFT

• FSDP shards base model parameters.
• LoRA params replicated (small).
• Communication: only gather base for forward, reduce LoRA grads.
• Standard for multi-GPU PEFT.

13.7 DeepSpeed ZeRO + PEFT

• ZeRO-3 shards optimizer state, gradients, parameters.
• PEFT only affects LoRA params; minimal memory.
• Used by Axolotl, OpenRLHF.

14. PEFT Frameworks and Tools

14.1 HuggingFace PEFT

• Standard library; supports LoRA, QLoRA, prompt tuning, prefix tuning, AdaLoRA, IA3, OFT.
• Integrates with Transformers, TRL.
• "Adapter modules" API.

14.2 bitsandbytes

• 8-bit optimizers, 4-bit weights, NF4.
• Foundation of QLoRA.
• Standard quantization library for PEFT.

14.3 Unsloth

• 2–5× speedup on QLoRA training via custom kernels.
• Single-GPU friendly (fits 70B with 48GB VRAM).
• Standard for consumer-grade fine-tuning in 2025–26.

14.4 Axolotl

• Higher-level training framework.
• LoRA / QLoRA / DPO / ORPO / SimPO supported.
• YAML-based config.
• Most popular open SFT / DPO framework.

14.5 LLaMA-Factory

Similar to Axolotl; one-stop fine-tuning library. Strong UI / CLI.

14.6 TRL (HuggingFace)

• SFT trainer, DPO trainer, PPO trainer, GRPO trainer.
• LoRA-friendly.
• Standard for RLHF / preference fine-tuning.

14.7 vLLM / S-LoRA / LoRAX

• Multi-LoRA serving at inference time.
• Load / unload adapters per request.
• Standard for multi-tenant / per-user customization.

14.8 Diffusers (HuggingFace)

• Standard PEFT for diffusion (SD, SDXL, FLUX).
• LoRA / DreamBooth / Textual Inversion built-in.
• ComfyUI integrates.

14.9 LyCORIS

Family of advanced PEFT methods specifically for diffusion: LoCon, LoHa, LoKr, Diag-OFT, BOFT, etc.

15. PEFT Recipes by Use Case

15.1 LLM SFT (general instruction tuning)

• LoRA rank 16–64 on attention + MLP.
• LR \(1\times10^{-4}\); cosine to 0.
• 1–3 epochs.
• Effective batch size 64–256.
• Mixed precision BF16; QLoRA if memory-tight.

15.2 LLM DPO

• LoRA rank 8–64.
• LR \(5\times10^{-7}\)–\(5\times10^{-6}\) (much lower than SFT).
• 1 epoch (more risks overfit).
• \(\beta = 0.1\) default.

15.3 LLM domain SFT (medical, legal, code)

• LoRA rank 32–128.
• Full target: attention + MLP.
• Larger dataset; longer training.
• LR moderate.

15.4 LLM continual pretraining

• LoRA rank 128–512 (large).
• Or: full fine-tune if budget allows.
• Massive data (> 10B tokens).
• PEFT may underperform full FT here.

15.5 Diffusion personalization

• LoRA rank 8–32 on cross-attention.
• DreamBooth pattern: unique token + prior preservation.
• 5–20 reference images.
• 500–3000 steps.
• LR \(1\times10^{-4}\).

15.6 Diffusion concept / style

• LoRA rank 16–128.
• Larger dataset (50–500 images).
• Apply to U-Net + sometimes text encoder.
• LoCon / LyCORIS for SD; LoRA for SDXL / FLUX.

15.7 VLM fine-tune

• LoRA on LLM backbone, rank 16–128.
• Freeze vision encoder + projector.
• LR moderate.
• Often combined with image augmentation.

15.8 VLA per-task

• LoRA on VLM backbone, rank 32–256.
• 100–1000 demonstrations.
• Small LR.
• Stack per-task LoRAs at deployment.

15.9 Reasoning RL (GRPO)

• LoRA optional: full fine-tune more common at scale.
• If LoRA: rank ≥64 to capture reasoning capability.
• Cold-start SFT (LoRA) + RL stages.

16. Multi-Adapter Serving and Composition

16.1 Multi-LoRA inference (S-LoRA, LoRAX, vLLM)

• Base model loaded once.
• Per-request LoRA selected and loaded on the fly.
• Hot adapters cached in GPU memory.
• Cold adapters paged from CPU / disk.

16.2 Memory profile

For 7B base + 100 LoRA adapters at rank 16:

• Base: ~14 GB BF16 / ~4 GB Q4.
• Per LoRA: ~40 MB.
• 100 LoRAs: ~4 GB additional.
• Total: 8–18 GB depending on quantization.

16.3 Adapter composition

• Linear sum: \(W + \sum_i \alpha_i B_i A_i\).
• Per-layer mixing (MoLE: Mixture of LoRA Experts).
• Concatenation: stack along rank axis.
• Routing-based: per-token expert selection.

16.4 LoRA Hub

Search across many fine-tuned LoRAs to find good combinations for new tasks. Cross-task transfer.

16.5 Conflict / interference

• Stacking too many LoRAs degrades quality.
• Adapter Fusion (AdapterFusion) trains a fusion module to combine.
• LoRA-MoE / X-LoRA: gated combination.

16.6 Civitai-style ecosystem

• Public LoRA marketplaces.
• Per-style / per-character / per-concept LoRAs.
• Standard in image / video diffusion.

★ 2026 SOTA update — High-throughput multi-LoRA serving - Punica: SGMV CUDA kernel batches many distinct LoRAs over one base copy; ~12x throughput vs prior multi-LoRA serving. - dLoRA: dynamically merges/unmerges adapters and migrates requests+adapters across replicas with cross-adapter batching; up to 1.8x lower latency than S-LoRA (OSDI 2024). - CaraServe: CPU-assisted, cold-start-free, rank-aware LoRA serving; overlaps CPU/GPU to cut latency up to 50% at 99% SLO attainment.

17. Theoretical Foundations

17.1 Intrinsic dimension (Aghajanyan et al.)

Pretrained models have low "intrinsic rank" on task-specific gradients. The fewer dimensions you can fit fine-tunes within, the lower the rank you need. Empirically supports LoRA's design.

17.2 LoRA convergence theory

Recent results (Hayou et al.) show that LoRA can match full FT in expressivity if rank ≥ task complexity.

Practically: most task complexities are low.

17.3 LoRA initialization theory

Why B = 0, A ~ N? Symmetry breaking + identity at init. Other inits explored: orthogonal, Xavier; mostly small differences.

17.4 Spectral analysis

LoRA effectively low-pass filters task gradients. Higher rank = wider band; lower rank = stronger regularization.

17.5 Generalization / forgetting

PEFT tends to preserve pretrained knowledge better than full FT. Implicit regularization via low-rank constraint.

17.6 Capacity-quality trade-off

Quality scales with rank, but with diminishing returns. Most tasks: r = 16–64 suffices; harder: r = 128–512.

17.7 When PEFT underperforms

• Massive distribution shift.
• Long-horizon training (capacity ceiling).
• New modality / vocabulary.
• Use full FT in these cases.

18. Stability, Hyperparameters, Pitfalls

18.1 Common bugs

• Training too few / too many params (rank wrong).
• Wrong LR (too high → instability; too low → underfit).
• Forgetting to zero-init B.
• Mismatched merge / unmerge at inference.
• Gradient checkpointing + LoRA: subtle bugs.
• LR scheduler not anneal-aware.

18.2 Common debug procedure

• Verify trainable params (< 5%).
• Check loss curve (smooth descent).
• Inspect base-model outputs at step 0 (should be unchanged).
• Validate on small held-out set.
• Compare to no-LoRA inference.

18.3 Hyperparameter sensitivity

• LR: most sensitive.
• Rank: usually ±1 rank class doesn't matter.
• α: sometimes important; standard \(\alpha = r\) or 2r.
• Target modules: often matters more than rank.

18.4 Stability tricks

• Linear warmup over ~10% of steps.
• Cosine or WSD anneal.
• Gradient clipping: \(\tau = 1.0\).
• EMA of LoRA weights (rare).

18.5 Cold-start issues

For RL fine-tuning (e.g., GRPO with LoRA): cold-start with SFT-LoRA before RL avoids early collapse.

18.6 Loss-of-capability regression

PEFT can quietly degrade some capabilities. Always test on held-out general benchmarks (MMLU subset, etc.) post-fine-tune.

19. Production Deployment

19.1 Adapter merging strategies

• Permanent merge: single deployment-grade model.
• Hot-swap: keep adapters separate; swap at request time.
• Stack: compose multiple at inference.

19.2 Multi-tenant pattern

• One base model.
• Per-customer LoRA.
• Loaded on demand.
• Sub-second adapter load via vLLM / S-LoRA / LoRAX.

19.3 Continuous fine-tuning

• Periodically update LoRA on fresh data.
• Versioned adapter registry.
• Canary deployment for new versions.

19.4 Cost-quality positioning

• Free tier: base model only.
• Standard tier: shared LoRAs (popular customizations).
• Premium tier: per-user LoRAs.
• Enterprise: full fine-tune for absolute customization.

19.5 LoRA inference latency

• Merged: zero overhead.
• Unmerged + S-LoRA: ~5–10% latency overhead.
• Multi-LoRA stacking: linear in number of adapters.

19.6 Storage

• Per-LoRA: 10–500 MB (rank-dependent).
• Per-base-model: 4–400 GB depending on size + quantization.
• Multi-tenant scalability: 1000s of LoRAs, single base.

19.7 Versioning

• Track base model version + LoRA version.
• Regression test combinations.
• Rollback adapters separately from base.

20. PEFT for RL / Alignment

20.1 LoRA for DPO

Standard for cheap alignment:

• Base model frozen / LoRA-adapted.
• Reference policy = base model itself (no extra memory!).
• 1/2× memory of full DPO.

20.2 LoRA for PPO-RLHF

• LoRA on policy.
• Reference: base model.
• Reward model: separate.
• Value head: small MLP fully trained or LoRA.

20.3 LoRA for GRPO / R1-style RL

• LoRA enables smaller-scale RL on consumer hardware.
• Cold-start SFT-LoRA → GRPO with LoRA.
• Open-R1 / TinyZero recipes use this.

20.4 ORPO + LoRA

Combined SFT + preference learning with LoRA. Memory-efficient alignment.

20.5 SimPO + LoRA

Reference-free preference learning + LoRA. Memory-efficient.

20.6 Why PEFT + RL is powerful

• Reference-policy shared with base.
• No memory doubling.
• Quick experimentation cycle.
• Multi-objective (per-LoRA per-objective).

★ 2026 SOTA update — LoRA-only RL for reasoning - Tina: GRPO-style RL on a 1.5B model with LoRA ONLY; >20% reasoning gain, 43% AIME24 Pass@1 for ~$9 (~260x cheaper) (ICLR 2026). - LoRA Without Regret: Thinking Machines finding that LoRA matches full FT for RL even at small rank, and for SFT if applied to ALL layers (esp. MLP/MoE) with ~10x higher LR.

21. Frontier 2025–2026

21.1 Trends

• LoRA / QLoRA standard for SFT on consumer GPU.
• DoRA matching full FT in many production settings.
• Multi-LoRA serving (S-LoRA, vLLM) production-grade.
• LyCORIS / LoCon ecosystem mature for diffusion.
• PEFT + R1-style RL (GRPO) standard for accessible reasoning fine-tunes.
• VeRA-class shared-low-rank methods reducing storage.

21.2 Hybrid full FT + PEFT

• Full FT critical layers; LoRA the rest.
• Layer-importance analysis decides.
• Used in some production frontier labs.

21.3 Adapter routing / LoRA-MoE

• Per-token routing to adapters.
• More capacity at fixed cost.
• X-LoRA, MoLE, MoLA active research.

21.4 Adapter generation

• Hyper-network generates LoRA from task description.
• Few-shot: instant new adapter.
• Active research; not yet production-grade.

21.5 Composable safety adapters

• Train safety / refusal LoRAs separately.
• Compose with capability LoRAs at deploy.
• Modular alignment.

21.6 Open research

• Theoretical PEFT capacity vs full FT for hard tasks.
• Adapter generalization across base models.
• Conflict-free multi-adapter composition.
• Continual / lifelong PEFT learning.
• Hyper-network adapter generators.

22. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. LoRA: \(W' = W + \alpha \cdot BA/r\); \(A\) Gaussian, \(B\) zero.
2. Standard rank: 8–64 for LLM SFT; 16–128 for diffusion.
3. LoRA matches full FT at ≥7B for many tasks at 1/100–1/1000 params.
4. QLoRA: NF4 base + BF16 LoRA + paged optimizer + double quant.
5. Unsloth: 2–5× QLoRA training speedup.
6. DoRA: magnitude + direction decomposition; closer to full FT than LoRA.
7. rsLoRA: \(\alpha/\sqrt{r}\) scaling; better at high rank.
8. ReLoRA: iteratively merge + re-init for higher effective rank.
9. OFT / BOFT: orthogonal rotation; preserves spectrum.
10. VeRA: shared A, B across layers; even fewer params.
11. LoHa, LoKr: Hadamard / Kronecker decompositions.
12. IA3: per-feature scaling vectors; tiny param count.
13. BitFit: train only biases.
14. Adapters (Houlsby / Pfeiffer): bottleneck modules.
15. Soft prompts: learnable embedding tokens prepended.
16. Prefix tuning: per-layer learnable KV prefixes.
17. LoRA target modules: attention (Q,K,V,O) + MLP (up, down).
18. For DPO: LoRA + ref = base (no memory doubling).
19. For RL (GRPO): cold-start SFT-LoRA → RL-LoRA.
20. Multi-LoRA serving: vLLM / S-LoRA / LoRAX.
21. LoRA stacking: \(W + \sum_i \alpha_i B_i A_i\).
22. LoCon / LoHa / LoKr / DoRA in LyCORIS for diffusion.
23. LoRA + FSDP / DeepSpeed ZeRO for multi-GPU PEFT.
24. Layer-importance / AdaLoRA for adaptive rank allocation.
25. Don't use PEFT for: massive shift, vocab change, very long continual.

Appendix B: Decision Tree — "Which PEFT?"

1. LLM SFT, single GPU? → QLoRA + Unsloth (consumer) or LoRA + FSDP.

2. LLM DPO / SimPO? → LoRA + TRL; rank 8–64; low LR.

3. LLM GRPO / R1-style RL? → LoRA + cold-start + OpenRLHF / verl.

4. Need full FT quality but single GPU? → DoRA or QLoRA at rank 256.

5. Diffusion personalization (subject)? → LoRA + DreamBooth pattern.

6. Diffusion style / concept? → LoRA / LyCORIS LoCon / LoHa / LoKr.

7. Diffusion identity-preserving? → OFT / BOFT.

8. Tiny adapter (storage-tight)? → VeRA / IA3 / BitFit.

9. Multi-tenant serving? → vLLM / S-LoRA / LoRAX with hot-swap.

10. NLU / ViT classification? → LoRA / VPT / Adapters.

11. Massive distribution shift? → Full fine-tune (PEFT may ceiling).

12. RL alignment with shared reference? → LoRA + base-as-reference (no memory doubling).

Appendix C: Year-by-Year PEFT Milestones

• 2019: Houlsby Adapter; Pfeiffer Adapter; AdapterHub.
• 2020: Aghajanyan et al. intrinsic dimension; BitFit; prompt tuning.
• 2021: LoRA (Hu et al.); prefix tuning; P-tuning; Compacter.
• 2022: AdapterFusion; IA3; CoOp / CoCoOp for CLIP; CLIP-Adapter / Tip-Adapter.
• 2023: QLoRA / NF4 (Dettmers); LyCORIS (LoCon / LoHa / LoKr); P-tuning v2; AdaLoRA.
• 2024 (early): DoRA; rsLoRA; LoRA+; S-LoRA / LoRAX (multi-LoRA serving); OFT / BOFT for diffusion.
• 2024 (mid–late): Unsloth (consumer-GPU PEFT); VeRA; LoRA-FA; LoRA-XS; ReLoRA; Civitai LoRA marketplace mainstream; multi-LoRA serving production-grade.
• 2025: PEFT + GRPO (R1-style RL via LoRA); cold-start SFT-LoRA standard; LoRA-MoE / X-LoRA active research; FLUX LoRAs ubiquitous; SD 3.5 LoRAs.
• 2026: PEFT mainstream for fine-tuning at all scales; multi-LoRA per-user customization standard in commercial APIs; DoRA / OFT mainstream alongside LoRA; PEFT theory reaching practical maturity (capacity bounds, conflict-free composition).