Distillation — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations of Knowledge Distillation
2. Response Distillation
3. Feature Distillation
4. Relational Distillation
5. Self-Distillation and EMA Teachers
6. Vision Model Distillation
7. Language Model Distillation: Encoder-Only
8. Language Model Distillation: Decoder-Only / Generative
9. Reasoning Distillation: R1-Distill and Beyond
10. Diffusion Model Distillation
11. Multimodal and Cross-Modal Distillation
12. Cross-Architecture Distillation
13. Multi-Teacher Distillation
14. Online vs Offline Distillation
15. Distillation + RL Pipelines
16. Augmentation and Synthetic Data
17. Calibration and Temperature Tricks
18. Distillation for Compression and Pruning
19. Production Patterns
20. Open Distilled Models Worth Knowing
21. Failure Modes and Mitigations
22. Theory and Analysis
23. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which Distillation Method?"
• Appendix C: Year-by-Year Milestones

1. Foundations of Knowledge Distillation

1.1 The teacher-student paradigm

A large teacher model \(T\) is trained first; a smaller student model \(S\) is trained to mimic the teacher. Student inherits much of the teacher's capability at fraction of the inference cost.

1.2 Why distill?

• Compression: shrink a large model into a small one for deployment.
• Capability transfer: bake reasoning / agentic skills from a frontier model into a smaller open one.
• Few-step generation: turn a 50-step diffusion sampler into a 1–4 step one.
• Cross-architecture: transfer between very different architectures (Transformer ↔︎ SSM, ConvNet ↔︎ ViT).
• Privacy / sovereignty: open self-hosted student of a closed API teacher.
• Specialization: distill a generalist into a domain expert.

1.3 Hinton's original (2015)

Soft targets from teacher's softmax with temperature \(\tau\):

\[p_T(y\mid x) = \mathrm{softmax}(z_T(x)/\tau).\]

Student trained on combined hard label + soft target loss:

Key

Classical KD loss:

\[\mathcal{L}_{\mathrm{KD}} = \alpha\,\mathrm{CE}(y, \mathrm{softmax}(z_S)) + (1-\alpha)\,\tau^2\,\mathrm{KL}\big(\mathrm{softmax}(z_T/\tau)\,\|\,\mathrm{softmax}(z_S/\tau)\big).\]

\(\tau \in [1, 10]\) typical; \(\alpha \in [0.1, 0.5]\). The \(\tau^2\) scaling preserves gradient magnitude.

1.4 Why soft targets help

Soft targets contain "dark knowledge": the relative probabilities across non-target classes. Cat-vs-dog: teacher emits \(\sim\) 0.99 cat, but the 0.005 on "lynx" and 0.001 on "car" tell the student about feature similarity.

1.5 Three families

1. Response distillation: match output distribution / final logits.
2. Feature distillation: match intermediate hidden states.
3. Relational distillation: match relations among samples (pairwise / structural).

1.6 Distillation vs other transfer methods

• vs Fine-tuning: distillation uses teacher signal, not just data labels.
• vs Transfer learning: distillation transfers behavior, not just weights.
• vs Pretraining: distillation needs a teacher; pretraining is unsupervised.

2. Response Distillation

2.1 Logit matching

The cleanest form: minimize divergence between teacher and student output distributions.

Forward KL (mass-covering): student covers all teacher modes. Risk: dilutes precision.

\[\mathcal{L}_{\mathrm{FwdKL}} = \mathrm{KL}(p_T \,\|\, p_S) = \sum_y p_T(y)\log\frac{p_T(y)}{p_S(y)}.\]

Reverse KL (mode-seeking): student picks one teacher mode and sharpens.

\[\mathcal{L}_{\mathrm{RevKL}} = \mathrm{KL}(p_S \,\|\, p_T) = \sum_y p_S(y)\log\frac{p_S(y)}{p_T(y)}.\]

2.2 When to use which

• Forward KL: standard for classification; covers all classes.
• Reverse KL: better for generation tasks where student capacity is too small to cover the full teacher distribution; produces sharper, more decisive outputs.
• JSD / symmetric: balance between the two.

2.3 Temperature schedule

• Higher \(\tau\): softer targets, more dark knowledge but smaller gradient.
• Lower \(\tau\): closer to hard labels, stronger gradient.
• \(\tau \to 1\): just soft cross-entropy.
• Often anneal \(\tau\) from high to low across training.

2.4 Sequence-level distillation (Kim & Rush 2016)

For sequence models, instead of token-level KL, distill on teacher's argmax sequence:

\[\mathcal{L}_{\text{seq-KD}} = -\log p_S(\hat{y}_T \mid x), \qquad \hat{y}_T = \arg\max_y p_T(y\mid x).\]

Cheap (no per-token teacher distribution); often as effective as token-level. Standard in NMT.

2.5 Generalized KD (GKD, Agarwal et al. 2023)

On-policy: sample sequences from student; compute teacher log-probs on those samples; minimize KL or JSD.

Avoids exposure bias of teacher-forcing.

\[\mathcal{L}_{\mathrm{GKD}} = \mathbb{E}_{y\sim \pi_S(\cdot\mid x)}\big[\mathrm{KL}\big(p_T(\cdot\mid x, y_{<t}) \,\|\, p_S(\cdot\mid x, y_{<t})\big)\big].\]

2.6 Top-k logit distillation

Distill only on top-\(k\) teacher logits (rest grouped as "other"). Reduces communication for cross-tokenizer or remote distillation.

2.7 Black-box distillation

Teacher accessible only through samples (not logits). Use sequence-KD or generated-data SFT. Needed when distilling from a closed-API model (GPT-4, Claude).

3. Feature Distillation

3.1 FitNets (Romero et al. 2014)

First feature distillation: match an intermediate feature map of teacher and student via a learned projection \(W_p\):

\[\mathcal{L}_{\mathrm{FitNet}} = \|f_T(x) - W_p\, f_S(x)\|_2^2.\]

3.2 Attention Transfer (AT, Zagoruyko & Komodakis)

Match attention maps (sum over channels) of intermediate layers:

\[\mathcal{L}_{\mathrm{AT}} = \left\|\frac{Q_T}{\|Q_T\|_2} - \frac{Q_S}{\|Q_S\|_2}\right\|_2, \qquad Q = \sum_c |F_c|^p.\]

Cheap and effective for CNNs.

3.3 Hidden-state distillation in Transformers

TinyBERT / MobileBERT match:

• Embedding layer outputs.
• Each Transformer block's hidden state.
• Each block's attention matrix.

Each via MSE with a per-layer projection.

3.4 Gram matrix matching

For style transfer / texture: match Gram matrix \(G = F^\top F\) of features. Captures second-order statistics.

3.5 Activation boundary

Match the boundaries (sign-pattern) of activations rather than raw values. More tolerant of capacity mismatch.

3.6 Where to inject feature loss

• Lots of layers: information-rich; expensive; may over-constrain student.
• Few critical layers (e.g., every 3rd): cheap; works in practice.
• Last layer only: simplest; effectively response distillation.
• Mid-network: often the sweet spot for capacity mismatch.

4. Relational Distillation

4.1 Relational KD (RKD, Park et al.)

Match pairwise distances and triplet angles of samples in feature space:

\[\mathcal{L}_{\text{RKD-D}} = \sum_{i,j} \ell\big(\psi_D(f_T^i, f_T^j),\, \psi_D(f_S^i, f_S^j)\big),\]

where \(\psi_D\) is normalized distance.

4.2 Similarity-Preserving KD (SPKD)

Match the pairwise similarity matrix of features in a batch.

4.3 Contrastive Representation Distillation (CRD)

Train student so its features are pulled toward teacher's features for the same input, pushed away from teacher's features for different inputs (InfoNCE-style).

4.4 Why relational?

Robust to capacity mismatch: student doesn't have to match raw features (impossible if dimensions differ), only the relations. Often better than feature distillation alone.

5. Self-Distillation and EMA Teachers

5.1 Born-Again Networks (BAN, Furlanello et al.)

Train a student with the same architecture as the teacher; iterate (BAN-1 → BAN-2 → …). Each generation slightly improves. Why it works is debated — ensemble effect, regularization, or feature reorganization.

5.2 Self-distillation across layers

Deep network: deeper layers' outputs serve as teachers for earlier layers' classifier heads. Improves training; modest gains.

5.3 Snapshot ensembles

Save snapshots during training (cyclic LR); average / ensemble for distillation target.

5.4 Mean Teacher (Tarvainen & Valpola)

Semi-supervised: student trained on labeled data + consistency loss with EMA-of-student teacher's predictions on unlabeled data:

\[\mathcal{L}_{\mathrm{MT}} = \mathcal{L}_{\mathrm{sup}} + \lambda\, \mathbb{E}_x \big\|p_S(x) - p_{\bar S}(x')\big\|_2^2,\]

\(\bar S\) is EMA of \(S\), \(x'\) is augmented \(x\). Foundation of SSL for vision.

5.5 EMA targets in modern SSL

BYOL: online net + EMA target net + predictor; no negatives; collapse avoided by stop-gradient on target.

DINO / DINOv2 / DINOv3: student-teacher self-distillation with sharpening + centering; teacher is EMA of student. The most successful self-distillation paradigm at scale.

V-JEPA / V-JEPA 2: predict EMA teacher's latent (not pixels) for video.

5.6 EMA in diffusion

Maintain EMA of model weights \(\bar\theta_t = \alpha\,\bar\theta_{t-1} + (1-\alpha)\,\theta_t\), \(\alpha \sim 0.999\)–\(0.9999\). Sample with \(\bar\theta\). Worth +1–3 quality points; required for diffusion sample fidelity.

5.7 Self-distillation as regularization

Adding a self-KD loss to standard training (without an explicit teacher) often improves generalization. Cheap regularizer.

6. Vision Model Distillation

6.1 DeiT (Touvron et al.)

ViT trained on ImageNet-1k via distillation from a strong CNN teacher (RegNet). Adds a special [DIST] token that learns to match teacher's predictions:

\[\mathcal{L}_{\mathrm{DeiT}} = \mathcal{L}_{\mathrm{CE}}([\mathrm{CLS}]) + \mathcal{L}_{\mathrm{KD}}([\mathrm{DIST}], \text{teacher}).\]

Made ViT viable without giant pre-training datasets.

6.2 DINO / DINOv2 / DINOv3

Self-distillation as SSL. Student gets a small crop, teacher gets a global crop; student matches teacher's softened predictions:

\[\mathcal{L}_{\mathrm{DINO}} = -\,\mathrm{softmax}\!\big((g_\xi(v_t) - c)/\tau_t\big)^\top \log\,\mathrm{softmax}\big(g_\theta(v_s)/\tau_s\big).\]

EMA teacher \(\xi\), sharpening (low \(\tau_t\)) + centering (subtract running mean \(c\)) prevent collapse.

6.3 EVA / EVA-02

Distill frozen CLIP features into a ViT trained with masked image modeling. Combines two strong sources of supervision.

6.4 MAE / SimMIM cross-distillation

MAE-pretrained backbone distilled into smaller ViT for downstream. Standard pre-training-then-distill pattern.

6.5 ConvNeXt V2 + FCMAE

ConvNet pretrained with masked-image modeling; distilled / fine-tuned for downstream.

6.6 SAM 2 distillation

EfficientSAM, MobileSAM, FastSAM: distilled from SAM into smaller image / mobile-friendly variants. Mask-prediction distillation; combine pixel and feature losses.

6.7 Detection distillation

• LD (Localization Distillation): distill bounding-box distribution.
• DETR-style: distill object queries' outputs.
• Cross-frame KD for video detection.

7. Language Model Distillation: Encoder-Only

7.1 DistilBERT (Sanh et al. 2019)

• Student: BERT with half the layers (initialized from alternate teacher layers).
• Loss: \(\alpha_{\mathrm{MLM}}\mathcal{L}_{\mathrm{MLM}} + \alpha_{\mathrm{KD}}\mathcal{L}_{\mathrm{KL}}(\text{teacher}, \text{student}) + \alpha_{\cos}\mathcal{L}_{\cos}(\text{features})\).
• 60% smaller, 60% faster, 97% performance.

7.2 TinyBERT

Two-stage:

1. Pretrain student with general distillation from teacher (embedding + attention + hidden).
2. Task distillation: fine-tune teacher on task; distill to student.

\(\sim\) 7× smaller, \(\sim\) 9× faster than BERT-Base.

7.3 MobileBERT

Student narrower (256 hidden vs 768) but as deep. Bottleneck blocks. Distill embedding + each layer + final logits.

7.4 MiniLM

Distill only the attention distributions of the last layer (and value-relations).

Architecture-agnostic; deeper student possible.

7.5 ALBERT

Not distillation per se, but parameter-efficient via factorized embedding + cross-layer sharing. Often combined with KD for further compression.

7.6 Best practices for encoder-only KD

• Initialize student layers from selected teacher layers.
• Use both MLM (data) and KL (teacher) losses.
• Match attention maps and hidden states, not just logits.
• Pretrain student with KD before task fine-tune.

8. Language Model Distillation: Decoder-Only / Generative

8.1 Distilling from GPT-style models

Sequence-KD (teacher forcing on teacher's argmax sequences) is the workhorse. Token-level KL also used. On-policy GKD often best.

8.2 Phi family (Microsoft)

Phi-1, Phi-1.5, Phi-2, Phi-3, Phi-4: small (< 14B) models trained heavily on synthetic textbook-quality data generated by a frontier teacher (GPT-4). Demonstrated dramatic capability transfer via curated data, not just KL.

8.3 Gemma 2 / Gemma 3

Open Google models distilled from Gemini Pro on web + synthetic data. Gemma 2 explicitly cites distillation in its training recipe.

8.4 Llama 3 / Llama 4 SFT

Use teacher (larger Llama or other frontier model) outputs as part of the SFT data mix. Less explicit "KD" framing, more "teacher-generated SFT data."

8.5 Open-Synthetic distillation pattern

1. Teacher (GPT-4, Claude, etc.) generates instructions + responses.
2. Filter for quality.
3. SFT a smaller open model on the filtered set.
4. Optionally: DPO with teacher-graded preferences.

Underlies many strong open instruction-tuned models (Vicuna, WizardLM, OpenChat, Tulu, OpenHermes, etc.).

8.6 Tool-use distillation

Distill tool-calling traces from an agentic teacher. Covers tool selection, argument generation, multi-step orchestration.

8.7 Reasoning distillation (the new wave)

Covered in detail in §12; the dominant LLM distillation pattern of 2025–26.

8.8 Cross-tokenizer distillation

• Different tokenizers → token-level KL impossible.
• Solutions: align via shared subword decoder; distill sequence-level only; distill at character / byte level.
• Universal Logit Distillation, ULD: align via decoder.

★ 2026 SOTA update — Distilled speculative drafters

• EAGLE-3: draft head trained by 'training-time test' with direct token prediction (not feature regression) + multi-layer feature fusion from the frozen target; acceptance length scales with training data, giving SoTA lossless speedup (NeurIPS 2025).

9. Reasoning Distillation: R1-Distill and Beyond

9.1 The pattern

Generate long chain-of-thought traces with a strong reasoner (DeepSeek-R1, OpenAI o1, etc.); SFT a smaller model on those traces. The smaller model inherits reasoning patterns at fraction of inference cost.

9.2 R1-Distill series (DeepSeek 2025)

DeepSeek released R1-Distill checkpoints: Qwen 1.5B / 7B / 14B / 32B and Llama 8B / 70B fine-tuned on \(\sim\) 800k R1 traces. Strong reasoning quality at far smaller scale; outperformed many larger non-reasoning models on math + code.

9.3 Trace quality matters

• Filter for correct final answers.
• Prefer concise but complete chains over rambling.
• Diversity: multiple solution styles per problem.
• Reject reward-hacked patterns (format-only, repetitive).
• Per-problem: keep best-of-N traces.

9.4 Cold-start data for downstream RL

Distilled traces serve as the SFT cold-start before further GRPO. Stabilizes RL and accelerates convergence:

• Without cold-start: pure RL needs scale (\(\ge\) 7B base) to bootstrap reasoning.
• With cold-start: smaller models can do GRPO RL successfully.

9.5 Distillation vs direct RL

• Distillation: cheap, fast, leverages a frontier teacher; bounded by teacher quality.
• Direct RL: more expensive, but the student can surpass any specific teacher; required if no strong teacher exists.

In practice: distill to bootstrap, then RL to improve beyond teacher.

9.6 Multimodal reasoning distillation

• Vision-R1 / VLM-R1 / MM-EUREKA: distill from larger reasoning VLMs.
• Visual CoT traces filtered by visual-grounded verifier.
• Cross-modal grounding verified by external classifier.

9.7 Open replications

• Open-R1 (HuggingFace): full R1 reproduction; distillation and RL stages.
• TinyZero, SimpleRL: small-scale distillation + R1-Zero style.
• QwQ-distill, Skywork-o1-distill: open reasoning distill checkpoints.

★ 2026 SOTA update — Data-efficient long-CoT SFT

• s1: 1K curated reasoning traces + 'budget forcing' (append 'Wait' to extend thinking) match o1-preview; s1K-1.1 regenerates traces with DeepSeek-R1 for a big boost.
• LIMO: ~800 strategically-curated traces reach 63.3% AIME24 / 95.6% MATH500 using ~1% of prior data; 'Less-Is-More' hypothesis—reasoning is elicited, not taught, when pretraining already encodes the knowledge.

10. Diffusion Model Distillation

10.1 Why distill diffusion

Diffusion samplers need 25–50 NFEs (network function evaluations) for high quality. Production deployment wants 1–4 NFEs. Distillation collapses many teacher steps into few student steps.

10.2 Progressive Distillation (Salimans & Ho)

Teacher runs \(N\) steps; student trained to imitate two teacher steps in one. Iterate: \(N \to N/2 \to N/4 \to \cdots \to 1\).

Slow (multiple training rounds) but reliable.

10.3 Consistency Models (Song et al.)

Train \(f_\theta(x_t, t) \approx x_0\) for any \(t\) along the same ODE trajectory:

\[\mathcal{L}_{\mathrm{CM}} = \mathbb{E}\, d\big(f_\theta(x_{t_{n+1}}, t_{n+1}),\, f_{\theta^-}(\hat{x}_{t_n}, t_n)\big),\]

EMA target \(\theta^-\), distance \(d\) = LPIPS or L2. 1–4 step inference.

10.4 Latent Consistency Models (LCM)

Same framework in latent space; SDXL / SD3 / FLUX have LCM variants. The standard distilled SDXL through 2024.

10.5 DMD / DMD2 (Distribution Matching Distillation)

Train one-step student by matching the score field of the teacher:

\[\nabla_\theta \mathcal{L}_{\mathrm{DMD}} = \mathbb{E}\big[(s_{\mathrm{real}}(\hat{x}) - s_{\mathrm{fake}}(\hat{x}))\cdot \partial\hat{x}/\partial\theta\big].\]

DMD includes a regression loss; DMD2 drops it and adds a GAN-style discriminator. Reaches teacher quality in one step.

10.6 Hyper-SD (Bytedance)

Blends consistency + adversarial + ODE-trajectory losses; few-step SDXL/FLUX with high quality. Production-friendly.

10.7 Phased Consistency Models (PCM)

Divide ODE into segments; apply consistency within each. Recovers quality at very few NFEs without quality cliff.

10.8 Score Identity Distillation (SiD)

One-step distillation that doesn't need a trajectory teacher. Uses a Stein-style score-matching identity to derive the loss.

10.9 InstaFlow / Reflow

Use rectified-flow recipe to straighten then distill. 1–2 step text-to-image at SDXL quality.

10.10 LADD (SD Turbo)

Latent Adversarial Diffusion Distillation: train 1–4 step student with discriminator in latent space; teacher is the original SDXL.

10.11 Distillation lineup (open frontier)

Watch out

Distilled models are lower-diversity than teachers. For best aesthetic / creative range (chat, real-time editing), sample with the teacher. For latency-critical apps, distilled wins.

★ 2026 SOTA update — f-divergence and adversarial score distillation

• f-distill: generalizes DMD's reverse-KL score matching to the full f-divergence family (gradient = teacher-student score gap × density-ratio weight); JS divergence gives SoTA 1-step ImageNet64 and zero-shot COCO.
• ADM: Adversarial Distribution Matching replaces reverse-KL with diffusion-based discriminators aligning real/fake score estimators; hybrid latent+pixel discriminators for 1-step; fixes DMD mode collapse, works for image and video (ICCV 2025 Highlight).
• Uni-Instruct: unifies 10+ one-step distillers (DMD, SiD, f-distill) via an expanded-f-divergence theory; 1-step FID 1.02 on ImageNet64, beating its 79-step teacher (NeurIPS 2025).

11. Multimodal and Cross-Modal Distillation

11.1 Vision-language distillation

• EVA: distill frozen CLIP features into a ViT trained with MIM.
• TinyCLIP: distill OpenCLIP into a smaller dual-encoder via image-text alignment.
• OpenCLIP-base distilled from OpenCLIP-large.

11.2 VLM distillation

• Distill outputs of a strong VLM (Qwen-VL, InternVL) into a smaller VLM.
• Match per-token logits + intermediate features.
• Particularly strong for OCR / dense visual reasoning where teacher is much larger.

11.3 Speech-to-text distillation

Distil-Whisper: distill OpenAI Whisper-large into smaller Whisper variants. 6× faster, 49% smaller, \(\sim\) 1% WER regression.

11.4 Video distillation

• InternVideo distilled from larger video encoders.
• V-JEPA-Distill: smaller video encoder matching V-JEPA-Giant.

11.5 3D distillation

• LRM family: large-model triplane outputs distilled into smaller 3D feed-forward models.
• Gaussian Splatting compression via distillation: LightGaussian, CompGS.

12. Cross-Architecture Distillation

12.1 Transformer → CNN

Early DeiT did the inverse: ViT distilled from CNN. The reverse (Transformer-to-CNN) less common.

12.2 Large → Small same-architecture

The most common case: same layer pattern, fewer layers / narrower hidden. Initialize student from selected teacher layers.

12.3 Transformer → SSM (Mamba)

• Distill softmax-attention model into linear-attention / SSM model.
• Quality bridge: student inherits language understanding; gains long-context efficiency.
• Mamba2-distill, LoLCATs: open recipes.

12.4 MoE → Dense

Distill a sparse MoE into a dense student of the same active-param count. Loses memory advantage; gains serving simplicity.

12.5 Dense → MoE (sparse upcycling)

Inverse: replicate dense MLPs \(N\) times (with noise) + add router; train. \(\sim\) 5–10% of original data recovers MoE quality. Cheap path to MoE without pretraining from scratch.

12.6 High-res → low-res (vision)

Distill ViT trained at \(224^2\) to ViT inferring at \(96^2\). Teacher's intermediate features serve as targets.

★ 2026 SOTA update — MoE-specific distillation

• Every Expert Matters: non-activated experts hold useful knowledge—Knowledge Augmentation + Student-Aware Router extract signal from ALL experts for MoE-to-dense/small KD (standard KD wastes it).
• ZEDA: converts a post-trained static MoE into a dynamic one by injecting parameter-free zero-output experts and two-stage self-distillation (frozen MoE as teacher); skips >50% expert FLOPs at marginal accuracy loss, ~1.2x speedup on Qwen3-30B-A3B.

13. Multi-Teacher Distillation

13.1 Why multi-teacher

• Different teachers are strong at different tasks.
• Diversity reduces variance.
• Domain specialists → generalist student.

13.2 Methods

• Mean ensemble target: average teacher logits.
• Weighted ensemble: per-task weights based on teacher accuracy.
• Per-input gating: learn which teacher to trust for which input.
• Round-robin: alternate teachers across batches.

13.3 Teacher Pool patterns

• Train multiple specialists; distill into one student.
• "Teacher of Teachers": hierarchical distillation.
• Mixture of Experts as multiple teachers (Branch-Train-MiX).

14. Online vs Offline Distillation

14.1 Offline (precomputed)

Teacher inferences computed once, cached. Student trains on cached outputs.

• Pro: cheap student training (no teacher cost per epoch).
• Con: stale; student can't adapt to new prompts.

14.2 Online (live teacher)

Teacher runs alongside student during training.

• Pro: fresh signal; teacher can adapt.
• Con: \(\sim\) 2× training cost.

14.3 On-policy distillation (GKD)

Sample sequences from student; teacher provides per-step targets. Avoids teacher-forcing exposure bias.

14.4 Two-stage hybrid

Stage 1: offline KD from teacher's pre-generated data. Stage 2: online distillation on student rollouts (GKD).

Fast and high-quality.

14.5 Self-distillation (online special case)

Teacher = previous version of student (EMA or hard snapshot). DINO, BYOL, V-JEPA all do this.

★ 2026 SOTA update — On-policy distillation revival

• On-Policy Distillation (Thinking Machines, 2025): dense per-token reverse-KL supervision scored on the student's own rollouts—RL's on-policy error-correction with SFT's reward density; replicates Qwen3 reasoning gains at a fraction of RL cost via the Tinker API.

15. Distillation + RL Pipelines

15.1 Distill-then-RL

1. Distill long-CoT traces from teacher (e.g., R1).
2. SFT on distilled traces.
3. Apply GRPO with verifiable rewards.
4. Result: small model with reasoning + further improvement beyond teacher.

DeepSeek's R1 pipeline; replicated by Open-R1, TinyZero.

15.2 RL-then-distill

1. Train a large model with RL (e.g., R1).
2. Generate high-quality traces.
3. Distill into smaller model.

The R1-Distill series.

15.3 Iterative distill-RL-distill

1. SFT student on teacher traces.
2. RL fine-tune student.
3. Sample traces from improved student.
4. Re-distill into next-generation student.
5. Repeat.

"Bootstrapping" pattern that beats either alone.

15.4 Reward distillation

Distill a reward model's preferences into a student. Useful for offline RL or for preference fine-tuning.

16. Augmentation and Synthetic Data

16.1 Data augmentation in distillation

Augmenting student inputs (cropping, masking, mixup) while teacher sees original → student learns invariances.

16.2 Mixup distillation

\[\tilde{x} = \lambda x_i + (1-\lambda)x_j, \qquad \tilde{p} = \lambda p_T(x_i) + (1-\lambda)p_T(x_j).\]

Student trained on \((\tilde{x}, \tilde{p})\). Smoothing effect.

16.3 Synthetic data generation by teacher

Teacher generates new (input, output) pairs; student trains on these. Can supplement or replace real data:

• Self-Instruct (Wang et al.): teacher generates instructions + responses.
• Phi training: textbook-quality synthetic data.
• Distill across domains via teacher-generated examples.

16.4 Best-of-N synthetic data

• Generate \(N\) samples per input; keep top-\(K\) by reward / quality.
• Higher signal-to-noise than naive single-sample.

16.5 Data filtering

Critical:

• Filter for correctness (verifiable tasks).
• Filter for diversity (avoid mode collapse).
• Filter for safety (avoid distilling unsafe behaviors).
• Filter for length (avoid teacher's verbosity bias).

16.6 Constitutional distillation

Teacher critiques + revises responses against a constitution; student trained on revised data. Distills aligned behavior, not just task capability.

★ 2026 SOTA update — Reward-guided dataset distillation

• AdvDistill: sample \(N\) teacher responses per prompt, score with rule-based verifiers, and use the normalized rewards as per-example training weights—moves beyond copying in-distribution teacher outputs and lifts SLM math/reasoning generalization.

17. Calibration and Temperature Tricks

17.1 Temperature schedule

• Start higher (\(\tau \sim 4\)); soften early.
• Anneal to \(\tau \sim 1\) as training proceeds.
• Per-task / per-layer temperature possible.

17.2 Logit clamping

Cap teacher logits to avoid extreme outliers. Especially for sequence distillation.

17.3 Label smoothing on teacher targets

Combine teacher soft targets with \((1 - \epsilon)\) teacher + \(\epsilon/K\) uniform. Robustness to teacher overconfidence.

17.4 Per-class temperature

For long-tail classes, use higher temperature so student gets enough signal on rare classes.

17.5 Distillation in mixed-precision

Compute teacher in BF16/FP16; compute KD loss in FP32 for numerical stability.

18. Distillation for Compression and Pruning

18.1 Layer pruning + distillation

Drop teacher layers; fine-tune the pruned model with KD against the original teacher. Standard recipe for layer reduction.

18.2 Width reduction + distillation

Reduce hidden dim or attention heads; KD recovers most of the lost quality.

ALBERT-style factorization combined with KD.

18.3 Quantization + distillation

QAT (Quantization-Aware Training) often combined with KD: student is the quantized version, teacher is the FP version. Gradient flows through STE; KD loss helps recover quantization error.

18.4 Sparse architecture distillation

Magnitude / Wanda pruning + KD recovery. Standard for production model compression.

18.5 LayerDrop, Stochastic Depth

Random layer skipping during training → inherently distillation-friendly student that's robust to layer removal.

18.6 Distillation of attention sparsity

Distill from dense-attention teacher into sparse-attention student (e.g., Longformer-style). Student gets long-context efficiency.

19. Production Patterns

19.1 Distill at deployment time

• Train large teacher in cloud.
• Distill into tier-specific students (cloud-large, edge-small, mobile-tiny).
• Deploy student per platform.

19.2 Continual distillation

• As teacher improves (new training run, RLHF iteration), re-distill student.
• Maintain teacher-student version mapping.
• A/B test before swap.

19.3 Cascades + distillation

• Tier-0: tiny distilled model handles easy queries cheaply.
• Tier-1: medium model on uncertain queries.
• Tier-2: full teacher on hard queries.
• Cost reduction: 70–90% with comparable quality.

19.4 Safety-aware distillation

• Teacher's refusal behavior must transfer.
• Distill on safety-test prompts; ensure student inherits.
• Constitutional + KD: safety baked in from teacher.

19.5 Open-source distillation pattern

• Generate \(\sim\) 10k–1M synthetic SFT examples using closed teacher.
• Filter + dedupe.
• SFT smaller open base.
• Optionally: DPO with teacher-judge preferences.
• Examples: Vicuna, WizardLM, OpenChat, Tulu, OpenHermes.

Watch out

Closed-API ToS often restricts using outputs to train competing models. Check legal terms. Open-data alternatives (Tulu, OpenAssistant) avoid this issue.

20. Open Distilled Models Worth Knowing

20.1 Encoder-only

DistilBERT, TinyBERT, MobileBERT, MiniLM, ALBERT, ModernBERT-base.

20.2 Decoder-only general-purpose

DistilGPT-2: legacy. Phi-1/1.5/2/3/3.5/4: synthetic-data distilled. Gemma 2/3: distilled from Gemini Pro.

Open-instruction series (Vicuna, OpenChat, OpenHermes, Tulu): distilled from frontier APIs into Llama/Mistral.

20.3 Reasoning

R1-Distill-Qwen-1.5B / 7B / 14B / 32B, R1-Distill-Llama-8B / 70B: DeepSeek's open distillation series.

Strong reasoning at modest scale.

20.4 Vision

DeiT: original distill-trained ViT. EVA / EVA-02: CLIP-feature distillation. TinyCLIP, TinyViT, MobileViT: mobile vision models. MobileSAM, EfficientSAM, FastSAM: distilled SAMs. Distil-DINOv2.

20.5 Speech

Distil-Whisper: ASR distilled. Distil-Conformer.

20.6 Diffusion

LCM-SDXL, SDXL Turbo, SDXL Lightning, Hyper-SDXL, PCM-SDXL, DMD2-SDXL, FLUX schnell, Hyper-FLUX, LCM-FLUX.

20.7 Robotics

Smaller VLAs distilled from larger pretrained ones. \(\pi_0\)-mini-style distillations forthcoming.

21. Failure Modes and Mitigations

21.1 Capacity mismatch

Student too small to fit teacher distribution. Fix: relational distillation; reverse KL (mode-seeking); narrower task scope.

21.2 Mode collapse

Student over-fits to teacher's argmax; loses diversity. Fix: lower temperature on teacher; mixup; data augmentation.

21.3 Teacher overconfidence

Teacher produces near-one-hot; soft target signal weak. Fix: higher temperature; label smoothing on teacher.

21.4 Wrong inductive bias

Teacher's architecture poorly matched to student's strengths. Fix: relational distillation (architecture-agnostic); cross-arch hybrid pretraining.

21.5 Catastrophic forgetting during distillation

Student's pretrained capabilities erased. Fix: mix general data with KD data; lower learning rate; LoRA-only updates.

21.6 Length explosion (sequence)

Distilled student inherits teacher's verbose output style. Fix: filter by length; hybrid loss with explicit length penalty.

21.7 Reward / quality regression

Distilled student worse on real metrics than soft-target loss suggests. Fix: hold-out human eval; check for teacher artifacts; retrain on cleaner data.

21.8 Distillation of unsafe behaviors

Teacher's bugs / unsafe outputs distilled into student. Fix: filter teacher data through safety classifier; constitutional distillation.

21.9 Tokenizer mismatch

Different tokenizers → token-level KL impossible. Fix: sequence-KD only; aligned tokenizer; ULD-style decoder alignment.

22. Theory and Analysis

22.1 Why does distillation work?

• Privileged information: teacher's soft targets encode dark knowledge unavailable in hard labels.
• Regularization: soft targets are smoother; student less prone to over-fit.
• Optimization landscape: KD provides denser gradient signal.
• Implicit ensemble: matching teacher's distribution captures teacher's implicit ensemble.

22.2 Information-theoretic view

• Teacher \(T\) maximizes \(I(X; Y)\) on the training task.
• Student matching \(T\)'s distribution approximates this \(I\).
• Capacity gap = lost mutual info; KD loss measures the gap.

22.3 Born-Again paradox

A student of the same architecture as the teacher, trained with KD, often outperforms the teacher. Hypothesized causes: (a) ensemble effect across iterations, (b) regularization from soft targets, (c) reorganization of features.

22.4 Distillation as continued pretraining

Modern view: distillation \(\sim\) pretraining on a curated, smoothed version of the original task. Recipes converge with self-distillation, EMA, and SSL.

22.5 Scaling laws for distillation

Empirically: distill quality scales with (a) teacher quality, (b) student capacity, (c) distillation data size. Tighter Pareto than direct training.

23. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Hinton's KD loss with \(\tau^2\) scaling.
2. Forward KL: mass-covering; reverse KL: mode-seeking.
3. Sequence KD (teacher argmax) often as good as token-level.
4. GKD: on-policy distillation; samples from student.
5. FitNets: feature-level distillation with projection.
6. Attention transfer: match attention maps.
7. RKD: relational (pairwise distances + angles).
8. DeiT distillation token: special token for teacher signal.
9. DINO/DINOv2: self-distillation as SSL with EMA + sharpening + centering.
10. Mean Teacher: EMA-of-student as semi-supervised target.
11. EMA in diffusion: required for sample fidelity.
12. DistilBERT loss: MLM + KL + cosine.
13. TinyBERT two-stage: general distill then task distill.
14. MiniLM: distill last-layer attention only.
15. Phi family: distillation via synthetic textbook data.
16. Open-instruction pattern: filter teacher data + SFT smaller model.
17. R1-Distill series: long-CoT traces → smaller student.
18. Distill cold-start enables small-model GRPO.
19. Progressive distillation: halve steps each round.
20. Consistency Models: predict \(x_0\) from any \(t\).
21. DMD2: one-step diffusion via score matching + GAN.
22. LCM / Hyper-SD / DMD2 / Lightning are the standard SDXL/FLUX few-step distills.
23. Sparse upcycling: dense → MoE via expert replication.
24. Constitutional distillation: critique + revise + train.
25. Cascades + distillation: 70–90% cost reduction.

Appendix B: Decision Tree — "Which Distillation Method?"

1. Closed-API teacher, no logits? → Sequence-KD on teacher samples (open-instruction pattern).

2. Same architecture, smaller / faster? → Hidden-state + attention + logit KD (TinyBERT, DistilBERT).

3. Cross-architecture (Transformer → SSM, ViT → CNN)? → Relational + sequence KD (RKD); avoid feature-matching.

4. Diffusion model, want few-step inference? → LCM (medium quality), DMD2 / Hyper-SD (top quality).

5. Want to bake in long-CoT reasoning? → R1-Distill pattern: SFT on filtered long-CoT traces.

6. Want to compress without losing diversity? → Higher temperature; mixup; reverse KL avoided.

7. Multi-domain, multi-teacher? → Per-input gating or weighted ensemble target.

8. Need both small & aligned? → Constitutional distillation (critique + revise + train).

9. Vision SSL student? → DINOv2-style self-distillation with EMA target.

10. Edge / mobile deployment? → Mobile-arch student (MobileViT, MobileSAM) + KD.

Appendix C: Year-by-Year Milestones

• 2014: FitNets (Romero) — feature distillation.
• 2015: Hinton et al. — the original KD with soft targets and temperature.
• 2016–2018: Sequence-KD (Kim & Rush), Born-Again Networks (BAN), Mean Teacher, Attention Transfer.
• 2019: DistilBERT, RKD, MobileBERT.
• 2020: TinyBERT, MiniLM, DeiT (with distillation token), Reformer-distill.
• 2021: BYOL, SimSiam, DINO — self-distillation as SSL becomes a standard pre-training paradigm.
• 2022: Progressive distillation for diffusion (Salimans & Ho); DINOv2; Distil-Whisper later.
• 2023: Consistency Models, Latent Consistency Models, SDXL Turbo (LADD), DMD; Vicuna / WizardLM (open-instruction); Phi-1/1.5; GKD.
• 2024: DMD2, Hyper-SD, PCM, FLUX schnell; Phi-3 / 3.5; Gemma 2 (distill from Gemini); Distil-Whisper-v2; MobileSAM / FastSAM; ULD cross-tokenizer.
• 2025: R1-Distill series (DeepSeek) — reasoning distillation goes mainstream; Phi-4; Gemma 3; multimodal R1-distill (Vision-R1, VLM-R1 distill); cold-start + GRPO standard.
• 2026: Distillation + RL pipelines standard; cross-architecture (Transformer → Mamba); native multimodal distillation; constitutional distillation in alignment pipelines.