Pruning — Parameters, Gaussians, Tokens

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations: Three Pruning Targets
2. Parameter Pruning Foundations
3. Structured vs Unstructured Pruning
4. Modern LLM Pruning Methods
5. Hardware-Aware Sparsity (N:M)
6. Lottery Ticket Hypothesis and Theory
7. Gaussian Pruning (3DGS)
8. Token Pruning Foundations
9. ViT Token Pruning
10. VLM Token Pruning
11. KV Cache Token Eviction
12. Diffusion Token Pruning
13. Mixed Pruning Approaches
14. Pruning + Quantization Composition
15. Pruning Evaluation
16. Implementation Patterns
17. Production Recipes
18. Frontier 2025–2026
19. Failure Modes and Pitfalls
20. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "Which Pruning?"
• Appendix C: Year-by-Year Pruning Milestones

1. Foundations: Three Pruning Targets

1.1 What we mean by pruning

Removing or zeroing out part of the computational state to reduce cost while preserving quality. Three target spaces in modern AI:

1. Parameters (weights, neurons, heads, layers): classical neural-network pruning.
2. Gaussians: 3DGS-specific pruning of explicit primitives.
3. Tokens: per-input attention tokens (image patches, KV cache, sequence positions).

1.2 Why pruning matters

• Inference cost: matmul / attention scales with active params and tokens.
• Memory: smaller models / fewer tokens fit on edge.
• Bandwidth: less data movement.
• Energy: lower-precision compute uses less power.
• Many parameters / Gaussians / tokens are redundant.

1.3 The redundancy hypothesis

Trained networks have substantial redundancy. The Lottery Ticket Hypothesis (Frankle & Carbin 2018) goes further: pretrained networks contain sparse subnetworks that, trained from scratch with the right initialization, match the dense network's accuracy.

1.4 Pruning × other compression

• Pruning × quantization: compounding gains; both standard.
• Pruning × distillation: prune student from teacher.
• Pruning × KV-cache compression: token + per-token bytes.
• Pruning × MoE: structured sparsity meets active-param sparsity.

1.5 The three pruning regimes

1.6 The 2026 production reality

Key

At the frontier, every production stack uses some form of pruning. Standard composition: N:M structured sparsity (parameter) + ToMe / FastV (token) + KV cache eviction (H2O / SnapKV), on top of quantization + distillation. Compounding gains routinely push 5–20× over baseline.

2. Parameter Pruning Foundations

2.1 Magnitude pruning

The simplest baseline: zero out smallest-magnitude weights.

\[\text{prune } w_{ij} \text{ if } |w_{ij}| < \tau.\]

\(\tau\) chosen for target sparsity (e.g., 50%). Trivial; surprisingly competitive.

2.2 Gradual magnitude pruning

Iteratively prune → fine-tune → prune more. Smooth ramp from 0% to target sparsity over training.

\[s_t = s_f + (s_i - s_f)\left(1 - \frac{t - t_0}{N\Delta t}\right)^3.\]

2.3 Iterative Magnitude Pruning (IMP)

1. Train.
2. Prune smallest 20%.
3. Reset surviving weights to original init.
4. Retrain.
5. Repeat.

Source of the Lottery Ticket Hypothesis.

2.4 Movement pruning (Sanh et al. 2020)

Prune based on movement-of-magnitude during fine-tuning, not absolute magnitude:

\[\text{score} = \text{sign}(w \cdot \nabla_w L) \cdot |w|.\]

Better than magnitude for transfer learning.

2.5 ℓ0 regularization

Use a continuous relaxation of \(\ell_0\) (Hard Concrete distribution) to learn binary masks. Soft + train end-to-end.

2.6 Pruning at initialization (PaI)

Prune before training, based on initialization information:

• SNIP: connection-sensitivity based on \(|g \odot w|\).
• GraSP: gradient-signal-preservation; saves higher-impact weights.
• SynFlow: synaptic flow conservation; iteratively scores.

Cheap; sometimes lossy at high sparsity.

2.7 Random pruning baseline

Random masks at the same sparsity level.

Surprisingly competitive at moderate sparsity (< 50%) for overparameterized networks. A useful baseline for "does my method actually do anything?"

2.8 Sensitivity vs uniformity

• Uniform sparsity: same rate per layer.
• Layer-wise sensitivity: different rate per layer based on quality drop.
• Most production: sensitivity-aware.

3. Structured vs Unstructured Pruning

3.1 Unstructured pruning

Remove individual weights. Maximum compression at given quality. Doesn't accelerate inference on dense GPUs unless special kernels are used.

3.2 Structured pruning

Remove whole rows / columns / channels / heads. Loss of compression but immediate inference speedup.

3.3 Granularity hierarchy

• Weight (unstructured).
• 2:4 / N:M block-wise.
• Channel.
• Filter.
• Head (attention).
• Layer.
• Block (sub-network).

3.4 Channel pruning

Remove entire conv channels / linear rows / cols. Reduces FLOPs proportionally. Standard for CNN compression.

3.5 Head pruning (attention)

Remove unimportant attention heads. Many heads are redundant in pretrained models.

• Voita et al. (2019): identify "confident" heads; keep them.
• Most modern LLMs: head pruning rarely yields large gains because GQA / MQA already shares.

3.6 Layer pruning

Remove whole Transformer layers.

• ShortGPT: Block Importance metric prunes least-important layers.
• LLM-Streamline: layer-skip with distillation recovery.
• SLEB, FuseLLM: variations.

Surprisingly effective: 7B Llama can lose ~25% layers with <5% quality loss.

3.7 Block / sub-network pruning

Prune sequence of contiguous layers. Used in mobile deployment.

3.8 Comparison

4. Modern LLM Pruning Methods

4.1 SparseGPT (Frantar & Alistarh 2023)

One-shot post-training pruning via approximate Hessian:

\[\Delta W_{q,\,j>k} = -\frac{(W_q - W)_{:,k}\,\cdot\,[H^{-1}]_{k,\,j>k}}{[H^{-1}]_{kk}}, \quad H = 2XX^\top.\]

Layer-by-layer; updates remaining weights to compensate for pruned. INT4 + 50% sparsity feasible.

4.2 Wanda (Sun et al. 2023)

Pruning by Weights And Activations:

\[\text{score}_{ij} = |w_{ij}| \cdot \|x_j\|_2.\]

Activation-aware; cheap (no Hessian); often competitive with SparseGPT.

4.3 LLM-Pruner

Structured pruning for LLMs via dependency-graph analysis. Removes whole channels / heads.

4.4 ShortGPT (Men et al. 2024)

Layer-pruning for LLMs:

• Block Importance (BI) metric: cosine similarity between layer input and output.
• Low BI = layer doesn't change activation much → prune.
• 7B Llama 2 to ~5.6B: 2–5% quality loss.

4.5 LLM-Streamline

Layer-pruning with distillation recovery. Better quality than naive layer removal at same compression.

4.6 SliceGPT (Microsoft 2024)

Apply orthogonal transformations to W to isolate "unimportant" principal components; slice them off.

• Reduces hidden dim by 25–30% without accuracy loss.
• Hardware-friendly (smaller dense matmul).

4.7 Compresso, FlexGen, FastFormer

Various LLM compression approaches combining pruning + quantization + distillation.

4.8 Pruning + LoRA recovery

1. Apply pruning method.
2. Add LoRA adapters.
3. Train LoRA briefly to recover quality.
4. Cheap; effective.

4.9 Comparison table

4.10 Practical recipe (LLM)

1. Calibration set (~128 samples).
2. Wanda or SparseGPT for unstructured / 2:4.
3. Or ShortGPT for layer pruning.
4. LoRA recovery ~1M tokens.
5. Combine with INT4 quantization.

★ 2026 SOTA update — Minitron: prune plus distill

• Minitron: NVIDIA prune-and-distill recipe; joint width (hidden/attention/MLP) and depth pruning of a large model, then knowledge distillation on ~100B tokens recovers quality with far less compute than pretraining. Produced Mistral-NeMo-Minitron-8B from 12B and Llama-3.1-Minitron-4B from 8B; the standard modern structured-pruning + recovery pipeline.

5. Hardware-Aware Sparsity (N:M)

5.1 The 2:4 sparsity pattern

2 of every 4 weights zero. 50% sparse, hardware-supported.

5.2 Hopper / Blackwell support

• H100 / B200: native 2:4 sparse matmul at ~2× throughput vs dense.
• Combined with FP8: ~4× effective speedup vs BF16 dense.
• Critical for production LLM serving.

5.3 NVIDIA Sparse Tensor Cores

TensorRT-LLM and CUTLASS provide kernels for 2:4. Compose with quantization (FP8 / INT8 / INT4).

5.4 4:8 and N:M generalizations

• 4:8 (50% sparse, larger granularity): less compression flexibility.
• N:M (general pattern): slowly emerging hardware support.

5.5 Training with 2:4

• Apply 2:4 mask during training (mask + dense matmul).
• Or use sparse tensor cores during training (Hopper supports).
• Quality at par with dense for most tasks.

5.6 Fine-tune to 2:4

Take pre-trained dense model; apply 2:4 + brief fine-tune. Industry-standard for LLM compression.

5.7 Memory savings

2:4 sparse weights stored as half values + index. 50% memory reduction, in addition to quantization.

5.8 Combining with quantization

Standard: 2:4 sparse + FP8/INT4 weights. NVIDIA's recommended LLM serving recipe.

★ 2026 SOTA update — MaskLLM: learnable N:M sparsity

• MaskLLM: learns the 2:4 / N:M mask itself as a distribution via Gumbel-Softmax (frozen or trainable weights), instead of using a fixed magnitude/Hessian criterion. End-to-end trainable, masks transfer across domains; large PPL gains over SparseGPT/Wanda at 2:4 (e.g. 6.72 vs >10 PPL on LLaMA-2). NeurIPS 2024 spotlight.

6. Lottery Ticket Hypothesis and Theory

6.1 The lottery ticket hypothesis (Frankle & Carbin 2018)

A randomly-initialized dense network contains a sparse sub-network ("winning ticket") that, trained in isolation, can match the full network's accuracy.

6.2 IMP (Iterative Magnitude Pruning) for tickets

1. Train dense network.
2. Prune smallest-magnitude weights.
3. Reset to original initialization.
4. Train again.
5. Repeat.

Discovers "winning tickets."

6.3 Strong Lottery Ticket

Even before any training, a random network contains a sparse subnetwork that achieves good accuracy with the right mask. Theoretical result; impractical to find directly.

6.4 Why does it work?

Open question. Hypotheses:

• Over-parameterization smooths optimization.
• Pruned subnetwork has the right inductive bias.
• Training implicitly identifies critical paths.

6.5 Lottery tickets at scale

Mostly demonstrated on smaller networks (CNNs, small Transformers). Hard to find true tickets at LLM scale; SparseGPT / Wanda are practical alternatives.

6.6 Pruning theory

• Optimal Brain Damage (OBD): Hessian-based importance.
• Optimal Brain Surgeon (OBS): extension with weight updates.
• SparseGPT is essentially per-layer OBS.

7. Gaussian Pruning (3DGS)

7.1 Why Gaussians need pruning

3D Gaussian Splatting scenes can have 1M–10M+ Gaussians. Memory + render speed scale with count. Many are redundant after optimization.

7.2 Built-in 3DGS density control

Original 3DGS already has pruning:

• Prune Gaussians with \(\alpha < \tau\) (\(\tau \approx 0.005\)).
• Prune Gaussians with huge screen-space size (likely floaters).
• Periodic opacity reset → prune redundant.

7.3 Post-training Gaussian pruning

After training, additional pruning for deployment:

• Score Gaussians by importance.
• Drop bottom-k% by score.
• Optional: fine-tune remaining Gaussians.

7.4 LightGaussian (Fan et al. 2024)

• Importance score: per-Gaussian contribution to rendered images.
• Cumulative Gaussian importance via differentiable rendering.
• Prune low-importance.
• Combined with INT8 quantization + SH coefficient distillation.
• ~10× smaller storage; comparable quality.

7.5 CompGS (Compact Gaussian Splatting)

Codebook-based compression:

• Cluster Gaussians by feature similarity.
• Per-cluster shared parameters.
• Pruning by cluster importance.

7.6 Compact3D, Mini-Splatting, RDOGS

Various approaches:

• Mini-Splatting: progressive simplification with quality target.
• RDOGS: rate-distortion-optimized; minimize bits + maximize quality.
• Compact3D: vector quantization of Gaussian parameters.

7.7 Importance scoring

• Opacity: prune low-\(\alpha\) Gaussians.
• Gradient-based: how much a Gaussian's removal affects rendering loss.
• Pixel coverage: how many pixels does this Gaussian touch?
• Visibility: per-view visibility frequency.
• Information content: Shannon-style.

7.8 Joint pruning + quantization

• Prune position FP16, scales FP16 (precision matters).
• Quantize opacity INT8.
• Quantize SH coefficients INT8 / codebook.
• Combine with pruning for ~20× compression.

7.9 Web deployment of 3DGS

For browser / mobile delivery:

• LightGaussian / CompGS for compression.
• Streaming chunks (city-scale).
• Level-of-Detail (Hierarchical 3DGS, Octree-GS).

7.10 Production patterns

• Train: ~5M Gaussians.
• Light prune: drop low-\(\alpha\) during training (built-in).
• Heavy prune: LightGaussian at deploy.
• Quantize: INT8 / codebook on remaining.
• Stream: web-friendly chunks.

★ 2026 SOTA update — MaskGaussian: probabilistic Gaussian pruning

• MaskGaussian: treats each Gaussian as probabilistic (existence probability) rather than hard-removing it; masked-rasterization lets unsampled Gaussians still receive gradients and revive. Prunes aggressively while keeping PSNR, cutting count, memory, and training time; works from-scratch or as post-training refinement. CVPR 2025.

8. Token Pruning Foundations

8.1 Why token pruning

Attention scales as \(O(n^2)\) in sequence length. Reducing \(n\) by \(r\) gives \(r^2\) speedup in attention. For long sequences (long context, video, high-res images), this is enormous.

8.2 Three settings

1. Image / patch tokens (ViT): prune unimportant patches.
2. Video / frame tokens: drop redundant frame patches.
3. KV cache (LLM decode): evict old / unimportant tokens.

8.3 Decision criteria

• Attention magnitude (which tokens get attended to most).
• Similarity (merge similar tokens).
• Position (keep recent / sink).
• Learned scoring.

8.4 Pre- vs post-attention

• Pre-attention: drop tokens before attention; saves attention compute.
• Post-attention: drop after; saves downstream FFN compute.
• Most modern methods are pre-attention.

8.5 Drop vs merge

• Drop: remove token; lose its information.
• Merge: combine multiple tokens into one (averaging features); preserve some information.
• Merging usually preserves quality better.

9. ViT Token Pruning

9.1 DynamicViT (Rao et al. 2021)

Per-layer learned token-importance predictor; drop low-importance tokens. Differentiable via Gumbel-Softmax.

Dynamic: different tokens dropped per input.

9.2 A-ViT (Adaptive)

Per-token early-exit: tokens that "finish" early skip remaining layers. Adaptive compute per token.

9.3 IA-RED, EViT

Various per-layer dropping schemes with different scoring. Accuracy-speed Pareto.

9.4 ToMe (Token Merging, Bolya & Hoffman 2023)

• Bipartite soft-matching: split tokens into sets A, B.
• For each \(a \in A\), find most-similar \(b \in B\).
• Merge top-r pairs (weighted average of features).
• Run subsequent layers on reduced set.

No training required; drop-in for ViT. ~2× speedup with < 1 accuracy point loss.

9.5 ToMe in practice

• ViT-L at \(384^2\): 729 tokens → ~300 tokens via aggressive merging.
• Per-layer r: 8–16 merges per layer.
• Combine with quantization for further speedup.

9.6 ToMeSD (token merging for Stable Diffusion)

Apply ToMe inside SD U-Net's self-attention. Unmerge before residual to preserve spatial info. ~1.5× inference speedup at minor quality loss.

9.7 ToMe variants

• ToFu (Token Fusion): combines ToMe with token re-distribution.
• ATS (Adaptive Token Sampling): importance-weighted sampling.
• SiT: similarity-based + redundancy reduction.

9.8 Why merging beats dropping

Drop loses information; merge preserves a weighted average. Empirically, merging gives smoother quality-speed Pareto.

10. VLM Token Pruning

10.1 The VLM token problem

A VLM with high-res tile-encoded image can have thousands of vision tokens. After early layers, the LLM rarely attends to most of them. Pruning vision tokens unlocks massive speedup with minimal quality loss.

10.2 FastV (Chen et al. 2024)

Insight: in VLMs, after layer K (K ~2–4), most attention from text tokens to vision tokens is concentrated on a few. Drop the rest after layer K.

Drop vision-tokens with

\[\sum_t a_{t\to i} < \tau \quad \text{after layer } K.\]

• ~50% FLOP reduction.
• Minimal quality loss (often < 1%).
• No training required.

10.3 VTW (Visual Token Withdrawal)

Withdraw vision tokens entirely after layer K; LLM operates only on text + summary. Even more aggressive than FastV.

10.4 LLaVA-PruMerge

Token clustering + merging for LLaVA-class VLMs. Reduces vision tokens by 14×.

10.5 SparseVLM

Per-layer adaptive vision token sparsity. Different sparsity per layer.

10.6 VisionZip

Compress vision tokens before LLM input. Tokenizer-side reduction.

10.7 TokenPacker

Hierarchical packing of vision tokens. Multi-scale.

10.8 Production VLM pattern (2026)

1. Vision encoder produces ~1024 tokens.
2. Pixel unshuffle / packing to ~256.
3. LLM forward; FastV-style drop after layer 2–4.
4. Effective vision tokens: ~50 during late layers.
5. Speedup: 3–4× vs no pruning; <1% quality loss.

10.9 Quality vs speed Pareto

• Aggressive (drop 90%+): noticeable degradation on dense visual tasks.
• Moderate (drop 50%): negligible loss.
• Light (drop 25%): no loss; modest speedup.

★ 2026 SOTA update — DivPrune: diversity-based token pruning

• DivPrune: training-free VLM visual-token pruning framed as a Max-Min Diversity Problem — select the subset maximizing mutual diversity rather than attention/importance, minimizing redundancy among kept tokens. SOTA over 16 image/video benchmarks at high pruning ratios, cutting latency and GPU memory. CVPR 2025; a FastV/VisionZip successor.

11. KV Cache Token Eviction

11.1 The KV cache pruning problem

Long-context decode reads the entire KV cache per step. Memory + bandwidth scale linearly with n.

11.2 H2O (Heavy-Hitter Oracle, Zhang et al. 2023)

Insight: a small fraction of tokens ("heavy hitters") receive most of the attention mass.

• Compute per-token cumulative attention score during decode.
• Keep top-k heavy hitters + sliding window of recent tokens.
• Evict the rest.
• ~5× cache reduction; ~1% perplexity loss.

11.3 Scissorhands

Persistent-importance-based eviction. Tokens unimportant once tend to remain unimportant. Cheaper to compute than H2O.

11.4 SnapKV (Li et al. 2024)

Prefill-time compression for long-prompt scenarios:

• During prefill, examine the last few tokens' attention to all prior tokens.
• Pool importance scores; keep top-k.
• Discard rest before decode.
• E.g., 100K → 4K tokens.
• Minimal quality loss on QA / summarization.

11.5 Pyramid KV (Cai et al. 2024)

Different layers retain different amounts: lower layers more, deeper layers fewer. Insight: deeper layers concentrate attention on fewer tokens. Memory savings without uniform pressure.

11.6 StreamingLLM (Xiao et al.)

Sliding window + sink tokens: keep first k ~4 "sink" tokens always + sliding window of w recent. Effective unbounded streaming with constant cache.

11.7 Quest (Tang et al. 2024)

Query-aware retrieval: at each query, retrieve only the top-k most-relevant past KV blocks. Combines paged KV with importance-based retrieval. Long-context speedup with quality preserved.

11.8 TokenButler

Predicts query-aware token importance and prunes before attention.

11.9 FastV (KV-cache application)

Drop vision-token KV after early layers in VLM decoding. Discussed in VLM section.

11.10 Adaptive eviction policies

• Per-layer different keep-fractions.
• Per-head different policies.
• Schedule-based (more aggressive late in generation).

11.11 Comparison table

★ 2026 SOTA update — CAKE: layer-aware KV eviction

• CAKE: cascading, layer-adaptive KV-cache eviction ('cake-slicing'). Allocates per-layer cache budgets from attention dynamics (spatial dispersion + temporal shift) and evicts in a cascading manner during prefill, using an eviction indicator that tracks shifting token importance. Beats uniform-budget SnapKV/PyramidKV at equal memory. ICLR 2025.

12. Diffusion Token Pruning

12.1 ToMe-SD recap

Apply ToMe in SD's self-attention layers. Merge similar spatial tokens; unmerge before residual. ~1.5× speedup.

12.2 ToMe for video diffusion

Apply ToMe across temporal + spatial axes. Especially effective for video where adjacent frames have high redundancy.

12.3 DyDiT (Dynamic DiT)

Per-token-per-step adaptive compute:

• Different timesteps need different token counts.
• Different tokens have different importance.
• Learned routing: each token decides whether to skip current denoising block.

12.4 Block-cache patterns (DeepCache, TGATE)

Not strictly pruning, but conceptually related: skip recomputation of stable feature maps across denoising steps.

12.5 Token-aware step distillation

Distill samples that handle token pruning gracefully. Combines distillation + pruning.

12.6 Patch / region pruning for high-res

For high-res image generation: drop patches outside object regions; saves compute on background. Used in some commercial pipelines.

★ 2026 SOTA update — TinyFusion: diffusion depth pruning

• TinyFusion: learnable depth (layer) pruning for Diffusion Transformers. Differentiable layer-mask sampling co-optimized with a simulated fine-tuning step so the pruned DiT is highly recoverable — explicitly optimizes post-fine-tune performance, not immediate loss. ~2x speedup with quality restored. CVPR 2025 highlight; DiT structured pruning beyond feature-caching.

13. Mixed Pruning Approaches

13.1 Token + parameter

• Token pruning at attention level.
• Parameter pruning at weight level.
• Compose for compounding speedup.

13.2 Gaussian + activation

• Prune Gaussians (LightGaussian).
• Prune low-importance features in subsequent neural-shading.

13.3 Layer + token (extreme compression)

• Drop every other Transformer layer (layer pruning).
• Token-merge in remaining layers.
• 5–10× inference speedup.

13.4 MoE + sparsity

• MoE: per-token expert routing (already activates only k of N).
• Add 2:4 sparsity within active experts.
• Effective compute: \(k/N \cdot 0.5\) of dense.

13.5 Dynamic at inference

Per-input adaptive compute:

• Easy queries: aggressive pruning, fast.
• Hard queries: less pruning, full quality.
• Rationale: inference-time scaling laws.

13.6 Cascades + pruning

• Tier 0: heavily pruned model handles easy queries.
• Tier 1: lightly pruned for medium.
• Tier 2: full model for hard.

★ 2026 SOTA update — STUN: MoE expert pruning

• STUN: structured-then-unstructured pruning for Mixture-of-Experts. First drops redundant whole experts (expert-level structured pruning via cheap behavioral-similarity scoring, no calibration), then applies unstructured/N:M sparsity inside surviving experts; the two compound and beat either alone, scaling to very large MoEs (e.g. Snowflake Arctic, DeepSeek-scale). ACL 2025.

14. Pruning + Quantization Composition

14.1 The standard production stack

1. Train dense FP32 / BF16.
2. Apply 2:4 sparsity (Wanda / SparseGPT).
3. Quantize to INT4 / FP8 (GPTQ / AWQ).
4. Brief LoRA recovery.
5. Deploy.

Compounding: 2× from sparsity + 4× from quantization = 8× memory + speedup.

14.2 Order matters

• Prune first, then quantize: standard.
• Quantize first, then prune: rare.
• Joint: research direction.

14.3 Mixed precision + sparsity

• Important channels: FP16 dense.
• Less important: INT4 + 2:4 sparse.
• Per-layer / per-channel mixing.

14.4 NVIDIA's recommended LLM serving recipe

• Base: Llama-3-70B.
• 2:4 sparse weights.
• FP8 weights and activations.
• TensorRT-LLM.
• Result: ~4× throughput vs BF16 dense.

15. Pruning Evaluation

15.1 Metrics to report

• Compression rate (parameters or memory).
• Inference speedup (FLOPs or wall-clock).
• Quality drop (per-task accuracy).
• Energy consumption (if claimed).

15.2 Pareto fronts

Plot quality-vs-compression. Compare methods at matched compression. Report multiple operating points.

15.3 Per-layer sensitivity analysis

For each layer:

• Prune only that layer.
• Measure quality drop.
• Higher drop = more sensitive layer.
• Allocate sparsity budget accordingly.

15.4 Ablation methodology

• Compare to magnitude / random baseline.
• Same calibration data, training budget.
• Held-out eval (avoid contamination from calibration).
• Multiple seeds / random masks.

15.5 Common evaluation pitfalls

• Reporting only aggregate metric (hides per-task regression).
• Ignoring inference speedup (compression \(\ne\) speedup).
• Not comparing to baselines.
• Cherry-picked hyperparameters.

15.6 Eval suites

• LLM: MMLU, GSM8K, HellaSwag, ARC, perplexity on WikiText.
• ViT: ImageNet top-1.
• VLM: MMMU, MathVista, ChartQA.
• 3DGS: PSNR / SSIM / LPIPS on novel views; storage size.

16. Implementation Patterns

16.1 Mask-based pruning

• Apply boolean mask on weights / tokens.
• Standard PyTorch implementation.
• No actual sparse computation; just zeros.
• For experimentation, not deployment.

16.2 Hard removal

• Physically remove pruned elements.
• Requires re-architecting the module.
• Standard for structured pruning.
• Production-friendly.

16.3 Sparse kernels

• cuSPARSE: NVIDIA's sparse linalg.
• Sparse Tensor Cores (Hopper / Blackwell): 2:4 sparse matmul.
• Triton sparse kernels: custom; flexible.
• TVM sparse: cross-platform.

16.4 Frameworks supporting structured sparsity

• PyTorch + torch.sparse.
• NVIDIA Apex (legacy).
• TensorRT-LLM (production).
• NVIDIA Model Optimizer (toolkit).

16.5 Triton kernels for ToMe / FastV

• Custom Triton or Ad-hoc PyTorch for token operations.
• Native support emerging in vLLM / SGLang.

16.6 Per-layer rate scheduling

• Uniform: same rate per layer.
• Sensitivity-based: per-layer determined by analysis.
• Adaptive: learned during training.
• Pyramid: increasing / decreasing with depth.

17. Production Recipes

17.1 LLM serving (cloud)

• 2:4 sparse + FP8 + paged KV + H2O / SnapKV.
• Multi-LoRA serving.
• 4–8× throughput vs baseline BF16 dense.
• NVIDIA TensorRT-LLM standard.

17.2 LLM serving (consumer GPU)

• Q4 (GGUF) + StreamingLLM-style sliding window for long context.
• llama.cpp / Unsloth with optimizations.
• Memory-tight scenarios: aggressive layer / token pruning.

17.3 VLM serving

• FastV after layer 2–4: drop most vision tokens.
• Pixel unshuffle for compression.
• Quantize LLM backbone.
• 3× speedup typical.

17.4 Diffusion serving

• ToMe-SD or Block-cache.
• Step distillation (4–8 NFEs).
• INT4 weights + FP8 activations.
• Combined: 10× vs naive baseline.

17.5 3DGS serving

• LightGaussian / CompGS for compression.
• INT8 / codebook quantization.
• Streaming chunks for city-scale.
• Web-friendly: ~10s of MB per scene.

17.6 Edge deployment

• Aggressive pruning (50%+ params).
• INT4 / INT8 quantization.
• Layer pruning if possible.
• Distillation recovery.
• Mobile NPU (ANE / Hexagon) compatibility.

17.7 Reasoning model serving

• Long generation (10k+ tokens).
• KV-cache eviction critical (H2O / Pyramid KV).
• StreamingLLM-style sinks.
• Speculative decoding integration.

18. Frontier 2025–2026

18.1 Trends

• N:M structured sparsity standard for LLM serving.
• FastV / vision-token pruning standard for VLM.
• H2O / SnapKV / Pyramid KV widespread for long context.
• LightGaussian-class compression for 3DGS web deployment.
• Combined pruning + quantization + distillation as the production default.

18.2 Adaptive / dynamic pruning

Per-input adaptive compute:

• Mixture-of-Depths (MoD): per-token layer routing.
• Mixture-of-Recursions (MoR): adaptive iteration count.
• Dynamic KV eviction: per-query.
• Inference-time scaling: more compute for hard queries.

18.3 Native sparsity training

DeepSeek NSA (Native Sparse Attention): train with sparse attention from scratch. End-to-end optimized; bypasses lossy post-hoc pruning.

18.4 Token-aware MoE pruning

• Per-expert importance scoring.
• Cold-expert pruning + variable bits.
• Combined with 2:4 sparsity.

18.5 Adaptive Gaussian generation / pruning

• Generative 3DGS models that produce already-compact representations (Trellis-class).
• No post-hoc pruning needed.

18.6 Open research directions

• Theoretical: when pruning preserves vs degrades capability.
• Pruning-aware training (vs post-hoc).
• Per-task / per-domain pruning.
• Automatic per-layer sensitivity analysis.
• Pruning + reasoning-RL interaction.

★ 2026 SOTA update — TEAL: training-free activation sparsity

• TEAL: training-free, magnitude-based activation sparsity applied to hidden states model-wide (works on modern SwiGLU models, not just ReLU). 40-50% activation sparsity with minimal quality loss and up to 1.8x decode speedup by skipping matmul rows; composes with weight quantization. Complements DeepSeek-NSA-style native sparsity from the input side.

19. Failure Modes and Pitfalls

19.1 Aggressive pruning regressions

• Quality OK on average; specific tasks degrade.
• Need per-task / per-segment eval.

19.2 Calibration overfit

Pruning method optimizes for calibration distribution; fails out-of-domain.

19.3 Layer pruning + reasoning

Layer pruning often hurts long-form reasoning more than short-form. Test on chain-of-thought tasks.

19.4 KV eviction + long context

H2O-style eviction can hurt very-long-context tasks where retrieval depends on tokens evicted.

19.5 ToMe + spatial information

Aggressive merging loses spatial detail; image quality regression.

19.6 Pruning + safety regression

Pruning can quietly degrade refusal / safety behavior. Always test post-prune on safety eval.

19.7 Sparsity-quantization interaction

Combined sparsity + INT4 quantization can have non-additive quality loss. Joint tuning needed.

19.8 Catastrophic regression at high sparsity

Most methods graceful up to 50%, then steep drop. Test multiple operating points.

20. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Three pruning targets: parameters / Gaussians / tokens.
2. Magnitude pruning: prune \(|w| < \tau\); surprisingly competitive baseline.
3. Lottery Ticket Hypothesis: sparse subnets in dense networks.
4. IMP: prune → reset → retrain.
5. Movement pruning: \(\text{sign}(w \cdot \nabla L) \cdot |w|\).
6. Pruning-at-init (SNIP / GraSP / SynFlow).
7. Random pruning is a strong baseline.
8. 2:4 N:M sparsity: hardware-supported on H100 / Blackwell, 2× matmul.
9. SparseGPT: layer-wise OBS with Hessian.
10. Wanda: \(|w| \cdot \|x\|\); cheap activation-aware.
11. ShortGPT: layer pruning via Block Importance.
12. SliceGPT: orthogonal slicing of hidden dims.
13. LLM-Pruner: structured channel pruning.
14. Pruning + LoRA recovery: standard recipe.
15. 3DGS density control: clone / split / prune / opacity reset.
16. LightGaussian: importance-based pruning + INT8 + SH distill.
17. CompGS: codebook compression for 3DGS.
18. ToMe (Bolya & Hoffman): bipartite soft-matching merge.
19. ToMeSD: ToMe in SD U-Net.
20. DynamicViT: learned per-token importance.
21. FastV: drop vision tokens after layer 2–4 in VLM.
22. H2O: heavy hitters + recent for KV eviction.
23. SnapKV: prefill-time importance pooling.
24. Pyramid KV: deeper layers retain fewer tokens.
25. StreamingLLM: sinks + sliding window for unbounded streaming.

Appendix B: Decision Tree — "Which Pruning?"

1. LLM weight pruning, post-training? → SparseGPT or Wanda + 2:4 + FP8 + LoRA recovery.
2. LLM layer-level extreme compression? → ShortGPT or LLM-Streamline.
3. LLM hidden-dim compression? → SliceGPT.
4. LLM long-context decode (KV cache)? → H2O or Pyramid KV or SnapKV (prefill).
5. LLM streaming? → StreamingLLM (sinks + window).
6. ViT inference acceleration? → ToMe (drop-in, no training).
7. VLM inference acceleration? → FastV (drop vision tokens after layer K).
8. Diffusion image gen? → ToMe-SD + Block-cache + distill.
9. 3DGS deployment compression? → LightGaussian or CompGS.
10. Edge / mobile? → Combined: layer prune + 2:4 / unstructured prune + INT4 quant + distill.
11. Hardware-friendly speedup (Hopper / Blackwell)? → 2:4 sparsity + FP8 via TensorRT-LLM.
12. Adaptive per-input compute? → Mixture-of-Depths or Quest (KV) or DyDiT (diffusion).

Appendix C: Year-by-Year Pruning Milestones

• 1989: Optimal Brain Damage (LeCun et al.).
• 1993: Optimal Brain Surgeon (Hassibi & Stork).
• 2015: Deep Compression (Han et al.) — prune + quantize + Huffman.
• 2018: Lottery Ticket Hypothesis (Frankle & Carbin); SNIP.
• 2019–2020: GraSP, SynFlow; movement pruning; structured pruning for BERT.
• 2020: NVIDIA 2:4 sparsity announced (Ampere).
• 2021–2022: DynamicViT; A-ViT; structured pruning for ViT.
• 2023 (early): ToMe (Bolya & Hoffman); ToMeSD.
• 2023 (mid): SparseGPT; Wanda; H2O; LLM-Pruner.
• 2024: Scissorhands; StreamingLLM (sinks + window); SnapKV; Pyramid KV; FastV (vision-token drop); ShortGPT (layer pruning); SliceGPT (Microsoft); LightGaussian; CompGS; Compact3D; Mini-Splatting; RDOGS; Mixture-of-Depths (Raposo et al.); Quest (query-aware KV).
• 2025: Native Sparse Attention (DeepSeek NSA) — end-to-end sparse training; LLM-Streamline; combined pruning + FP8 + 2:4 standard for LLM serving; FastV-style drop standard in production VLMs.
• 2026: pruning + quantization + distillation as the unified production default; adaptive per-input compute mainstream; native sparse pretraining viable; 3DGS web deployment with 20× compression normal.