Scaling Laws — Technologies & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. What Are Scaling Laws?
2. Pre-Chinchilla: Hestness and Kaplan
3. Chinchilla Scaling Laws
4. Beyond Chinchilla: Over-Training
5. Per-Axis Scaling
6. Inference-Time Scaling Laws
7. MoE Scaling Laws
8. Vision Scaling Laws
9. Multimodal Scaling Laws
10. Reasoning / RL Scaling
11. Long-Context Scaling
12. Distillation Scaling
13. Data Scaling Laws
14. Hyperparameter Scaling: µP
15. Capability Emergence
16. Compute and Cost Scaling
17. Practical Allocation Decisions
18. Diffusion / Generative Scaling
19. Robotics / VLA Scaling
20. Frontier 2025–2026 Trends
21. Production Stack 2026

• Appendix A: Twenty-Five Things to Know
• Appendix B: Decision Tree — "How to Scale?"
• Appendix C: Year-by-Year Scaling Milestones

1. What Are Scaling Laws?

1.1 The basic claim

Empirical relationships between model performance and the inputs that shape it (parameters, data, compute).

Performance is typically a smooth power-law function:

\[L(N) = L_\infty + \left(\frac{N_c}{N}\right)^{\alpha},\]

where \(L\) is loss, \(N\) is parameters, \(L_\infty\) is irreducible loss, \(\alpha\) is the scaling exponent.

1.2 Why scaling laws matter

• Predict performance of frontier-scale models from smaller experiments.
• Allocate budget between parameters / data / compute.
• Roadmap planning: years of investment in next-generation runs.
• Research direction: where is improvement most efficient?
• Settle architectural debates: ablations at scale.

1.3 The three primary scaling axes

1. Parameters (N): model size.
2. Data (D): tokens / examples.
3. Compute (\(C\)): FLOPs.
4. Plus emerging: Inference compute.

1.4 Power law form

\[L = A \cdot X^{-\alpha} + L_\infty.\]

\(\alpha\): scaling exponent (typically 0.05–0.5). \(L_\infty\): irreducible loss (Bayes loss).

1.5 Why power laws?

Empirical observation; exact theoretical justification disputed. Possible mechanisms:

• Universal complexity of natural data.
• Hierarchy of features at different scales.
• Heuristic: simplest nontrivial smooth fit.

1.6 The 2026 lesson

Key

Frontier model design is now driven by scaling-law math: Chinchilla-optimal for training compute, Snell-style inference-compute scaling for capability, µP for hyperparameter transfer, and native sparse / MoE for active-vs-total parameter trade-offs. Every principal-level model design decision is grounded in these.

2. Pre-Chinchilla: Hestness and Kaplan

2.1 Hestness et al. (2017)

Earliest systematic study. Power-law improvement of test loss with dataset size across translation, character / word LMs, image classification, speech.

\[\epsilon(D) \propto D^{-\beta}.\]

Established that scaling laws are universal across domains.

2.2 Kaplan et al. (OpenAI 2020)

Foundational LLM scaling-laws paper. Established for autoregressive Transformers:

Key

\[L(N) = (N_c/N)^{\alpha_N}, \quad L(D) = (D_c/D)^{\alpha_D}, \quad L(C) = (C_c/C)^{\alpha_C}.\]

With \(\alpha_N \approx 0.076\), \(\alpha_D \approx 0.095\), \(\alpha_C \approx 0.057\) (the famous numbers).

2.3 The Kaplan compute-optimal recipe

For fixed compute \(C\):

\[N_{\mathrm{opt}}(C) \propto C^{0.73}, \quad D_{\mathrm{opt}}(C) \propto C^{0.27}.\]

Implication: scale parameters faster than data. "GPT-3 spec."

2.4 Why Kaplan's recipe was followed for years

• Cleanly fit small-scale experiments.
• Predicted GPT-3-class behavior (175B, 300B tokens).
• Drove "larger model, less data" for 2020–2021.

2.5 Limitations of Kaplan

• Used fixed LR / batch sizes (didn't account for hyperparameter tuning across scales).
• Smaller-scale fit; large models showed deviations.
• Implicit assumption: train to convergence.

2.6 Other early scaling work

• Hoffmann et al. (DeepMind 2020): pre-Chinchilla.
• Henighan et al. (OpenAI): scaling for image / video / math models; cross-modal generality.
• Bahri et al.: theoretical exposition.

3. Chinchilla Scaling Laws

3.1 Hoffmann et al. (DeepMind 2022)

Re-ran scaling experiments with proper hyperparameter tuning (warmup, batch size, LR schedule).

Found Kaplan was wrong.

3.2 The corrected formula

Key

For compute-optimal training:

\[N_{\mathrm{opt}}(C) \propto C^{0.5}, \quad D_{\mathrm{opt}}(C) \propto C^{0.5}.\]

Scale parameters and data equally.

3.3 The 20 tokens / parameter rule

At compute-optimal allocation:

\[D_{\mathrm{opt}} \approx 20 \cdot N_{\mathrm{opt}}.\]

A 70B-parameter model should be trained on ~1.4T tokens for compute-optimal loss.

3.4 The Chinchilla model itself

70B parameters, 1.4T tokens. Same compute as Gopher (280B, 300B tokens) but outperformed Gopher by significant margin.

3.5 Implications

• GPT-3, Gopher, etc., were under-trained.
• Don't make models too big for available data.
• "Less is more" for parameters at fixed compute.
• Drove the wave of smaller, better-trained models (Llama 1 family).

3.6 Three approaches to fit scaling

Hoffmann et al. used three independent methods:

1. Fix model size, vary tokens.
2. Fix tokens, vary model size (IsoFLOP).
3. Parametric fit to all data.

All agreed: \(\alpha_N \approx \alpha_D \approx 0.5\).

3.7 The Chinchilla loss formula

\[L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}},\]

\(E \approx 1.69\) (irreducible), \(A \approx 406\), \(B \approx 411\), \(\alpha \approx 0.34\), \(\beta \approx 0.28\).

3.8 Compute approximation

\(C \approx 6ND\). Plug into Chinchilla loss; minimize over \((N, D)\) subject to \(C\). Result: \(N_{\mathrm{opt}} \propto C^{0.5}\), \(D_{\mathrm{opt}} \propto C^{0.5}\).

3.9 Subsequent corrections

• Refit by Besiroglu et al. (2024): suggested \(D/N \approx 22\).
• Hoffmann et al. Chinchilla constants slightly off.
• Practical recipes use \(D/N \in [15, 25]\).

4. Beyond Chinchilla: Over-Training

4.1 Why train past Chinchilla-optimal?

Chinchilla minimizes training compute for target loss. Inference is a separate (and often dominant) cost. For models that will be deployed at scale, over-training the smaller model is more efficient.

4.2 Llama 2 / 3 / 4 over-training

• Llama 2 7B: 2T tokens (286 tokens / param vs Chinchilla's 20).
• Llama 3 8B: 15T tokens (1875 / param!).
• Llama 3.1 70B: 15T tokens (215 / param).

Far past Chinchilla-optimal; better inference economics.

4.3 Inference-aware scaling laws

Sardana et al. (Mosaic, 2024): incorporate inference cost into the scaling law:

Total cost = \(C_{\mathrm{train}}\) + \(C_{\mathrm{inference}}\) · users.

For high-volume inference, optimal \(N\) shifts smaller; \(D\) shifts larger.

4.4 Diminishing returns at extreme over-training

After ~1000 tokens / param, gains slow. But still gains over Chinchilla-optimal up to ~2000 tokens / param.

4.5 DeepSeek's revised laws

DeepSeek (2024): suggest data quality matters as much as quantity past a threshold. "20 tokens / param" is not universal; depends on data quality.

4.6 When to over-train

• High-volume deployment (LLM serving, mobile).
• Quality data plentiful.
• Inference cost dominates training cost.
• Hardware-constrained inference.

4.7 When NOT to over-train

• Research setting (don't deploy).
• Compute-constrained training.
• Data quality limited.
• Model size unconstrained.

5. Per-Axis Scaling

5.1 Parameters (N)

\(N\) = total parameter count. For Transformers:

\[N \approx L \cdot (12 d^2 + \text{embedding}),\]

\(L\) layers, \(d\) hidden size. Embedding negligible at scale.

5.2 Data (D)

\(D\) = training tokens. For images: pixels; for video: frame-tokens.

5.3 Compute (\(C\))

\(C\) = total training FLOPs. For Transformers:

\[C \approx 6ND.\]

The factor 6: forward + backward + recomputation.

5.4 Effective FLOPs

With FP8 / FP4 / sparsity, "effective FLOPs" > utilized FLOPs. Scaling laws should compare on utilized FLOPs for fair comparison.

5.5 Tokens-vs-parameters chart

5.6 Memory at scale

At fixed compute, memory ~\(N\) (parameters). For training with optimizer states (AdamW): ~\(14N\) bytes (BF16 + FP32 master + Adam \(m\), \(v\)).

★ 2026 SOTA update — Precision / quantization scaling laws

• Scaling Laws for Precision: Kumar et al. 2024 (465+ runs). Low-precision training cuts a model's "effective parameter count"; unified law predicts loss from training precision + post-train quantization. Post-training quant degradation grows with more pretraining data, so extra tokens can become harmful; training big models in low precision can be compute-optimal.
• Scaling Law for QAT: unified W4A4 quantization-aware-training law over 268 runs. Quant error falls with model size but rises with more training tokens and coarser group size; FC2 activation outliers are the W4A4 bottleneck, fixed by mixed precision.

6. Inference-Time Scaling Laws

6.1 The new axis

Compute can be spent at inference, not just training. Different scaling regime; complementary.

6.2 Snell et al. (Stanford 2024)

For a fixed model and task, accuracy improves with inference compute. Specifically: a 14× smaller model with optimal inference compute can match a larger model.

6.3 Inference-compute mechanisms

• Best-of-N (parallel sampling).
• Self-consistency (majority vote).
• Tree-of-Thought / Beam search.
• MCTS / verifier-guided search.
• Long chain-of-thought (more reasoning tokens).
• Iterative refinement / Self-Refine.

6.4 Best-of-N scaling

\[\mathbb{E}\left[\max_{i \le N} r_i\right] \approx \mu + \sigma \sqrt{2 \ln N}.\]

Diminishing returns; Pareto with \(N \in [8, 64]\).

6.5 Optimal allocation per-difficulty

• Easy queries: best-of-N (parallel).
• Hard queries: search (sequential).
• Adaptive routing matches a much larger model at fixed compute.

6.6 The o1 / R1 implication

• Train via RL on verifiable rewards \(\to\) baked-in long-CoT.
• Spend compute at inference for harder tasks.
• Scaling axis shifted: more inference compute beats more training compute past a threshold.

6.7 The economic shift

• Pretraining cost: amortized across many requests.
• Inference cost: per request.
• For high-volume tasks: train smaller, inference more.
• For research / state-of-art: train larger.

6.8 Inference scaling laws (research)

Recent work (DeepMind, OpenAI, Stanford):

• Pass@k scales as ~\(1 - (1 - p_1)^k\).
• Search-based: ~logarithmic in compute.
• RL-trained CoT: power-law in training compute.
• Inference + training combined: roughly multiplicative gains.

★ 2026 SOTA update — Repeated-sampling power-law mechanism

• How Do Large Language Monkeys Get Their Power (Laws)?: Schaeffer et al. 2025. Per-problem success falls exponentially in attempts, yet aggregate \(-\log(\text{avg success})\) is a power law because single-attempt success probs are heavy-tailed. Distributional view forecasts the best-of-N exponent with ~2-4 orders of magnitude less inference compute.

7. MoE Scaling Laws

7.1 Why MoE scaling differs

MoE: total parameters \(N_{\mathrm{total}}\) vs active parameters \(N_{\mathrm{active}}\) per token.

• Compute per token: dictated by \(N_{\mathrm{active}}\).
• Capacity (knowledge): dictated by \(N_{\mathrm{total}}\).
• Memory: dictated by \(N_{\mathrm{total}}\).

7.2 Effective parameter ratio

effective \(\approx N_{\mathrm{active}} \cdot \text{ratio}(N_{\mathrm{total}}/N_{\mathrm{active}})\).

Where ratio > 1, increasing with sparsity ratio.

7.3 DeepSeek MoE scaling laws

• Loss scales mostly with \(N_{\mathrm{active}}\) at fixed data.
• Diminishing but positive return on \(N_{\mathrm{total}}\).
• Optimal sparsity: \(N_{\mathrm{total}}/N_{\mathrm{active}} \in [10, 30]\).

7.4 Compute-quality Pareto

At fixed FLOPs (per token), MoE outperforms dense:

• DeepSeek-V3 (37B active / 671B total) matches dense ~80B+.
• Mixtral 8×7B (13B active / 47B total) matches dense ~30B.

7.5 Optimal expert count

• Fewer, larger experts (Mixtral 8): simpler, easier to balance.
• More, smaller experts (DeepSeek 256): better specialization, more capacity.
• DeepSeek shows fine-grained > coarse at scale.

7.6 Top-k scaling

• Top-1 (Switch): cheapest.
• Top-2 (Mixtral): standard.
• Top-6/8 (DeepSeek with shared expert): more capacity per token.

7.7 Compute-optimal MoE Chinchilla-style

For MoE, compute-optimal \(D/N_{\mathrm{active}} \approx 30\)–\(50\) (vs 20 for dense). MoE benefits from over-training the active params more.

7.8 Total tokens for frontier MoE

• DeepSeek-V3: ~14T tokens / 37B active = 380.
• GPT-4 (rumored): ~13T tokens.
• Llama 4 Scout: ~15T tokens / 17B active.

Confirms over-training for MoE actives.

★ 2026 SOTA update — Optimal MoE sparsity laws

• Parameters vs FLOPs: Abnar et al. 2025 (Apple). Disentangles total params from active FLOPs; under fixed params or compute there is an optimal sparsity level (fraction of inactive params) that jointly maximizes training efficiency and pretraining + few-shot quality.
• Optimal Sparsity for Reasoning: Nakamura et al. 2025 (ICLR 2026 oral). Two regimes: memorization improves with more total params; reasoning is data-hungry, favoring higher active FLOPs and an optimal tokens-per-parameter. GRPO and test-time compute do not change the trend; sparsity must be set jointly by active FLOPs and TPP.

8. Vision Scaling Laws

8.1 ViT scaling

ViT-22B (Dehghani et al. 2023): demonstrated ViTs scale cleanly with proper training. Key tricks: parallel attn+MLP, QK-LN, no biases.

8.2 DINOv2 / DINOv3 scaling

• DINOv2: ViT-Giant (1.1B params), 142M images.
• DINOv3: ViT-7B, 1.7B params, scaled with gram-matching loss to preserve dense quality.
• Power-law improvement of features with scale.

8.3 CLIP / SigLIP scaling

• CLIP: ViT-B/L/H/G; ALIGN scaled to 1.8B pairs.
• SigLIP: smaller batch needed; shared scaling story.
• Loss approximately power-law in dataset size.

8.4 SAM scaling

SAM: ViT-B/L/H. Performance scales with model size + data scale (1.1B masks). SA-1B dataset critical.

8.5 ImageNet-scale saturation

Most architectures saturate ImageNet at modest scale. JFT-300M and JFT-3B (Google internal) needed for true scaling tests.

8.6 Pretrain-vs-finetune scaling

Pretrain compute \(\to\) better features. Finetune is small fraction of pretrain compute. Standard pattern: 100–1000× more pretrain than fine-tune compute.

8.7 Visual tokenizer scaling

• Image / video tokenizer quality (reconstruction PSNR) scales with parameters.
• LFQ / FSQ avoid codebook collapse at scale.
• Cosmos Tokenizer scaled to 8K resolution.

9. Multimodal Scaling Laws

9.1 VLM scaling

• LLaVA scales with LLM backbone (cf. MLLM trends).
• Multi-image / video VLMs scale with token count.
• Native multimodal models scale better than adapter-style at frontier.

9.2 Diffusion scaling laws (Peebles & Xie 2023)

DiT scaling: FID power-law-decreases with model size + training compute. Diffusion Transformers scale predictably, like LLMs.

9.3 Video diffusion scaling

• Sora / Veo demonstrate clean scaling to 60s+ clips.
• Frame-level FVD improves with both model + data scale.
• Audio-joint: scales with audio data + model.

9.4 Imagen / Parti scaling

Imagen showed power-law improvement with model size + diffusion training. T5-XXL text encoder with smaller image model beats large image + small text. Text encoder matters.

9.5 DPO / RL scaling

• Preference model quality scales with training compute.
• RL fine-tuning has diminishing returns past threshold.
• Diffusion-DPO: alignment improvement plateaus quickly.

9.6 Emergent multimodal capabilities

Compositional reasoning, OCR-in-the-wild, long video understanding emerge at frontier scale (GPT-4o, Gemini 2.5, Claude Opus). Smooth as scale scales; emergence may be threshold artifact (Schaeffer 2023).

10. Reasoning / RL Scaling

10.1 R1-Zero emergent reasoning

Pure RL on base LLM with verifiable rewards leads to spontaneous long CoT. Requires base model > ~7B for emergence.

10.2 Training compute for reasoning

• More RL compute \(\to\) better reasoning.
• Power-law-like improvement in math accuracy.
• Plateau when easy tasks saturate; harder needed.

10.3 Inference compute scales reasoning

At fixed RL training: more inference compute (long CoT, search) further improves. Multiplicative interaction with training compute.

10.4 Distillation of reasoning

R1 traces \(\to\) smaller models. Smaller distilled models match larger non-reasoning at modest scale.

10.5 Process Reward Model scaling

• PRM data scales with examples + auto-labeling (Math-Shepherd).
• Quality improves predictably.
• PRM-guided search saturates beyond a critical PRM size.

10.6 Open scaling questions

• Limit of test-time compute scaling.
• Hard-to-verify-task scaling.
• Multi-step credit assignment.
• Reasoning generalization across domains.

★ 2026 SOTA update — Predictable RL-compute scaling

• The Art of Scaling RL Compute (ScaleRL): Khatri et al. 2025, 400k+ GPU-hours. RL reward vs compute follows a sigmoidal (saturating) curve, not a power law; recipe details (loss aggregation, normalization, curriculum, off-policy) mostly shift compute efficiency not the asymptote. ScaleRL recipe extrapolates reliably to a single 100k GPU-hour run.

11. Long-Context Scaling

11.1 Cost vs context length

Quadratic attention: \(O(n^2)\) FLOPs + memory. Long context bites quickly.

11.2 Linear attention scaling

Mamba / RetNet / RWKV: \(O(n)\) scaling. Better long-horizon economics; quality gap at moderate context.

11.3 Position-extrapolation scaling

• PI: linear; cheap; quality loss with > 4× extension.
• NTK-aware: better quality.
• YaRN: 10–32× extension with light fine-tune.
• LongRoPE: 2M context.

11.4 Long-context training scaling

• Train at moderate context, fine-tune at long.
• Long-context-only training expensive; sequence parallelism + ring attention required.
• Quality scales with long-context tokens up to a point.

11.5 Inference-time long-context scaling

• KV cache scales linearly with \(n\).
• Compression (MLA, GQA) reduces per-token cost.
• Sliding window + sink (StreamingLLM): constant cache.

12. Distillation Scaling

12.1 Teacher-student gap

Student approaches teacher quality with:

• More distillation data.
• Larger student.
• Better tokenizer alignment.
• Sequence-level + token-level losses.

12.2 Distillation compute economy

Distillation typically requires ~0.1–1× pretraining compute of student. Cheaper than pretraining from scratch by \(10\times\).

12.3 R1-Distill scaling

• Distill R1 traces into Llama / Qwen students.
• Quality improves with student size.
• Some saturation: large students approach teacher.

12.4 Born-Again students

Same architecture as teacher, distilled. Often slightly better than teacher (2–3% improvement). Suggests room for distillation-driven gains.

12.5 Sample efficiency

Distillation ~10× more sample-efficient than from-scratch training (for matched final quality).

★ 2026 SOTA update — Distillation scaling law

• Distillation Scaling Laws: Busbridge et al. 2025 (Apple, ICML 2025). Predicts student loss from compute split between teacher and student; gives compute-optimal recipes. With many students or an existing teacher, distillation beats supervised learning up to a student-size-dependent compute level; for a single student needing a fresh teacher, supervised training wins. Over-strong teachers hurt (capacity gap).

13. Data Scaling Laws

13.1 Data quantity scaling

\[\text{Power-law: } L \propto D^{-\alpha}. \text{ Diminishing returns.}\]

13.2 Data quality scaling (FineWeb-Edu evidence)

High-quality data ~5–10× more efficient than naive web data. FineWeb-Edu showed: classifier-filtered "educational" web outperforms more raw data at fewer tokens.

13.3 Repeated data

Training on same data multiple epochs: repeated data ~0.6× as effective per repetition. After ~4 epochs, marginal benefit.

13.4 Data + parameters compound

Scaling laws are joint: \(L(N, D)\). Improvements compound; can't fix one and ignore the other.

13.5 Synthetic data scaling

• Phi family: textbook-quality synthetic data outperforms naive web.
• R1-Distill: long-CoT traces.
• Data quality matters more than raw quantity past saturation.

13.6 Multi-modal data scaling

• Image-text: 100M–5B pairs (LAION-5B).
• Video-text: 100M+ clip-caption.
• 3D: Objaverse-XL 10M+.
• Robot: Open-X-Embodiment 1.4M trajectories.

13.7 Open data limits

• Common Crawl: ~100T tokens raw; ~15T high-quality.
• Multi-modal data scarcer than text.
• Pre-2024 internet eventually exhausted.

★ 2026 SOTA update — Optimal data-mixture scaling laws

• Scaling Laws for Optimal Data Mixtures: Shukor et al. 2025 (Apple). Loss law in (\(N\), \(D\), domain-weight vector \(h\)) predicts performance for unseen mixtures and larger scales from a few small runs, then derives optimal domain weights for any target domain, replacing trial-and-error. Validated on LLM, native-multimodal, and vision pretraining.

14. Hyperparameter Scaling: µP

14.1 The hyperparameter problem

Optimal hyperparameters (LR, batch size, etc.) change with model scale. Tuning at frontier is impossible.

14.2 µP (Maximal Update Parameterization, Yang et al.)

Width-aware parameterization: scale (a) initialization variance, (b) learning rate, (c) optional update multipliers as functions of fan-in width.

Key

Result: optimal hyperparameters chosen on a 200M proxy model carry over almost exactly to a 7B / 70B target. Saves 99% of GPU budget for sweeps.

14.3 µTransfer

The practical transfer recipe:

1. Choose µP-parameterized base model.
2. Sweep hyperparameters at narrow / small width.
3. Apply same hyperparameters to wide / large width.

14.4 Why µP works

At infinite width, certain quantities (feature norms, gradient norms) become well-defined limits. µP keeps them invariant across widths \(\to\) optimal hyperparameters don't drift.

14.5 Variants and extensions

• Tensor Programs framework.
• Spectral parameterization (orthogonal at init).
• µP for depth: more debated.
• Mixed µP / standard parameterization.

14.6 Production usage

• Almost universal at frontier labs by 2024.
• Cerebras, MosaicML, Anthropic, OpenAI all use µP-style.
• Public open-source: Mosaic Composer.

14.7 LR scaling

Without µP: empirical \(\eta \propto B/L\) where \(B\) is batch, \(L\) depends on width. Brittle.

14.8 Batch size scaling

• Critical batch size: \(B^*\) beyond which more batch hurts efficiency.
• Linearly increases with model + data.
• Bigger models tolerate bigger batches.

15. Capability Emergence

15.1 The emergent abilities claim

Wei et al. 2022: certain LLM capabilities (multi-step arithmetic, instruction following, certain reasoning) appear "suddenly" at a critical scale.

15.2 Schaeffer 2023: emergence may be a mirage

Some emergent abilities are artifacts of discontinuous metrics (exact-match accuracy). Underlying continuous metrics (loss, partial-credit) scale smoothly. "Emergence" often disappears with proper measurement.

15.3 What actually scales smoothly

• Pretraining loss.
• Most multiple-choice accuracy.
• Continuous reward signals.
• Bits-per-byte across domains.

15.4 What may genuinely emerge

• Long chain-of-thought (R1 emergence).
• Tool use.
• Multi-step planning.
• Some compositional reasoning.

Even these may be smooth on the right metric.

15.5 Implications for roadmaps

• Don't assume thresholds; predict capability via smooth metrics.
• Test at smaller scale first.
• Smooth scaling means progress is predictable.

15.6 The frontier scaling hypothesis

• Capability roughly correlated with pretrain loss.
• Pretrain loss decreases predictably with \(C\).
• Therefore: capability gains are predictable.
• Caveat: post-training (SFT, RL) adds non-trivial gains.

16. Compute and Cost Scaling

16.1 Compute as money

• H100: ~989 TFLOPs BF16; ~$2/hr cloud.
• B200: ~2.5× H100; ~$4-5/hr cloud.
• Pretraining cost \(\propto C\).
• GPT-4-class: ~$50M–$100M training cost.
• Sora-class video: ~$10M–$50M.

16.2 Hardware utilization

Utilized FLOPs \(\ne\) peak FLOPs. Modern frontier runs achieve 40–55% of peak. Improvements (FlashAttention 3, Megatron-Core) raise utilization.

16.3 Effective FLOP budget growth

• GPU FLOPs: ~10× every 2–3 years.
• Cluster size: ~10× every 2–3 years.
• Combined: ~100× every 2–3 years training compute.

16.4 Stargate / similar mega-clusters

• OpenAI / Microsoft Stargate: $~100B over 5 years.
• xAI Colossus: 100k+ H100 cluster.
• Anthropic / Amazon: similar scale.
• Implied training-compute curve: order-of-magnitude growth per year.

16.5 Energy scaling

• Per-FLOP energy: dropping with FP8 / FP4 / sparsity.
• Cluster total power: H100 cluster ~100 MW; Stargate target ~5 GW.
• Carbon / heat / water becomes geopolitical.

16.6 Inference cost scaling

• Per-token cost: ~\(6N\) FLOPs.
• Total inference: ~users × tokens.
• Inference often > 10× training cost over model lifetime.

17. Practical Allocation Decisions

17.1 Frontier model planning

Given budget \(C\):

1. Choose \(N\), \(D\) via Chinchilla or inference-aware.
2. Choose dense vs MoE.
3. Choose architecture (depth, width, heads).
4. µP-search hyperparameters at small.
5. Plan training schedule.

17.2 Token-budget allocation

For a fixed \(D\) tokens:

• Pretrain: 80–95%.
• Continued pretrain (long context, math, code): 5–10%.
• SFT: 0.1–1%.
• RLHF / DPO / GRPO: 0.1–1%.

17.3 When to scale parameters vs data

• Compute-constrained: Chinchilla-optimal ratio.
• Inference-cost-aware: smaller \(N\), more \(D\).
• Quality-priority: more \(D\) at quality.
• Data-limited: bigger \(N\) with high-quality \(D\).

17.4 Architecture choice

• Dense < 30B: simplest.
• MoE 30B–500B: cost-optimal.
• MoE 500B+: frontier.
• Hybrid (Mamba+Transformer): for very long context.

17.5 Test-time compute investment

• Worth investing in PRMs / search infrastructure if reasoning is the goal.
• Self-consistency / best-of-N: cheap deployment.
• Long CoT: bake into RL training.

17.6 Decision matrix

18. Diffusion / Generative Scaling

18.1 DiT scaling (Peebles & Xie 2023)

FID scales as power-law in DiT model size + training compute. Same Transformer-style scaling.

18.2 Compute-optimal diffusion

For diffusion, the compute-optimal ratio differs from LLMs. Empirically \(D/N \approx 5\)–\(10\) for image diffusion.

18.3 Cascaded vs single-stage

Imagen's cascade (64 \(\to\) 256 \(\to\) 1024) trades training compute for inference compute. Stable Cascade follows similar logic.

18.4 Latent diffusion scaling

SD's latent space (\(f = 8\)) reduces compute ~64×. Allows larger effective models for fixed compute.

18.5 Video diffusion scaling

• Token count grows quadratically with resolution + linearly with time.
• Causal 3D VAE compresses by ~256×.
• Sora-class: hundreds of millions of dollars.

18.6 Distillation scaling

• Few-step (LCM, DMD2) can match many-step at same NFE budget.
• Distillation compute ~0.1× pretraining.
• Combines compounds with quantization for deployment.

19. Robotics / VLA Scaling

19.1 Open-X-Embodiment scaling

• 1.4M trajectories across 22 embodiments.
• Positive transfer demonstrated.
• Scaling laws for robot data emerging.

19.2 π0-class scaling

• ~10k hours of robot data.
• PaliGemma backbone scales with web data.
• FM action head smaller.
• Quality scales with robot data + backbone scale.

19.3 Sim-to-real scaling

• More diverse sim trajectories \(\to\) better real-world transfer.
• Domain randomization scales with randomization budget.
• Asymmetric A-C with privileged information helps.

19.4 Cross-embodiment transfer

• Open-X-Embodiment: positive transfer at scale.
• More embodiments \(\to\) better generalization.
• Cross-embodiment scaling laws emerging.

20. Frontier 2025–2026 Trends

20.1 Pretraining plateau?

• Some labs (Anthropic, OpenAI) report diminishing returns on pretraining loss.
• Frontier shifts to post-training: RL, reasoning, agents.
• Total "capability per dollar" still increasing rapidly.

20.2 Inference compute as the new axis

• o1 / R1 paradigm.
• Test-time compute matters as much as training.
• Multi-axis scaling: train + RL + inference.

20.3 Data quality > data quantity

• Phi / FineWeb-Edu / R1-Distill evidence.
• Synthetic data scales different than web.
• Curation engineering as a primary axis.

20.4 MoE + over-training dominant

DeepSeek-V3 / Llama 4 / Qwen 3 frontier models all use MoE + over-trained actives. Standard pattern.

20.5 Multimodal scaling

Native multimodal (Chameleon / Gemini 2.5 / GPT-5) scales better than adapter-style at frontier.

20.6 Test-time scaling laws

• More inference compute \(\to\) better answers (Snell et al.).
• Pareto-optimal allocation between train and infer.
• Inference scales with users; training is one-time.

20.7 Data exhaustion concerns

• Pre-2024 internet text ~exhausted at frontier.
• Synthetic data fills the gap.
• Multi-modal data scarcer.
• Quality engineering becomes primary lever.

20.8 Compute scaling

• Stargate-class clusters: 5 GW power, $100B investment.
• Per-GPU scaling: B200 ~10× A100 in BF16; FP4 ~30×.
• Total: 100× training compute over 2–3 years.

20.9 Open research questions

• Limit of pretraining loss scaling.
• Test-time compute scaling laws.
• Data quality formal scaling laws.
• Sample-efficient post-training.
• Cross-task scaling.

21. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Scaling laws are empirical power laws: \(L = A X^{-\alpha} + L_\infty\).
2. Kaplan (2020): scale params faster than data; \(\alpha_N = 0.076\).
3. Kaplan was wrong; Chinchilla (2022) corrected.
4. Chinchilla rule: 20 tokens per parameter at compute-optimal.
5. \(C \approx 6ND\) for Transformer training.
6. Chinchilla loss: \(L = E + A/N^{\alpha} + B/D^{\beta}\).
7. \(\alpha \approx 0.34\), \(\beta \approx 0.28\), \(E \approx 1.69\).
8. Llama 3 8B: 15T tokens / 8B = 1875; over-trained.
9. Inference-aware scaling shifts smaller \(N\), more \(D\).
10. Snell et al.: 14× smaller model with optimal test-time compute matches larger.
11. Best-of-\(N\): \(\mathbb{E}[\max] \approx \mu + \sigma \sqrt{2 \ln N}\).
12. MoE: \(L\) scales with \(N_{\mathrm{active}}\) at fixed data.
13. Optimal MoE sparsity: \(N_{\mathrm{total}}/N_{\mathrm{active}} \in [10, 30]\).
14. DeepSeek-V3: 671B total / 37B active / 14T tokens.
15. ViT-22B: parallel attn+MLP, QK-LN, no biases.
16. DiT scaling: power-law in FID with model + compute.
17. Distillation: ~0.1×–1× pretraining compute of student.
18. Born-Again students often beat teacher.
19. FineWeb-Edu evidence: data quality ~5–10× more efficient.
20. Repeated data: ~0.6× effective per epoch; 4 epochs cap.
21. µP: width-aware parameterization for HP transfer.
22. Schaeffer: emergent abilities may be metric artifact.
23. Hardware utilization: typically 40–55% of peak.
24. Inference cost often > 10× training cost over model lifetime.
25. Multi-axis scaling: pretrain + post-train + inference all matter.

Appendix B: Decision Tree — "How to Scale?"

1. Frontier-quality model from scratch? \(\to\) MoE + Chinchilla-overshoot + µP + extreme compute ($~10M+).

2. Best deploy economics? \(\to\) Dense small + extreme over-train (Llama 3 8B pattern).

3. Want reasoning? \(\to\) Pretrain + cold-start SFT + GRPO with verifiable rewards.

4. Cheap reasoning at small scale? \(\to\) Distill from R1-class (R1-Distill-Qwen-7B).

5. Long context? \(\to\) Linear attn (Mamba) or RoPE+YaRN + ring attention.

6. Image generation? \(\to\) DiT + flow matching + distillation (SD3 / FLUX pattern).

7. Video generation? \(\to\) Causal 3D VAE + spatiotemporal MM-DiT; lots of compute.

8. Vision encoder? \(\to\) DINOv3-style scale + gram-matching loss.

9. VLM? \(\to\) Native multimodal + scale + RL post-training.

10. Hyperparameter tuning at scale? \(\to\) µP + small-scale proxy sweep.

11. Inference-time compute? \(\to\) PRM + search + RL-trained long CoT.

12. Specialist model (medical / legal / code)? \(\to\) Distill + LoRA fine-tune from frontier base.

Appendix C: Year-by-Year Scaling Milestones

• 2017: Hestness et al. universal scaling laws.
• 2018: Larger ImageNet networks; first hints of LLM scaling.
• 2019: GPT-2; OpenAI begins scaling-laws research.
• 2020: Kaplan et al. scaling laws; GPT-3 (175B); \(C = 6ND\) approximation.
• 2021: Henighan et al. multi-modal scaling.
• 2022: Chinchilla (Hoffmann et al.) corrects Kaplan; 20 tokens/param.
• 2023: µP / µTransfer mainstream; Llama 1; emergent abilities debate.
• Late 2023–2024: Llama 2 / 3 over-training; Schaeffer "Emergence is a Mirage".
• 2024 (mid–late): DeepSeek MoE laws; inference-aware scaling (Sardana et al.); FineWeb-Edu data-quality story; Snell test-time compute scaling.
• Late 2024–2025: DeepSeek-R1 pure-RL reasoning emergence; Sora 2 / Veo 3 video scaling; o1 / o3 inference compute paradigm; Llama 3.1 / 3.2 / 4 over-training.
• 2025: Stargate-class clusters; B200 deployment; FP8 / FP4 training; data quality debates intensify; multimodal scaling formalized.
• 2026: pretraining plateau debates; test-time compute scaling laws formalized; multi-axis scaling (pretrain + RL + inference) standard.