Computer Vision — Principal Math — Interview Cheat Sheets

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

  1. Mathematical Foundations
  2. Multi-View Geometry, Calibration & SLAM
  3. Deep Learning Core: Attention, Norms, Initialization
  4. Self-Supervised Learning Losses
  5. Detection & Segmentation Math
  6. Vision-Language Foundation Models
  7. Diffusion Models & Flow Matching
  8. Video Generation & World Models
  9. NeRF & 3D Gaussian Splatting
  10. Autonomous Driving: BEV & Occupancy Math
  11. Reinforcement Learning Foundations
  12. RLHF, DPO, GRPO and Modern Preference Optimization
  13. Robotics: Imitation, VLA, Diffusion Policy
  14. Systems & Efficiency: Math You'll Need On A Whiteboard

1. Mathematical Foundations

1.1 Linear algebra

1.1.1 SVD and its geometric meaning

For \(A \in \mathbb{R}^{m\times n}\) of rank \(r\):

\[A = U\Sigma V^\top = \sum_{i=1}^{r} \sigma_i\, u_i v_i^\top, \qquad U^\top U = I,\; V^\top V = I,\; \sigma_1 \ge \cdots \ge \sigma_r > 0.\]

Eckart–Young. The best rank-\(k\) approximation in Frobenius and spectral norms is

\[A_k = \sum_{i=1}^{k}\sigma_i u_i v_i^\top, \quad \|A-A_k\|_F^2 = \sum_{i>k}\sigma_i^2, \quad \|A-A_k\|_2 = \sigma_{k+1}.\]

PCA = SVD on centered data. If \(X \in \mathbb{R}^{n\times d}\) is centered, the principal directions are the right-singular vectors of \(X\), and explained variance is \(\sigma_i^2/(n-1)\).

1.1.2 Pseudo-inverse and least squares

\[A^+ = V\Sigma^+ U^\top, \quad \hat{x} = \arg\min_x \|Ax-b\|_2^2 = A^+ b.\]

For full column rank \(A\): \(A^+ = (A^\top A)^{-1}A^\top\).

1.1.3 Eigendecomposition (symmetric case)

\(S = Q\Lambda Q^\top\) with \(Q\) orthogonal. Used for: covariance whitening \(W = Q\Lambda^{-1/2}Q^\top\), the operator norm \(\|S\|_2 = |\lambda_{\max}|\), and PSD square roots \(S^{1/2} = Q\Lambda^{1/2}Q^\top\).

1.2 Probability and information theory

1.2.1 KL divergence (the workhorse)

\[\mathrm{KL}(p\,\|\,q) = \mathbb{E}_{x\sim p}\left[\log\frac{p(x)}{q(x)}\right] \ge 0, \quad \mathrm{KL}(p\|q)=0 \iff p=q \text{ a.e.}\]

Asymmetric. Forward KL \(\mathrm{KL}(p\|q)\) is mass-covering; reverse KL \(\mathrm{KL}(q\|p)\) is mode-seeking. Variational inference minimizes reverse KL; ML estimation minimizes forward KL.

1.2.2 Jensen and the ELBO

For concave \(f\): \(f(\mathbb{E}[X]) \ge \mathbb{E}[f(X)]\). Applied to \(\log p(x)\):

\[\log p(x) = \log\int p(x,z)\,dz = \log\int q(z)\frac{p(x,z)}{q(z)}\,dz \ge \mathbb{E}_q\left[\log\frac{p(x,z)}{q(z)}\right] =: \mathcal{L}(q).\]

ELBO decomposition.

\[\log p(x) = \underbrace{\mathbb{E}_q[\log p(x|z)] - \mathrm{KL}(q(z|x)\,\|\,p(z))}_{\text{ELBO}} + \mathrm{KL}(q(z|x)\,\|\,p(z|x)).\]

The gap is exactly the KL between the variational and true posteriors.

1.2.3 Reparameterization trick

If \(z = \mu_\phi(x) + \sigma_\phi(x)\odot\epsilon,\; \epsilon\sim\mathcal{N}(0,I)\), then for any differentiable \(f\),

\[\nabla_\phi \mathbb{E}_{z\sim q_\phi}[f(z)] = \mathbb{E}_\epsilon[\nabla_\phi f(\mu_\phi + \sigma_\phi\odot\epsilon)].\]

1.2.4 Score matching

Score \(s(x) = \nabla_x\log p(x)\). Vincent's denoising score matching:

\[\mathcal{L}_{\mathrm{DSM}}(\theta) = \mathbb{E}_{x\sim p,\,\tilde{x}\sim q_\sigma(\tilde{x}|x)}\left\|s_\theta(\tilde{x}) - \nabla_{\tilde{x}}\log q_\sigma(\tilde{x}|x)\right\|^2.\]

For Gaussian noise \(q_\sigma(\tilde{x}|x) = \mathcal{N}(x,\sigma^2 I)\): \(\nabla_{\tilde{x}}\log q_\sigma = -(\tilde{x}-x)/\sigma^2\).

1.2.5 Fisher information

\[\mathcal{I}(\theta) = \mathbb{E}\left[(\nabla_\theta\log p_\theta(X))(\nabla_\theta\log p_\theta(X))^\top\right] = -\mathbb{E}[\nabla_\theta^2\log p_\theta(X)].\]

Cramér–Rao: \(\mathrm{Var}(\hat\theta) \succeq \mathcal{I}(\theta)^{-1}\).

1.3 Optimization

1.3.1 Convex preliminaries

\(f\) convex iff \(f(\lambda x + (1-\lambda)y) \le \lambda f(x) + (1-\lambda)f(y)\). \(L\)-smooth: \(\|\nabla f(x) - \nabla f(y)\| \le L\|x-y\|\). \(\mu\)-strongly convex:

\[f(y) \ge f(x) + \nabla f(x)^\top(y-x) + \frac{\mu}{2}\|y-x\|^2.\]

Convergence rates, gradient descent with \(\eta = 1/L\):

1.3.2 KKT conditions

For \(\min f(x)\) s.t. \(g_i(x)\le 0\), \(h_j(x)=0\): Lagrangian \(L = f + \sum_i\mu_i g_i + \sum_j\nu_j h_j\). KKT: stationarity (\(\nabla_x L = 0\)), primal feasibility, dual feasibility (\(\mu_i\ge 0\)), complementary slackness (\(\mu_i g_i = 0\)).

1.3.3 Modern DL optimizers

Adam: \(m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t\), \(v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2\), bias-correct, then

\[\theta_{t+1} = \theta_t - \eta\,\hat{m}_t/(\sqrt{\hat{v}_t} + \epsilon).\]

AdamW: decoupled weight decay \(\theta\leftarrow(1-\eta\lambda)\theta\) outside the moment ratio.

Lion: \(c_t = \beta_1 m_{t-1} + (1-\beta_1)g_t\); update \(\theta_{t+1} = \theta_t - \eta\,\mathrm{sgn}(c_t)\); \(m_t = \beta_2 m_{t-1} + (1-\beta_2)g_t\). Sign-based, memory-light.

Muon (matrix-aware, Newton–Schulz orthogonalization for hidden weight matrices) keeps the update on the closest orthogonal matrix to the momentum: \(U_t = \mathrm{NS}(M_t)\), \(W_{t+1} = W_t - \eta U_t\). Used at scale in 2025–2026 LLM/VLM runs.

1.4 Geometry: Lie groups SO(3) and SE(3)

For \(R\in SO(3)\) and \(t\in\mathbb{R}^3\):

\[T = \begin{pmatrix} R & t \\ 0 & 1 \end{pmatrix}.\]

1.4.1 Exponential and log maps for SO(3)

For \(\omega\in\mathbb{R}^3\), the skew-symmetric matrix

\[[\omega]_\times = \begin{pmatrix} 0 & -\omega_3 & \omega_2 \\ \omega_3 & 0 & -\omega_1 \\ -\omega_2 & \omega_1 & 0 \end{pmatrix}.\]

Rodrigues' formula.

\[R = \exp([\omega]_\times) = I + \sin\theta\,[\hat\omega]_\times + (1-\cos\theta)[\hat\omega]_\times^2.\]

Inverse: \(\theta = \cos^{-1}\left(\frac{\mathrm{tr}(R)-1}{2}\right)\), \([\omega]_\times = \frac{\theta}{2\sin\theta}(R - R^\top)\).

1.4.2 Quaternions

Unit quaternion \(q = (w, v)\), \(w^2 + v^\top v = 1\). Rotation of \(p\): \(qpq^{-1}\). Composition: Hamilton product. Cheaper than rotation matrices, no gimbal lock, and slerp interpolates uniformly:

\[\mathrm{slerp}(q_0, q_1; t) = \frac{\sin((1-t)\Omega)}{\sin\Omega}q_0 + \frac{\sin(t\Omega)}{\sin\Omega}q_1, \quad \cos\Omega = q_0\cdot q_1.\]

1.4.3 Right-perturbation Jacobian

For \(f(R)\), perturbation \(R\to R\exp([\delta]_\times)\): \(\left.\frac{\partial f}{\partial\delta}\right|_{\delta=0}\) is the right Jacobian; standard in BA / SLAM solvers.

2. Multi-View Geometry, Calibration & SLAM

2.1 Pinhole camera

World point \(X \in \mathbb{R}^3\), intrinsics \(K = \begin{pmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}\),

\[\lambda\begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = K[R\,|\,t]\begin{pmatrix} X \\ 1 \end{pmatrix}, \qquad P = K[R\,|\,t]\in\mathbb{R}^{3\times 4}.\]

Distortion (Brown–Conrady, radial-tangential): \(x_d = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2p_1 xy + p_2(r^2 + 2x^2)\), with \(r^2 = x^2 + y^2\).

2.2 Epipolar geometry

Two views with projection matrices \(P_1, P_2\), baseline \(t\), relative rotation \(R\).

Essential and fundamental matrices.

\[E = [t]_\times R, \qquad F = K_2^{-\top} E K_1^{-1}, \qquad x_2^\top F x_1 = 0.\]

\(E\) has 5 DoF (rotation 3 + translation up-to-scale 2). \(F\) has 7 DoF (rank 2 constraint \(\det F = 0\)).

8-point algorithm (Hartley normalized): linear solve from \(\ge 8\) correspondences for \(F\); enforce \(\mathrm{rank}(F) = 2\) by zeroing the smallest singular value.

5-point algorithm (Nistér): minimal solver for \(E\) given calibrated cameras; up to 10 real solutions.

2.3 Triangulation (DLT)

For correspondences \(x_1\leftrightarrow x_2\) with \(P_1, P_2\):

\[A = \begin{pmatrix} u_1 p_1^{3\top} - p_1^{1\top} \\ v_1 p_1^{3\top} - p_1^{2\top} \\ u_2 p_2^{3\top} - p_2^{1\top} \\ v_2 p_2^{3\top} - p_2^{2\top} \end{pmatrix}, \qquad A\hat{X} = 0 \;\;(\text{solve via SVD}).\]

2.4 PnP and bundle adjustment

PnP: given \(\{X_i\leftrightarrow x_i\}\) and \(K\), solve for \(R, t\). P3P (Gao), EPnP \(O(n)\) closed form, then refine with Gauss–Newton on reprojection error.

Bundle adjustment jointly optimizes camera poses \(\{T_j\}\in SE(3)^M\) and points \(\{X_i\}\in\mathbb{R}^{3N}\):

\[\min_{\{T_j\},\{X_i\}} \sum_{(i,j)\in\Omega} \rho\left(\|\pi(T_j, X_i) - x_{ij}\|_\Sigma^2\right),\]

solved with Levenberg–Marquardt; sparsity exploited via the Schur complement (eliminate point block first since it's block-diagonal).

2.5 RANSAC family

Probability of one all-inlier sample (\(s\) points, inlier ratio \(w\)): \(w^s\). Number of trials for confidence \(p\):

\[N = \frac{\log(1-p)}{\log(1-w^s)}.\]

MAGSAC++ drops the hard inlier threshold by marginalizing over noise scales; LO-RANSAC adds local optimization steps; GC-RANSAC adds graph-cut spatial coherence.

2.6 Visual SLAM losses (modern)

Photometric residual (direct method):

\[r_{ij}(T, d) = I_2(\pi(T\cdot\pi^{-1}(x_i, d_i))) - I_1(x_i),\]

optimized jointly with depth and motion (DSO, NeRF-SLAM, MonoGS).

2.7 Bundle adjustment with 3D Gaussians (GS-SLAM)

Photometric loss over rendered Gaussians: \(L = (1-\lambda)\|\hat{I} - I\|_1 + \lambda\,\mathcal{L}_{\text{D-SSIM}}(\hat{I}, I)\), with \(\hat{I}\) the splatted render (see §9).

Key

★ 2026 SOTA update

  • SuperPoint: Self-supervised CNN that jointly detects interest points and descriptors in one pass; the standard learned front-end feeding modern matching/SfM/SLAM pipelines.
  • SuperGlue: GNN + differentiable optimal-transport matcher that reasons jointly about both feature sets and rejects non-matchable points; large accuracy gains over heuristic matching.
  • LightGlue: Adaptive-depth matcher: as fast/accurate as SuperGlue but compute scales with pair difficulty, easier to train; the current practical learned matcher.
  • LoFTR: Detector-free transformer matcher producing dense coarse-to-fine matches, robust in textureless/low-repeatability areas where keypoint detectors fail.
  • MASt3R: Casts matching as a 3D task on top of DUSt3R with a dense-feature head and fast reciprocal matching; grounds correspondence directly in metric 3D geometry.
  • VGGT (Visual Geometry Grounded Transformer): Single feed-forward transformer inferring camera params, depth, point maps and 3D tracks from 1..hundreds of views in <1s, beating optimization-based post-processing (CVPR 2025).

3. Deep Learning Core: Attention, Norms, Initialization

3.1 Self-attention

Inputs \(X \in \mathbb{R}^{n\times d}\). Heads \(h\), head dim \(d_h = d/h\).

\[Q = XW_Q,\; K = XW_K,\; V = XW_V, \quad \mathrm{Attn}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_h}} + M\right)V.\]

Mask \(M\) adds \(-\infty\) to forbidden positions (causal, padding). FLOPs: \(O(n^2 d)\); memory: \(O(n^2)\).

3.1.1 Multi-Head Attention

\[\mathrm{MHA}(X) = \mathrm{Concat}(h_1, \ldots, h_H)W_O, \quad h_i = \mathrm{Attn}(XW_{Q_i}, XW_{K_i}, XW_{V_i}).\]

3.1.2 Why \(\sqrt{d_h}\)?

Variance argument: if \(q_i, k_j\) are zero-mean unit-variance with i.i.d. entries, \(\mathrm{Var}(q^\top k) = d_h\). Dividing by \(\sqrt{d_h}\) keeps logits at unit scale, preventing softmax saturation.

3.1.3 Online softmax (FlashAttention key idea)

Compute streaming max \(m\) and denominator \(\ell\) across K-tiles:

\[m^{\text{new}} = \max(m, \tilde{m}), \quad \ell^{\text{new}} = e^{m-m^{\text{new}}}\ell + e^{\tilde{m}-m^{\text{new}}}\tilde{\ell}, \quad O^{\text{new}} = \frac{e^{m-m^{\text{new}}}\ell\, O + e^{\tilde{m}-m^{\text{new}}}\tilde{\ell}\,\tilde{O}}{\ell^{\text{new}}}.\]

Achieves attention in \(O(n)\) HBM I/O instead of \(O(n^2)\).

3.2 Position encodings

Sinusoidal: \(PE(p, 2i) = \sin(p/10000^{2i/d})\), \(PE(p, 2i + 1) = \cos(p/10000^{2i/d})\).

RoPE (rotary): rotate every pair by angle \(\theta_i = 10000^{-2i/d}\). Inner product \(\langle R_p q, R_{p'} k\rangle = \langle q, R_{p'-p}k\rangle\) depends only on relative position.

ALiBi: add a linear bias \(-m(p_i - p_j)\) to logits; extrapolates to longer contexts.

3.3 Normalization

LayerNorm: \(\hat{x} = (x-\mu)/\sigma\), \(\mu, \sigma\) over feature dim. RMSNorm: drops mean centering, uses \(x/\sqrt{\mathbb{E}[x^2]}\).

BatchNorm: stats over batch+spatial, fragile at small batches. GroupNorm: stats over groups of channels.

Pre-norm vs post-norm: \(y = x + f(\mathrm{Norm}(x))\) (pre, stable for deep stacks) vs \(y = \mathrm{Norm}(x + f(x))\) (post, original Transformer).

3.4 Activations & gating

GELU: \(x\,\Phi(x)\). SiLU/Swish: \(x\,\sigma(x)\). SwiGLU block (modern):

\[\mathrm{SwiGLU}(x) = (\sigma(xW_1)\odot xW_2)W_3.\]

3.5 Initialization

Xavier/Glorot: \(\mathrm{Var}(W) = 2/(n_{\text{in}} + n_{\text{out}})\), suited to symmetric activations. Kaiming (He): \(\mathrm{Var}(W) = 2/n_{\text{in}}\), suited to ReLU. µP / µTransfer: scale \(W, \eta\) init with width \(n\) s.t. optimal hyperparameters transfer across widths — crucial when training a 7B model after tuning a 200M proxy.

3.6 Gradient clipping & stability

Global norm clip: \(g \leftarrow g\cdot\min(1, \tau/\|g\|)\). For mixed precision, keep an FP32 master copy of weights and a loss scaler.

3.7 Mixup, CutMix, label smoothing

Mixup: \(\tilde{x} = \lambda x_i + (1-\lambda)x_j\), \(\tilde{y} = \lambda y_i + (1-\lambda)y_j\), \(\lambda\sim\mathrm{Beta}(\alpha,\alpha)\). Label smoothing: \(\tilde{y}_k = (1-\epsilon)y_k + \epsilon/K\), prevents over-confidence and improves calibration.

4. Self-Supervised Learning Losses

4.1 Contrastive: InfoNCE

For positive pair \((z_i, z_i^+)\) and negatives \(\{z_k^-\}\):

\[\mathcal{L}_{\text{InfoNCE}} = -\log\frac{\exp(\langle z_i, z_i^+\rangle/\tau)}{\exp(\langle z_i, z_i^+\rangle/\tau) + \sum_k\exp(\langle z_i, z_k^-\rangle/\tau)}.\]

Equivalent to cross-entropy with the positive index as label. Lower bounds mutual information up to \(\log K\).

4.1.1 SimCLR / MoCo

Symmetrize over both views; MoCo replaces in-batch negatives with a momentum-updated queue and key encoder

\[\theta_k \leftarrow m\theta_k + (1-m)\theta_q.\]

4.1.2 BYOL (no negatives)

Online network \(f_\theta\circ g_\theta\circ q_\theta\); target \(f_\xi\circ g_\xi\) (EMA). Loss:

\[\mathcal{L} = 2 - 2\frac{\langle q_\theta(z), z_\xi'\rangle}{\|q_\theta(z)\|\,\|z_\xi'\|}, \quad \xi\leftarrow m\xi + (1-m)\theta.\]

Avoids collapse via predictor + EMA target + stop-gradient.

4.2 Sigmoid contrastive (SigLIP / SigLIP 2)

Per-pair sigmoid loss instead of softmax over batch:

\[\mathcal{L}_{\text{SigLIP}} = -\frac{1}{B}\sum_{i,j}\log\sigma(z_{ij}(t\langle x_i, y_j\rangle + b)), \quad z_{ij} = \begin{cases} +1 & i=j \\ -1 & i\ne j \end{cases}.\]

Avoids batch-wide normalization \(\Rightarrow\) scales to small or huge batches and is sample-efficient.

4.3 DINO / DINOv2 / DINOv3 (self-distillation)

Student \(g_\theta\), teacher \(g_\xi\) (EMA). Cross-view loss with sharpening (low temp \(\tau_t\) on teacher) and centering (subtract running mean \(c\)):

\[\mathcal{L}_{\text{DINO}} = -\,\mathrm{softmax}\big((g_\xi(v) - c)/\tau_t\big)^\top\log\mathrm{softmax}(g_\theta(v')/\tau_s).\]

DINOv3 add-ons: KoLeo regularizer (penalizes nearest-neighbor distances of normalized features for spread), gram-matching loss to keep dense feature similarity stable when scaling to ViT-7B.

4.4 Masked Image Modeling (MAE)

Mask 75% of patches; encoder sees only visible tokens, lightweight decoder reconstructs:

\[\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\|\hat{p}_i - p_i\|_2^2,\]

where \(p_i\) is the (per-patch normalized) target.

4.5 I-JEPA / V-JEPA (latent prediction)

Predict target patch embeddings (via EMA target encoder \(f_{\bar\theta}\)) from a context, instead of pixels:

\[\mathcal{L}_{\text{JEPA}} = \sum_{i\in\mathcal{T}}\left\|\hat{f}_\theta(x)_i - f_{\bar\theta}(x)_i\right\|_2^2.\]

4.6 What's the right SSL?

Pixel reconstruction \(\to\) strong dense, weaker semantics. Contrastive \(\to\) strong global, weaker dense. JEPA / EMA distillation \(\to\) both, with care against collapse (centering, sharpening, predictor, stop-grad).

5. Detection & Segmentation Math

5.1 IoU and variants

\[\mathrm{IoU}(A, B) = \frac{|A\cap B|}{|A\cup B|}.\]

GIoU: \(\mathrm{GIoU} = \mathrm{IoU} - \frac{|C\setminus(A\cup B)|}{|C|}\), \(C\) = smallest enclosing box. Differentiable everywhere.

DIoU: \(\mathrm{DIoU} = \mathrm{IoU} - \frac{\rho^2(b, b^{gt})}{c^2}\), \(\rho\) = center distance, \(c\) = diag of \(C\).

CIoU: adds aspect-ratio term \(\alpha v\), \(v = \frac{4}{\pi^2}\left(\arctan\frac{w^{gt}}{h^{gt}} - \arctan\frac{w}{h}\right)^2\).

5.2 NMS, Soft-NMS, NMS-free

Greedy NMS: sort by score; while non-empty, pop best \(b\), suppress all \(b'\) with \(\mathrm{IoU}(b, b') > \theta\).

Soft-NMS: decay scores instead of suppressing: \(s_i \leftarrow s_i\, e^{-\mathrm{IoU}(b_i, b_M)^2/\sigma}\).

NMS-free (DETR / YOLOv10): one-to-one Hungarian matching during training removes the need for NMS at inference.

5.3 Hungarian matching (DETR)

Cost between prediction \(\hat{y}_i\) and ground truth \(y_j\):

\[\mathcal{C}_{ij} = -\hat{p}_i(c_j) + \lambda_{L1}\left\|\hat{b}_i - b_j\right\|_1 + \lambda_{\text{giou}}(1 - \mathrm{GIoU}(\hat{b}_i, b_j)).\]

Solve \(\sigma^* = \arg\min_\sigma\sum_i\mathcal{C}_{i,\sigma(i)}\) with the Hungarian algorithm; backprop standard losses.

5.4 Focal loss

For dense detection (RetinaNet):

\[\mathrm{FL}(p_t) = -\alpha_t(1 - p_t)^\gamma\log(p_t), \quad p_t = \begin{cases} p & y=1 \\ 1-p & y=0 \end{cases},\]

typically \(\alpha = 0.25\), \(\gamma = 2\). Down-weights easy examples and addresses extreme foreground/background imbalance.

5.5 Mean Average Precision (COCO mAP)

For each class, sweep score threshold, compute precision \(P = \frac{TP}{TP+FP}\), recall \(R = \frac{TP}{TP+FN}\). AP = area under interpolated PR curve. COCO averages AP over IoU thresholds \(\{0.50, 0.55, \ldots, 0.95\}\) and all classes.

5.6 DICE loss (segmentation)

\[\mathrm{DICE}(p, y) = 1 - \frac{2\sum_i p_i y_i + \epsilon}{\sum_i p_i + \sum_i y_i + \epsilon}.\]

Often combined with cross-entropy: \(L = \mathrm{CE} + \lambda\,\mathrm{DICE}\).

5.7 Mask2Former / universal mask paradigm

Predict \(N\) binary masks \(m_i\) + class probabilities \(p_i\). Final per-pixel semantic class:

\[P(c\,|\,x) = \sum_{i=1}^{N} p_i(c)\,m_i(x).\]

Unifies semantic, instance, panoptic.

5.8 SAM 2 video memory module (sketch)

Streaming attention: key/value features of past frames stored in a memory bank \(M_t = \{(k_\tau, v_\tau)\}_{\tau<t}\); current-frame queries cross-attend to \(M_t\) (typically capped at \(\sim 8\) frames + first-frame anchor) for temporal consistency.

6. Vision-Language Foundation Models

6.1 CLIP (softmax InfoNCE in two directions)

Image and text encoders produce \(\ell_2\)-normalized embeddings \(u_i, t_j\). With temperature \(\tau = e^\theta\) (learned), logits \(L_{ij} = \langle u_i, t_j\rangle/\tau\).

\[\mathcal{L}_{\text{CLIP}} = -\frac{1}{2B}\sum_{i=1}^{B}\left[\log\frac{e^{L_{ii}}}{\sum_j e^{L_{ij}}} + \log\frac{e^{L_{ii}}}{\sum_j e^{L_{ji}}}\right].\]

6.2 SigLIP / SigLIP 2 (sigmoid pairwise)

\[\mathcal{L} = -\log\sigma(z_{ii}\cdot(\langle u_i, t_i\rangle/\tau + b)) - \sum_{j\ne i}\log\sigma(-(\langle u_i, t_j\rangle/\tau + b)).\]

Two learned scalars \(\tau\) and bias \(b\). SigLIP 2 adds captioning and self-distillation auxiliaries for stronger dense features.

6.3 Visual feature injection in MLLMs

Three families:

6.4 2D-RoPE for native dynamic resolution (Qwen2-VL)

Split feature dim into row and column halves; apply 1D RoPE with frequencies \(\theta_i\) to row half indexed by \(r\) and to column half indexed by \(c\). Inner product depends only on \((r - r', c - c')\), generalizing naturally to any \(H\times W\).

6.5 Pixel unshuffle for token compression

Reshape \((H, W, C)\to(H/r, W/r, Cr^2)\) then linearly project to \(C'\). Reduces token count by \(r^2\) while preserving spatial info; standard in InternVL family.

6.6 Long-video token packing

Token count per second \(\approx \mathrm{fps}\cdot HW/p^2\). For 1-hour video at 1 fps, \(14\times 14\) patches on \(224^2\): \(3600\cdot 256\approx 922{,}000\) tokens. Compression strategies: temporal pooling, similarity-based merging (ToMe), Q-Former summarization, 3D-tubelet patches.

7. Diffusion Models & Flow Matching

7.1 Forward and reverse processes (DDPM)

Forward Markov chain with variance schedule \(\beta_t\in(0, 1)\):

\[q(\mathbf{x}_t|\mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\,\mathbf{x}_{t-1}, \beta_t I).\]

With \(\alpha_t = 1-\beta_t\), \(\bar\alpha_t = \prod_{s\le t}\alpha_s\):

Closed-form marginal.

\[q(\mathbf{x}_t|\mathbf{x}_0) = \mathcal{N}(\mathbf{x}_t; \sqrt{\bar\alpha_t}\,\mathbf{x}_0, (1-\bar\alpha_t)I), \quad \mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\; \epsilon\sim\mathcal{N}(0, I).\]

Posterior:

\[q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}(\tilde\mu_t, \tilde\beta_t I), \quad \tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\,\beta_t}{1-\bar\alpha_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}\mathbf{x}_t, \quad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t.\]

7.2 Variational bound and the simplified objective

ELBO decomposes per step:

\[\mathcal{L}_{\text{vlb}} = \mathbb{E}_q\Big[\mathrm{KL}(q(\mathbf{x}_T|\mathbf{x}_0)\|p(\mathbf{x}_T)) + \sum_{t>1}\mathrm{KL}(q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0)\|p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)) - \log p_\theta(\mathbf{x}_0|\mathbf{x}_1)\Big].\]

With \(p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t) = \mathcal{N}(\mu_\theta(\mathbf{x}_t, t), \sigma_t^2 I)\) and the \(\epsilon\)-parameterization \(\mu_\theta = \frac{1}{\sqrt{\alpha_t}}\big(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\epsilon_\theta\big)\):

Simplified DDPM objective.

\[\mathcal{L}_{\text{simple}} = \mathbb{E}_{t,\mathbf{x}_0,\epsilon}\left\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\epsilon, t)\right\|^2.\]

7.3 Parameterizations

\(\epsilon\)-prediction, \(\mathbf{x}_0\)-prediction, and v-prediction \(v_t = \sqrt{\bar\alpha_t}\,\epsilon - \sqrt{1-\bar\alpha_t}\,\mathbf{x}_0\). Conversions:

\[\mathbf{x}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\epsilon}{\sqrt{\bar\alpha_t}}, \quad \epsilon = \sqrt{\bar\alpha_t}\,v + \sqrt{1-\bar\alpha_t}\,\mathbf{x}_0/\sqrt{\bar\alpha_t}.\]

v-prediction is more stable across noise levels and a better target for distillation.

7.4 DDIM (deterministic sampling)

Non-Markov forward with the same marginals; reverse step:

\[\mathbf{x}_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\hat{\mathbf{x}}_{0t} + \sqrt{1-\bar\alpha_{t-1} - \sigma_t^2}\,\epsilon_\theta(\mathbf{x}_t, t) + \sigma_t z,\]

with \(\hat{\mathbf{x}}_{0t} = (\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}\). \(\sigma_t = 0\) is fully deterministic and supports inversion.

7.5 Score SDE view

Forward SDE \(dx = f(x, t)\,dt + g(t)\,dw\), reverse:

\[dx = [f(x, t) - g(t)^2\nabla_x\log p_t(x)]\,dt + g(t)\,d\bar{w}.\]

Probability-flow ODE: \(dx = [f(x, t) - \frac{1}{2}g(t)^2\nabla_x\log p_t(x)]\,dt\).

7.6 Classifier-free guidance (CFG)

Train a single network \(\epsilon_\theta(\mathbf{x}_t, t, c)\) with \(c = \varnothing\) a fraction \(p_{\text{drop}}\) of the time. At inference:

\[\tilde\epsilon_\theta(\mathbf{x}_t, t, c) = \epsilon_\theta(\mathbf{x}_t, t, \varnothing) + w\,(\epsilon_\theta(\mathbf{x}_t, t, c) - \epsilon_\theta(\mathbf{x}_t, t, \varnothing)).\]

Equivalent to sampling from \(p(x|c)\propto p(x)p(c|x)^w\) asymptotically (Bayes log-trick).

7.7 Flow Matching / Rectified Flow

Define a probability path \(p_t(x)\) from \(p_0 = \mathcal{N}(0, I)\) to \(p_1 = p_{\text{data}}\) with vector field \(u_t\):

\[dx_t = u_t(x_t)\,dt, \qquad x_0\sim p_0.\]

Conditional FM (Lipman): pick affine path \(x_t = (1-t)x_0 + tx_1\); conditional vector field \(u_t(x|x_1) = x_1 - x_0\).

Loss:

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{t\sim U(0,1),\,x_0\sim p_0,\,x_1\sim p_{\text{data}}}\left\|v_\theta(x_t, t) - (x_1 - x_0)\right\|^2.\]

Rectified Flow straightens the trajectories iteratively and underlies SD3 and FLUX. One-step distillation (InstaFlow / Reflow) reuses \((x_0, x_1)\) pairs from a teacher.

7.8 Latent diffusion

Train an autoencoder \(E, D\) with KL or VQ regularization, then run diffusion in latent \(z = E(x)\):

\[\mathcal{L} = \mathbb{E}_{z,t,\epsilon,c}\left\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,z + \sqrt{1-\bar\alpha_t}\,\epsilon, t, c)\right\|^2.\]

Decode at sampling time with \(\hat{x} = \mathcal{D}(\hat{z})\).

7.9 Distillation: Consistency Models

Learn \(f_\theta(\mathbf{x}_t, t)\approx\mathbf{x}_0\) for any \(t\), with self-consistency: \(f_\theta(\mathbf{x}_t, t) = f_{\theta^-}(\mathbf{x}_{t'}, t')\) along the same ODE trajectory. Loss:

\[\mathcal{L}_{\text{CM}} = \mathbb{E}\,d\big(f_\theta(x_{t_{n+1}}, t_{n+1}), f_{\theta^-}(\hat{x}_{t_n}, t_n)\big),\]

with EMA target \(\theta^-\) and a metric \(d\) (LPIPS or L2). Gives 1–4 step samplers.

7.10 Score-Distillation Sampling (DreamFusion)

Differentiate a parametric scene \(\theta\) (NeRF or 3DGS) by passing renders \(x = g(\theta)\) through a frozen diffusion teacher:

\[\nabla_\theta\mathcal{L}_{\text{SDS}} = \mathbb{E}_{t,\epsilon}\big[w(t)(\epsilon_\phi(x_t, t, c) - \epsilon)\,\partial x/\partial\theta\big].\]

VSD (Variational SDS, ProlificDreamer) replaces the noise target with a learned variational distribution to reduce mode collapse.

7.11 Diffusion-DPO

Preferences over generated images \((x_w, x_l|c)\) collapse to:

\[\mathcal{L}_{\text{D-DPO}} = -\mathbb{E}\log\sigma\Big(-\beta\big(\mathcal{L}_\theta(x^w, c) - \mathcal{L}_{\theta_{\text{ref}}}(x^w, c) - \mathcal{L}_\theta(x^l, c) + \mathcal{L}_{\theta_{\text{ref}}}(x^l, c)\big)\Big),\]

with \(\mathcal{L}_\theta(x, c)\) the standard diffusion loss on \(x\). Used to fine-tune SDXL/SD3/FLUX on aesthetic pairs.

8. Video Generation & World Models

8.1 Spatiotemporal latent diffusion

Video \(V\in\mathbb{R}^{T\times H\times W\times 3}\). Causal 3D VAE encodes to latents \(Z\in\mathbb{R}^{T'\times H'\times W'\times C}\) (typically \(T' = T/4\), \(H' = H/8\)).

DiT operates on \(Z\) with 3D positional encodings; loss is the same diffusion / FM target.

8.2 Joint image + video training

A still image is the \(T = 1\) case; with packing, the same model trains on both. Loss weighting often boosts image gradients to retain per-frame quality: \(L = \lambda_I\mathcal{L}_{\text{image}} + \lambda_V\mathcal{L}_{\text{video}}\).

8.3 Vision tokenizers (LFQ / FSQ)

Lookup-Free Quantization (MAGVIT-v2): project each latent vector \(z\) to dimension \(L\), then sign-quantize: \(q = \mathrm{sgn}(z)\in\{-1, +1\}^L\), giving \(2^L\) codes. Loss:

\[\mathcal{L}_{\text{LFQ}} = \|\mathrm{sg}(q) - z\|^2 + \beta\,\text{Entropy-reg}.\]

FSQ: round each scalar to a small integer set \(\{-K, \ldots, K\}\).

8.4 World models (Dreamer V3 sketch)

RSSM with latent \(h_t\) (deterministic, GRU) and stochastic state \(z_t\):

\[h_t = f_\phi(h_{t-1}, z_{t-1}, a_{t-1}), \quad z_t\sim q_\phi(z_t|h_t, o_t), \quad \hat{z}_t\sim p_\phi(\hat{z}_t|h_t).\]

Losses: reconstruction \(-\log p_\phi(o_t|h_t, z_t)\), \(\mathrm{KL}(q\|\hat{z})\) (with two-hot target/symlog reward heads in V3), and policy training via imagined rollouts using REINFORCE + value baseline.

8.5 Video evaluation (sketch)

\[\mathrm{FVD} = \|\mu_r - \mu_g\|^2 + \mathrm{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big)\]

with I3D/InternVideo features (FID generalization). VBench / VBench-2 add 16+ axes (motion smoothness, dynamic degree, subject consistency, etc.).

9. NeRF & 3D Gaussian Splatting

9.1 Volumetric rendering equation

Along a ray \(r(t) = o + td\) from \(t_n\) to \(t_f\):

\[C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,c(\mathbf{r}(t), \mathbf{d})\,dt, \quad T(t) = \exp\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right).\]

Discretization with \(N\) samples and \(\delta_i = t_{i+1} - t_i\):

\[C\approx\sum_{i=1}^{N} T_i(1 - e^{-\sigma_i\delta_i})c_i, \quad T_i = \exp\Big(-\sum_{j<i}\sigma_j\delta_j\Big).\]

Loss: \(\mathcal{L} = \sum_{\mathbf{r}}\left\|C(\mathbf{r}) - \hat{C}(\mathbf{r})\right\|_2^2\).

9.1.1 Hierarchical sampling

Coarse net produces weights \(w_i = T_i(1 - e^{-\sigma_i\delta_i})\); sample fine ray points from the normalized PDF \(w_i/\sum_j w_j\).

9.1.2 Positional encoding (NeRF)

\[\gamma(p) = [\sin(2^k\pi p), \cos(2^k\pi p)]_{k=0}^{L-1}, \text{ applied per coordinate.}\]

9.2 3D Gaussian Splatting (3DGS)

Each Gaussian \(i\): position \(\mu_i\in\mathbb{R}^3\), anisotropic covariance \(\Sigma_i = R_i S_i S_i^\top R_i^\top\) (with \(R_i\in SO(3)\) from a quaternion, \(S_i\) diagonal scale), opacity \(\alpha_i\in[0, 1]\), and SH coefficients for view-dependent color \(c_i(d)\).

Projection to image plane. Approximate the 3D Gaussian by a 2D Gaussian after camera projection:

\[\Sigma' = JW\Sigma W^\top J^\top,\]

where \(W\) is the world-to-camera linearization and \(J\) the Jacobian of the perspective projection at \(\mu_i\).

Differentiable rasterization (per-pixel front-to-back over depth-sorted Gaussians):

\[C(p) = \sum_{i\in\mathcal{N}(p)} c_i\,\alpha_i'\prod_{j<i}(1 - \alpha_j'), \quad \alpha_i' = \alpha_i\cdot\exp\Big(-\tfrac{1}{2}(x - \mu_i')^\top\Sigma_i'^{-1}(x - \mu_i')\Big).\]

Training loss.

\[\mathcal{L}_{\text{3DGS}} = (1-\lambda)\|\hat{I} - I\|_1 + \lambda\,\mathcal{L}_{\text{D-SSIM}}(\hat{I}, I), \quad \lambda\approx 0.2.\]

Adaptive density control: clone (small under-reconstruction), split (large over-reconstruction), prune \((\alpha < \tau)\). Periodic opacity reset.

9.3 Variants

Mip-Splatting: 2D mip + 3D smoothing filters to reduce aliasing.

2D-GS: planar disks with stable normals for surfaces.

4D-GS / Deformable 3DGS: time-dependent \(\mu(t), \Sigma(t)\) via MLPs or spline bases.

Scaffold-GS: anchor points predict neural Gaussians on the fly, reducing memory.

LRM / GS-LRM: feed-forward transformer maps \(N\) posed images to \(N_g\) Gaussians end-to-end.

9.4 Feed-forward 3D from images: DUSt3R

Given two images, predict per-pixel pointmaps \(X_{1,1}, X_{2,1}\in\mathbb{R}^{H\times W\times 3}\) in the frame of camera 1. Loss:

\[\mathcal{L}_{\text{DUSt3R}} = \sum_{v\in\{1,2\}}\sum_{i\in\mathcal{V}_v}\left\|\frac{1}{z}\hat{X}_i^{v,1} - \frac{1}{\bar{z}}X_i^{v,1}\right\| + \lambda\,\mathrm{conf}_i,\]

with normalization to handle scale ambiguity. Camera intrinsics, extrinsics, depth all decode from the pointmap.

10. Autonomous Driving: BEV & Occupancy Math

10.1 Camera-to-BEV transforms

LSS (Lift-Splat-Shoot): for each pixel, predict a categorical depth distribution \(\alpha\in\Delta^{D-1}\) over \(D\) depth bins; lift features \(f\) to a frustum:

\[F(d) = \alpha_d\cdot f, \quad d\in\{d_1, \ldots, d_D\},\]

project frustum points to BEV cells via \(X = K^{-1}[u, v, 1]^\top d\) then \(X_w = R^\top(X - t)\).

10.1.1 BEVFormer (deformable cross-attention to image features)

BEV query \(q(x,y)\) samples around its projected pixels in each camera; deformable attention:

\[\mathrm{DA}(q, p) = \sum_{m=1}^{M} W_m\sum_{k=1}^{K} A_{mk}\,x(p + \Delta p_{mk}),\]

with offsets \(\Delta p_{mk}\) and weights \(A_{mk}\) predicted from \(q\). Temporal self-attention pulls from \(\text{BEV}_{t-1}\) for motion.

10.2 3D bounding box parameterization

\((x, y, z, w, l, h, \theta)\) in ego frame; loss uses smooth-L1 on residuals normalized by anchor scale, plus a classification head and an IoU-aware confidence head. Heading often predicted as \((\sin\theta, \cos\theta)\) to avoid wrap-around.

10.3 Occupancy prediction

3D voxel grid \(V\in\{0, 1\}^{X\times Y\times Z}\), multi-class semantics \(c\in\{1, \ldots, C\}\). Loss:

\[\mathcal{L}_{\text{occ}} = \mathrm{CE}_{\text{voxel}} + \lambda_g\,\mathcal{L}_{\text{Lovász}} + \lambda_a\,\mathcal{L}_{\text{Affinity}},\]

where Lovász handles class imbalance and affinity preserves local structure. Sparse occupancy (SparseOcc, OPUS) only predicts non-empty voxels for \(\sim 100\times\) memory savings.

10.4 Motion prediction (multi-modal)

Predict \(K\) trajectory modes with probabilities \(\pi_k\) and per-step Gaussians:

\[p(\tau|h) = \sum_{k=1}^{K}\pi_k(h)\prod_{t=1}^{T_f}\mathcal{N}(\tau_t; \mu_{k,t}, \Sigma_{k,t}).\]

Train with min-of-K ADE/FDE plus cross-entropy on the closest mode (winner-takes-all to avoid mode averaging).

10.5 End-to-end planning loss (UniAD-style)

\[\mathcal{L}_{\text{e2e}} = \sum_{\text{tasks}}\lambda_\tau\mathcal{L}_\tau + \lambda_p\mathcal{L}_{\text{plan}}, \quad \mathcal{L}_{\text{plan}} = \|\hat\tau - \tau^*\|_2 + \lambda_c\,\mathrm{CollisionPenalty}(\hat\tau).\]

11. Reinforcement Learning Foundations

11.1 MDP and value functions

\((\mathcal{S}, \mathcal{A}, P, r, \gamma)\). Returns \(G_t = \sum_{k\ge 0}\gamma^k r_{t+k+1}\). Bellman:

\[V^\pi(s) = \mathbb{E}_\pi[r + \gamma V^\pi(s')|s], \quad Q^\pi(s, a) = \mathbb{E}[r + \gamma\mathbb{E}_{a'\sim\pi}Q^\pi(s', a')|s, a].\]

Optimality: \(V^*(s) = \max_a\mathbb{E}[r + \gamma V^*(s')|s, a]\), \(Q^*(s, a) = \mathbb{E}[r + \gamma\max_{a'}Q^*(s', a')|s, a]\).

11.2 Policy gradient theorem

\[\nabla_\theta J(\theta) = \mathbb{E}_{s\sim d^\pi,\,a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a|s)\,Q^\pi(s, a)\big] = \mathbb{E}[\nabla_\theta\log\pi_\theta(a|s)\,A^\pi(s, a)],\]

with advantage \(A = Q - V\).

11.2.1 Generalized Advantage Estimation (GAE)

TD residual \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\). GAE:

\[\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}.\]

11.3 TRPO / PPO

TRPO: \(\max_\theta \mathbb{E}[r_\theta A]\) s.t. \(\mathbb{E}[\mathrm{KL}(\pi_{\text{old}}\|\pi_\theta)]\le\delta\); \(r_\theta = \pi_\theta/\pi_{\text{old}}\).

PPO clipped surrogate.

\[\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\big[\min(r_t(\theta)\hat{A}_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)\big].\]

Total loss: \(L = \mathcal{L}^{\text{CLIP}} - c_v\mathcal{L}^{\text{VF}} + c_e\mathcal{H}[\pi_\theta]\).

11.4 SAC (off-policy maximum entropy)

\[J(\pi) = \sum_t\mathbb{E}\big[r(s_t, a_t) + \alpha\mathcal{H}(\pi(\cdot|s_t))\big].\]

Soft Bellman: \(Q^*(s, a) = r + \gamma\mathbb{E}_{s'}\mathbb{E}_{a'\sim\pi}[Q^*(s', a') - \alpha\log\pi(a'|s')]\).

Policy update minimizes \(\mathrm{KL}(\pi_\theta(\cdot|s)\,\|\,\exp(Q/\alpha)/Z)\). Two Q-nets, target nets, and an automatically tuned \(\alpha\) via entropy constraint.

11.5 TD3, DDPG

Deterministic policy \(\mu_\theta\). DDPG: \(L_Q = (r + \gamma Q_{\bar\theta}(s', \mu_{\bar\theta}(s')) - Q(s, a))^2\). TD3 adds: clipped double-Q, target-policy smoothing \(\mu(s') + \mathrm{clip}(\mathcal{N}(0, \sigma), -c, c)\), and delayed policy updates.

11.6 Decision Transformer

\(\pi_\theta(a_t\,|\,R_t, s_t, a_{<t}, R_{<t}, s_{<t})\)

Conditioning on returns-to-go: train with cross-entropy on action tokens. At test, prompt with target return \(R_0\).

11.7 Dreamer V3 head equations

Symlog targets for stability: \(\mathrm{symlog}(x) = \mathrm{sgn}(x)\log(|x| + 1)\). Two-hot reward/value distributional heads. Imagined horizon 15–16 steps; actor optimizes a \(\lambda\)-return estimate with a moving baseline.

12. RLHF, DPO, GRPO and Modern Preference Optimization

12.1 Bradley–Terry reward learning

Pairs \((x, y_w, y_l)\) with \(y_w\) preferred. Reward model \(r_\phi\):

\[\mathcal{L}_{\text{RM}} = -\mathbb{E}\big[\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))\big].\]

12.2 KL-regularized RLHF (PPO recipe)

\[\max_\pi\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot|x)}\big[r_\phi(x, y)\big] - \beta\,\mathrm{KL}\big(\pi(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big).\]

Per-token reward \(\tilde{r}_t = -\beta\big(\log\pi(a_t|s_t) - \log\pi_{\text{ref}}(a_t|s_t)\big)\) at every step plus the terminal reward; optimize with PPO.

12.3 DPO derivation

The KL-constrained max has closed-form optimal policy:

\[\pi^*(y|x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\Big(\frac{1}{\beta}r(x, y)\Big).\]

Solve for \(r\):

\[r(x, y) = \beta\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x).\]

Substitute into the BT log-likelihood; the \(\log Z(x)\) cancels in differences:

DPO loss.

\[\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big).\]

No reward model, no PPO, no value function. Trade-offs: prone to over-fitting to preference pairs and to reducing \(\pi_\theta\) probability of both responses without care.

12.4 IPO, KTO, ORPO, SimPO

IPO (Identity Preference Optimization): squared-margin loss, robust to deterministic preferences.

\[\mathcal{L}_{\text{IPO}} = \mathbb{E}\Big(h_\theta(y_w, y_l; x) - \tfrac{1}{2\beta}\Big)^2, \quad h_\theta = \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}.\]

KTO: prospect-theory utility, needs only thumbs-up/thumbs-down per response (no pairs). ORPO: SFT + odds-ratio penalty, removes the SFT \(\to\) DPO two-stage pipeline. SimPO: length-normalized log-prob, drops the reference policy:

\[\mathcal{L}_{\text{SimPO}} = -\log\sigma\Big(\frac{\beta}{|y_w|}\log\pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l|x) - \gamma\Big).\]

12.5 GRPO (DeepSeek)

Sample a group of \(G\) responses \(\{y_i\}\) for prompt \(x\), score with reward (or programmatic verifier) \(r_i\). Group-relative advantage:

\[\hat{A}_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\}) + \epsilon}.\]

Per-token clipped objective without a learned value:

GRPO objective.

\[\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\big(\rho_{i,t}\hat{A}_i, \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_i\big) - \beta\,\mathrm{KL}[\pi_\theta\|\pi_{\text{ref}}]\right],\]

with \(\rho_{i,t} = \pi_\theta(y_{i,t}|x, y_{i,<t})/\pi_{\text{old}}(y_{i,t}|x, y_{i,<t})\). Removes the value network; relies on group baselines to reduce variance. Variants: DAPO (decoupled clip + dynamic sampling), Dr. GRPO (drop normalization to remove length/difficulty bias), REINFORCE++ (per-token rewards + KL).

12.6 Process reward models (PRMs)

Per-step labels \(y_t\in\{0, 1\}\) (good / bad partial reasoning):

\[\mathcal{L}_{\text{PRM}} = -\sum_t\big[y_t\log\hat{y}_t + (1 - y_t)\log(1 - \hat{y}_t)\big].\]

At inference, search/best-of-N/MCTS uses PRM as the value heuristic. Auto-labeling via Math-Shepherd / OmegaPRM rollouts.

12.7 Inference-time scaling: best-of-N analytics

For an outcome reward \(r\) with mean \(\mu\), std \(\sigma\) across responses, the expected best-of-N improves like \(\mathbb{E}[\max_{i\le N} r_i]\approx\mu + \sigma\sqrt{2\ln N}\) (Gaussian approximation). Diminishing returns with \(N\), but with MCTS / verifier-guided search it can be much better.

12.8 Multimodal RL (VLM-R1 / MM-EUREKA)

Same GRPO/PPO machinery; rewards are programmatic on visual outputs:

13. Robotics: Imitation, VLA, Diffusion Policy

13.1 Imitation learning baselines

Behavior Cloning: \(\min_\theta\mathbb{E}_{(s,a)\sim\mathcal{D}}[-\log\pi_\theta(a|s)]\). Distribution shift \(\Rightarrow\) DAgger: \(\mathcal{D}\leftarrow\mathcal{D}\cup\{(s, \pi^*(s))\}\) with \(s\sim\pi_\theta\).

13.2 Action chunking (ACT)

Predict a chunk of \(H\) actions from a history; train with reconstruction + a CVAE term:

\[\mathcal{L}_{\text{ACT}} = \mathbb{E}\big[\|\hat{a}_{t:t+H} - a_{t:t+H}\|_1 + \beta\,\mathrm{KL}(q(z|a_{t:t+H}, o)\|p(z))\big].\]

Temporal ensembling at test: average overlapping chunk predictions, smoothing actions.

13.3 Diffusion Policy

Action sequence \(a_{t:t+H}\) generated by a conditional diffusion model on observation history \(o_{t-k:t}\). Loss:

\[\mathcal{L} = \mathbb{E}_{k,\epsilon,a_0,o}\left\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_k}\,a_0 + \sqrt{1-\bar\alpha_k}\,\epsilon, k, o)\right\|^2.\]

Inference: sample \(H\) actions, execute receding-horizon (open-loop \(h < H\) steps).

13.4 Vision-Language-Action models (VLA)

Pretrained VLM \(f_\theta\) produces token sequence; an action head decodes:

13.5 Sim-to-real techniques

Domain randomization: at every episode sample \(\xi\sim p(\xi)\) over visual + dynamics nuisance, train \(\pi(s, \xi)\). Asymmetric A-C: critic \(V(s, \xi)\) uses privileged info; actor uses only \(s\); teacher→student distillation:

\[\mathcal{L}_{\text{distill}} = \mathbb{E}\left\|\pi_S(o) - \pi_T(s, \xi)\right\|^2.\]

13.6 Eureka / LLM-designed rewards

LLM proposes reward function code \(R_\phi\); train RL; evaluate via task success rate on a held-out evaluator; use signal to refine \(R_\phi\) (evolutionary loop). Massive parallelism (Isaac Gym / MJX, \(10^4\) envs) makes the loop tractable.

14. Systems & Efficiency: Math You'll Need On A Whiteboard

14.1 FlashAttention I/O complexity

Standard attention: \(O(N^2)\) HBM reads/writes. FlashAttention tiles \(Q, K, V\) in SRAM blocks \(B_r\times B_c\):

\[\text{HBM traffic} = O(N^2 d/M),\]

with \(M=\) SRAM capacity. Backward avoids materializing \(S = QK^\top\) via recomputation. FlashAttention 2 cleans up parallelism over heads/sequences; FA3 adds Hopper async TMA + WGMMA + FP8.

14.2 Distributed-training memory math

For a model with \(P\) parameters trained in mixed precision (BF16 weights/activations, FP32 master + Adam states):

Memory \(\approx 2P\) (bf16 weights) + \(4P\) (fp32 master) + \(8P\) (Adam \(m, v\)) + activations + gradients.

For 70B: \(\approx 14P\) bytes \(\approx 980\) GB just for state. ZeRO-3 / FSDP shards optimizer + grads + params across \(D\) devices: state per device \(\approx 14P/D\).

14.3 Parallelism composition (3D / 4D)

Total devices \(N = DP\cdot TP\cdot PP\cdot SP\cdot EP\). Effective batch = micro-batch \(\times\) DP replicas \(\times\) pipeline microbatches per step.

Tensor parallelism (Megatron): split MLP columns (\(A = [A_1, A_2]\), all-reduce after second linear); split attention heads. Communication: \(O(\text{batch}\cdot\text{seq}\cdot d/TP)\) all-reduce per layer.

Pipeline parallelism (1F1B / interleaved): bubble fraction \(\approx (PP - 1)/(M + PP - 1)\) for \(M\) microbatches.

Sequence parallelism: along the seq dim for activations of LayerNorm / dropout, complements TP.

14.4 Mixed precision and FP8

BF16: 8-bit exp, 7-bit mantissa; same dynamic range as FP32 but lower precision. FP8 formats: E4M3 (forward, narrower range, more precision), E5M2 (gradients, wider range). Per-tensor or per-block (per-128-block) scaling factor \(s\):

\[\hat{x} = \mathrm{round}(x/s\cdot Q_{\max})\cdot s/Q_{\max}.\]

Loss scaling for FP16: scale loss by \(S\), divide gradients by \(S\) before optimizer.

14.5 Quantization

Symmetric per-tensor: \(q = \mathrm{clip}(\mathrm{round}(x/s), -Q, Q)\), \(s = \max|x|/Q\).

GPTQ: layer-wise reconstruction with the inverse Hessian \(H^{-1} = (2XX^\top)^{-1}\), quantize columns one at a time and update remaining weights to compensate:

\[\Delta W = -\frac{(W_q - W)\cdot[H^{-1}]_{:,k}}{[H^{-1}]_{kk}}.\]

AWQ: salient-channel-aware per-channel scaling \(s\in\mathbb{R}^d\), minimize \(\|W\mathrm{diag}(s^{-1})(\mathrm{diag}(s)X) - WX\|\) on a calibration set; then symmetric quantize \(W\mathrm{diag}(s^{-1})\).

Activation-outlier handling: SmoothQuant migrates outlier scale from activations to weights \(W' = W\mathrm{diag}(s)\), \(X' = X\mathrm{diag}(s^{-1})\). QuaRot/SpinQuant apply learnable orthogonal rotations.

14.6 Diffusion-specific inference math

Number of NFEs (network function evals). DDPM: 1000. DDIM: 20–50. DPM-Solver-2/3: \(\sim 10\)–20. UniPC: 5–10. Consistency / LCM / DMD2: 1–4.

14.7 Speculative decoding

Draft model \(q\) proposes \(k\) tokens; target \(p\) verifies in one forward pass. Acceptance probability for token \(t\) is \(\min(1, p(t|\cdot)/q(t|\cdot))\); rejected position resampled from \(\mathrm{normalize}(\max(0, p - q))\). Expected speedup \(\approx\mathbb{E}[\text{accepted}] + 1\) tokens per target call.

14.8 Cost / capacity planning sketch

For \(C\) training FLOPs at hardware FLOPs/s \(F\) with utilization \(u\) (typically 0.4–0.55 on H100/B200): wall time \(T = C/(N\cdot F\cdot u)\). Chinchilla-like scaling: compute-optimal \(D\approx 20P\) training tokens for an LLM; for VLMs, image tokens count and image-vs-text mix is a free hyperparameter, often pretrain with \(\sim 30\)–50% image tokens.

14.9 Modern attention extensions

KV cache size: per token \(= 2\cdot L\cdot H\cdot d_h\) floats (\(L\) layers, \(H\) heads). Compression: MQA (1 KV head), GQA (\(H_{kv} < H\) KV heads), MLA (DeepSeek; low-rank latent KV).

Ring attention / sequence parallelism: split sequence across \(P\) devices; compute attention block-by-block while passing \(K, V\) around a ring. Total comm \(O(N\cdot d)\) vs \(O(N^2)\) activations.

Sliding window + sink tokens (StreamingLLM, Mistral): keep first \(k\) tokens (sinks) plus a window \(w\) of recent tokens to handle effectively unbounded streams.

Appendix A: Numbers Worth Memorizing

Appendix B: Five-Minute Derivations You Must Be Able To Do Cold

  1. ELBO from Jensen \(\to\) VAE objective.
  2. ELBO \(\to\) DDPM simplified loss (the variance-cancelling step).
  3. DDPM noise prediction \(\Leftrightarrow\) score matching: \(\epsilon_\theta = -\sigma_t s_\theta\).
  4. DPO from KL-constrained RLHF: closed-form policy \(\to\) Bradley–Terry on log-ratios.
  5. Policy gradient theorem from \(J(\theta) = \sum_s d^\pi(s)\sum_a\pi(a|s)Q^\pi(s, a)\).
  6. PPO clip \(\to\) trust region intuition.
  7. GRPO advantage and why removing the value head reduces variance/cost.
  8. Flow matching from continuous normalizing flows: \(\partial_t p_t + \nabla\cdot(p_t u_t) = 0\).
  9. Eckart–Young in two lines (SVD + orthogonal decomposition).
  10. NeRF discretization from the volume rendering equation (alpha-compositing).