Computer Vision — Principal Deep Dive — Senior-Principal Interview Prep

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

Part 1 — Mathematical and Classical CV Foundations

A principal-level interviewer doesn't ask whether you can multiply matrices — they ask why you'd choose SVD over eigendecomposition for a particular problem, or how a tiny perturbation to a calibration matrix propagates through bundle adjustment. This chapter is structured around the few primitives the entire field is built from. If you own them in depth, you can derive most modern systems on demand.

1.1 Linear algebra you must own in your sleep

1.1.1 Singular value decomposition (SVD)

For any \(A\in\mathbb{R}^{m\times n}\) of rank \(r\), there exist orthogonal matrices \(U\in\mathbb{R}^{m\times m}\), \(V\in\mathbb{R}^{n\times n}\) and a diagonal matrix \(\Sigma\in\mathbb{R}^{m\times n}\) with non-negative entries \(\sigma_1\ge\sigma_2\ge\cdots\ge\sigma_r>0=\sigma_{r+1}=\cdots\) such that

\[A = U\Sigma V^\top = \sum_{i=1}^{r}\sigma_i\, u_i v_i^\top.\]

The columns of \(U\) (\(V\)) are the left (right) singular vectors.

Geometric reading. Think of \(A:\mathbb{R}^n\to\mathbb{R}^m\) as a sequence of three operations: rotate input by \(V^\top\) (orthonormal change of basis), scale axes by \(\Sigma\), then rotate output by \(U\). So the unit ball in \(\mathbb{R}^n\) maps to an axis-aligned ellipsoid in \(\mathbb{R}^m\) with semi-axes \(\sigma_i u_i\).

Why we love it. SVD always exists and is numerically the most stable factorization. It gives you all four fundamental subspaces:

Eckart–Young–Mirsky. The best rank-\(k\) approximation of \(A\) in any unitarily invariant norm is

\[A_k = \sum_{i=1}^{k}\sigma_i u_i v_i^\top, \qquad \|A-A_k\|_F^2 = \sum_{i>k}\sigma_i^2, \qquad \|A-A_k\|_2 = \sigma_{k+1}.\]

Proof sketch (Frobenius). Pythagoras: for any rank-\(k\) \(B\), \(\|A-B\|_F^2 \ge \sum_{i>k}\sigma_i^2\) with equality iff \(B\) takes the top-\(k\) singular triplets. The proof uses the Courant–Fischer min-max characterization of singular values applied to the difference.

Key

Whenever you hear "compress / approximate / denoise / project / fit", think SVD. PCA is just SVD on centered data. LoRA is rank-\(k\) deltas from SVD-style reasoning. Robust PCA decomposes \(A = L + S\) with \(L\) low-rank and \(S\) sparse via convex relaxation.

PCA from first principles. For centered data \(X\in\mathbb{R}^{n\times d}\), the covariance \(C = \tfrac{1}{n-1}X^\top X = W\Lambda W^\top\) gives principal axes \(W\). Equivalently, SVD \(X = U\Sigma V^\top\) gives \(W = V\) (and \(\Lambda = \Sigma^2/(n-1)\)).

Project onto \(k\) components: \(Z = XV_k\). Reconstruction \(\hat{X} = ZV_k^\top\) minimizes Frobenius reconstruction error.

Interview probe. "Your dataset has 100k features and 1k samples. Do you compute the covariance, or do something smarter?" You should compute the \(1k\times 1k\) Gram \(XX^\top\), eigendecompose it, then map eigenvectors back via \(v_i = X^\top u_i/\sigma_i\). This is the dual / kernel-trick PCA pattern.

Pseudo-inverse and least squares. The Moore–Penrose pseudo-inverse:

\[A^+ = V\Sigma^+ U^\top, \qquad \Sigma^+_{ii} = \begin{cases}1/\sigma_i & \sigma_i > 0\\ 0 & \text{else.}\end{cases}\]

For full column rank: \(A^+ = (A^\top A)^{-1}A^\top\). For full row rank: \(A^+ = A^\top(AA^\top)^{-1}\). Always: \(A^+ b\) is the minimum-norm least-squares solution to \(\min_x\|Ax-b\|\).

Watch out

Forming \(A^\top A\) explicitly squares the condition number. For ill-conditioned \(A\) (skinny tall matrices in CV are often near-singular), use QR or SVD on \(A\) directly. Never normal equations on a rank-deficient system.

1.1.2 Eigendecomposition and matrix functions

For a symmetric \(S\), \(S = Q\Lambda Q^\top\) with \(Q\) orthogonal. Useful operator interpretations:

For general (non-symmetric) \(A\), use SVD; eigendecomposition may not even exist.

1.1.3 Matrix calculus you'll actually need

Memorize the differentials, then read off Jacobians/gradients:

\[d(AB) = dA\,B + A\,dB, \quad d(A^{-1}) = -A^{-1}\,dA\,A^{-1}, \quad d\log\det A = \mathrm{tr}(A^{-1}dA).\]

For vector \(x\) and scalar \(f(x) = x^\top M x\): \(\nabla f = (M + M^\top)x\); if \(M\) symmetric, \(2Mx\).

For a softmax \(p_i = e^{z_i}/\sum_j e^{z_j}\), the Jacobian is \(J = \mathrm{diag}(p) - pp^\top\), and the CE loss gradient w.r.t. logits is the celebrated \(\nabla_z\mathcal{L} = p - y\) (one-hot \(y\)).

1.1.4 Numerical stability primitives

1.2 Probability and information theory

1.2.1 Distributions you'll see often

Gaussian \(\mathcal{N}(\mu,\Sigma)\) with density \(\tfrac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}\exp\!\big(-\tfrac12(x-\mu)^\top\Sigma^{-1}(x-\mu)\big)\).

Affine closure: \(Ax + b \sim \mathcal{N}(A\mu+b,\, A\Sigma A^\top)\). Conditional Gaussian (essential for Kalman):

\[x_1\mid x_2 \sim \mathcal{N}\big(\mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2),\ \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}\big).\]

Categorical/multinomial; Bernoulli; Beta–Binomial conjugacy; Dirichlet–multinomial; Gamma–Poisson. Know which prior is conjugate to which likelihood; you'll see these in DPO derivations and reward models.

1.2.2 Change of variables

For invertible \(f\) and \(z = f(x)\): \(p_z(z) = p_x(x)\,|\det J_{f^{-1}}(z)|\). Normalizing flows use this; in flow matching, we instead match a vector field instead of integrating the Jacobian (cheaper).

1.2.3 KL, Jensen, ELBO

KL divergence \(\mathrm{KL}(p\,\|\,q) = \mathbb{E}_{x\sim p}[\log p(x) - \log q(x)] \ge 0\), with equality iff \(p = q\) a.e.

Jensen for a concave \(f\): \(\mathbb{E}[f(X)] \le f(\mathbb{E}[X])\). Apply to \(\log\):

\[\log p(x) = \log\int q(z)\frac{p(x,z)}{q(z)}\,dz \ge \mathbb{E}_{z\sim q}\Big[\log\frac{p(x,z)}{q(z)}\Big].\]

The RHS is the ELBO. Decompose:

\[\log p(x) = \mathbb{E}_q[\log p(x\mid z)] - \mathrm{KL}(q(z\mid x)\,\|\,p(z)) + \mathrm{KL}(q(z\mid x)\,\|\,p(z\mid x)),\]

the last term is the variational gap. VAEs maximize the first two; diffusion replaces the encoder by a fixed forward process.

Interview probe. "Derive the ELBO and identify each term's role." This is asked nearly every research-deep round. Practice it under 90 seconds.

1.2.4 Score and Fisher

Score function \(\nabla_x\log p(x)\). Fisher information \(\mathcal{I}(\theta) = \mathbb{E}\big[(\nabla_\theta\log p_\theta(X))(\nabla_\theta\log p_\theta(X))^\top\big]\). Cramér–Rao: variance of any unbiased estimator \(\hat\theta\) is bounded below by \(\mathcal{I}(\theta)^{-1}\).

Why this matters in CV. Diffusion is score matching; understanding why \(\mathbb{E}_p[(s_\theta(x) - \nabla_x\log p)^2]\) has the same gradient as the denoising loss is the cleanest way to internalize what a diffusion model is doing.

1.2.5 Maximum likelihood vs MAP vs Bayes

Variational inference approximates the posterior with \(q_\phi(\theta)\); ELBO maximization is variational MAP for \(\theta\) when \(q\) is a delta.

1.3 Optimization theory

1.3.1 Convexity, smoothness, strong convexity

\(f\) convex: \(f(\lambda x + (1-\lambda)y) \le \lambda f(x) + (1-\lambda)f(y)\) for \(\lambda\in[0,1]\). \(f\) is \(L\)-smooth (\(\|\nabla f(x)-\nabla f(y)\|\le L\|x-y\|\)), equivalently \(f(y)\le f(x)+\nabla f(x)^\top(y-x)+\tfrac{L}{2}\|y-x\|^2\). \(\mu\)-strongly convex: \(f(y)\ge f(x)+\nabla f(x)^\top(y-x)+\tfrac{\mu}{2}\|y-x\|^2\).

1.3.2 Gradient descent rates (clean)

1.3.3 Stochastic gradient and variance

SGD: \(x_{t+1} = x_t - \eta g_t\) with \(\mathbb{E}[g_t] = \nabla f(x_t)\) and \(\mathbb{E}\|g_t - \nabla f\|^2 \le \sigma^2\). Standard rate: \(O(1/\sqrt{T})\) in the convex case; mini-batch shrinks \(\sigma^2\) by batch size.

1.3.4 KKT conditions

For \(\min f(x)\) s.t. \(g_i(x)\le 0\), \(h_j(x)=0\), Lagrangian \(\mathcal{L} = f + \mu^\top g + \nu^\top h\). KKT (necessary at optimum, sufficient under convexity + Slater):

\[g_i(x)\le 0,\quad h_j(x)=0,\quad \mu_i\ge 0,\quad \nabla f + \sum_i\mu_i\nabla g_i + \sum_j\nu_j\nabla h_j = 0,\quad \mu_i g_i(x)=0.\]

1.3.5 Modern DL optimizers, derived intuitions

Adam and AdamW. Adam:

\[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t,\quad v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2,\quad \hat m_t = \tfrac{m_t}{1-\beta_1^t},\ \hat v_t = \tfrac{v_t}{1-\beta_2^t},\]

an empirical second moment that adapts the learning rate per parameter. AdamW decouples weight decay from the moment ratio:

\[\theta_t = (1-\eta\lambda)\theta_{t-1} - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.\]

This is mathematically distinct from \(\ell_2\) regularization combined with Adam, and consistently better for transformers.

Watch out

Common mistake: setting \(\beta_2 = 0.999\) with very large batch size. Effective averaging window becomes inadequate; modern LLM/VLM training often uses \(\beta_2\in[0.95, 0.98]\) to track second-moment shifts faster.

Lion. Update sign of momentum, no second moment:

\[c_t = \beta_1 m_{t-1} + (1-\beta_1)g_t,\quad \theta_t = \theta_{t-1} - \eta\,\mathrm{sign}(c_t),\quad m_t = \beta_2 m_{t-1} + (1-\beta_2)g_t.\]

Memory equal to one (not two) optimizer states; surprisingly robust at large scale. The price is sensitivity to learning-rate magnitude.

Muon (matrix-aware). Treat hidden weight matrices as matrices, not flat vectors, and orthogonalize the momentum step. Newton–Schulz iteration approximates the polar factor \(U_t\) of momentum \(M_t\):

\[X \leftarrow \tfrac{15}{8}X - \tfrac{5}{4}XX^\top X + \tfrac{3}{8}(XX^\top)^2 X, \qquad W \leftarrow W - \eta\,U_t.\]

Empirically faster wall-clock convergence than AdamW at scale; used in modern open LLM/VLM runs.

Schedules. Cosine decay: \(\eta_t = \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})\big(1+\cos(\pi t/T)\big)\), with warmup then optional cooldown; convenient for resuming and for compute-vs-quality trade studies.

µP / µTransfer. Width-aware parameterization where you scale (a) initialization variance, (b) learning rate, (c) optional update multipliers as functions of fan-in width. Result: optimal hyperparameters chosen on a 200M proxy carry over almost exactly to a 7B target. Crucial in modern training pipelines because hyperparameter sweeps at full scale are unaffordable.

1.4 Geometry: Lie groups and rigid motions

1.4.1 Why Lie groups?

Rotations live on the manifold \(SO(3)\); the configuration space of a rigid body lives on \(SE(3)\). Parameterizing them naively (e.g., 9 numbers for \(R\)) breaks gradient methods because perturbations leave the manifold. Lie groups give us:

1.4.2 SO(3) in detail

The hat/skew operator for \(\omega = (\omega_1,\omega_2,\omega_3)\):

\[[\omega]_\times = \begin{pmatrix} 0 & -\omega_3 & \omega_2\\ \omega_3 & 0 & -\omega_1\\ -\omega_2 & \omega_1 & 0\end{pmatrix}.\]

Lie bracket on \(\mathfrak{so}(3)\) matches the cross product: \(\big[[\omega_1]_\times,[\omega_2]_\times\big] = [\omega_1\times\omega_2]_\times\).

Exponential map (Rodrigues). For \(\omega\in\mathbb{R}^3\), \(\theta = \|\omega\|\), \(\hat\omega=\omega/\theta\):

\[R = \exp([\omega]_\times) = I + \sin\theta\,[\hat\omega]_\times + (1-\cos\theta)\,[\hat\omega]_\times^2.\]

Log map. Given \(R\in SO(3)\):

\[\theta = \arccos\!\Big(\tfrac{\mathrm{tr}(R)-1}{2}\Big), \qquad [\omega]_\times = \frac{\theta}{2\sin\theta}\big(R - R^\top\big).\]

Singularities at \(\theta=0\) and \(\theta=\pi\): handle with Taylor near zero and quaternion interpolation.

Right-perturbation Jacobian. For optimization, perturb \(R\) on the right: \(R \leftarrow R\exp([\delta]_\times)\). Then \(\tfrac{\partial}{\partial\delta}\big|_{\delta=0} Rp = -R[p]_\times\). In code: \(J = -R[p]_\times\). For a unit-quaternion-parameterized rotation, the Jacobian is \(\partial R/\partial q\) followed by chain rule.

1.4.3 SE(3)

An element \(T = \begin{pmatrix}R & t\\ 0 & 1\end{pmatrix}\) with algebra \(\xi = (\rho,\omega)\in\mathbb{R}^6\). Exponential:

\[\exp(\hat\xi) = \begin{pmatrix}\exp([\omega]_\times) & V\rho\\ 0 & 1\end{pmatrix}, \quad V = I + \frac{1-\cos\theta}{\theta^2}[\omega]_\times + \frac{\theta-\sin\theta}{\theta^3}[\omega]_\times^2.\]

1.4.4 Quaternions for engineers

Unit quaternion \(q = (w,v)\) with \(w^2 + \|v\|^2 = 1\) encodes a rotation by \(2\arccos w\) about \(v/\|v\|\).

Composition: Hamilton product. Rotate \(p\in\mathbb{R}^3\) via \(p' = qpq^{-1}\) where \(p\) is embedded as \((0,p)\).

Slerp:

\[\mathrm{slerp}(q_0,q_1;t) = \frac{\sin((1-t)\Omega)}{\sin\Omega}q_0 + \frac{\sin(t\Omega)}{\sin\Omega}q_1, \qquad \cos\Omega = q_0\cdot q_1,\]

which is constant-velocity in angle.

Watch out

Quaternions have a sign ambiguity: \(q\) and \(-q\) encode the same rotation. Always re-sign so that \(q\cdot q_{\text{prev}}\ge 0\) when filtering, otherwise jumps appear.

1.5 Multi-view geometry

1.5.1 Pinhole projection and intrinsics

World point \(X\in\mathbb{R}^3\) projects to image plane via

\[\lambda\begin{pmatrix}u\\v\\1\end{pmatrix} = K[R\mid t]\begin{pmatrix}X\\1\end{pmatrix}, \qquad K = \begin{pmatrix}f_x & s & c_x\\ 0 & f_y & c_y\\ 0 & 0 & 1\end{pmatrix}.\]

Intrinsics \(K\) has 5 DoF (4 if \(s=0\)). Extrinsics \([R\mid t]\) has 6 DoF.

1.5.2 Distortion

Brown–Conrady (radial–tangential):

\[x_d = x(1 + k_1 r^2 + k_2 r^4 + k_3 r^6) + 2p_1 xy + p_2(r^2 + 2x^2), \qquad r^2 = x^2 + y^2,\]

similarly for \(y_d\). Fisheye lenses use Kannala–Brandt (polynomial in the incidence angle).

1.5.3 Calibration (Zhang's method)

With a planar checkerboard observed from \(n\) poses, each plane induces a homography \(H = K[r_1, r_2, t]\) from world plane points to image. Constraints from \(r_1, r_2\) orthonormal:

\[h_1^\top B h_2 = 0, \qquad h_1^\top B h_1 = h_2^\top B h_2, \qquad B = K^{-\top}K^{-1}.\]

Linear solve for \(B\) across \(n\ge 3\) views, then Cholesky factor for \(K\). Refine with non-linear minimization of reprojection error including distortion.

1.5.4 Epipolar geometry

Two cameras with relative pose \((R,t)\) observing the same point. The epipolar constraint:

\[x_2^\top E\, x_1 = 0, \qquad E = [t]_\times R.\]

\(E\) has rank 2, two equal non-zero singular values; 5 DoF. The fundamental matrix accounts for unknown intrinsics, \(F = K_2^{-\top}E K_1^{-1}\), 7 DoF.

Eight-point algorithm (normalized). Each correspondence \((u,v)\leftrightarrow(u',v')\) gives a row in the linear system \(Af = 0\) where \(f = \mathrm{vec}(F)\). With \(n\ge 8\) correspondences, solve via SVD of \(A\) (last right singular vector). Enforce \(\det F = 0\) by SVD on \(F\), zeroing the smallest singular value, and reconstructing.

Hartley normalization is essential: shift each image so centroid is at origin and scale so average distance is \(\sqrt 2\). Without it, the 8-point is numerically catastrophic for typical pixel coordinates.

Five-point algorithm (Nistér). Calibrated case: solve for \(E\) from 5 correspondences. Each correspondence gives one linear constraint on \(E\) (10 unknowns, defined up to scale, so 9). With 5 constraints, \(E\) lives in a 4-dimensional subspace; the 10 polynomial constraints (rank-2 + two equal singular values) yield up to 10 candidate solutions, found via Gröbner basis or Macaulay matrix.

1.5.5 Triangulation

Linear (DLT). For \(x\leftrightarrow x'\) with cameras \(P,P'\) (rows \(p^{i\top}\)):

\[A = \begin{pmatrix} u\,p^{3\top} - p^{1\top}\\ v\,p^{3\top} - p^{2\top}\\ u'\,p'^{3\top} - p'^{1\top}\\ v'\,p'^{3\top} - p'^{2\top}\end{pmatrix}, \qquad \hat X = \arg\min_{\|X\|=1}\|AX\|\ \text{(SVD)}.\]

Mid-point and optimal. Mid-point: midpoint of the closest segment between two rays. Optimal (Hartley–Sturm): minimize the geometric reprojection error in both images jointly via 6th-order polynomial.

1.5.6 PnP (Perspective-n-Point)

Given \(n\) 3D–2D correspondences \(\{X_i\leftrightarrow x_i\}\) and \(K\), recover \(R,t\). Minimal: P3P, 4 candidate solutions; disambiguate with a 4th point. Non-minimal: EPnP \(O(n)\) closed-form via control-point parameterization. Iteratively refine via Gauss–Newton on reprojection error.

1.5.7 Bundle adjustment, properly explained

Joint optimization over \(M\) camera poses and \(N\) 3D points:

\[\min_{\{T_j\},\{X_i\}}\ \sum_{(i,j)\in\Omega}\rho\Big(\big\|\pi(T_j, X_i) - x_{ij}\big\|_{\Sigma_{ij}}^2\Big).\]

Levenberg–Marquardt. The LM step interpolates between Gauss–Newton (\(\lambda\to 0\)) and gradient descent (\(\lambda\to\infty\)):

\[\big(J^\top J + \lambda\,\mathrm{diag}(J^\top J)\big)\,\Delta\theta = -J^\top r.\]

Damping \(\lambda\) updated multiplicatively based on whether the step decreased the loss.

Sparsity and Schur complement. Stack parameters into camera block \(C\) and point block \(P\). The normal equations have block structure:

\[\begin{pmatrix}H_{CC} & H_{CP}\\ H_{CP}^\top & H_{PP}\end{pmatrix}\begin{pmatrix}\Delta C\\ \Delta P\end{pmatrix} = -\begin{pmatrix}b_C\\ b_P\end{pmatrix}.\]

\(H_{PP}\) is block-diagonal (each point is observed by some cameras independently). Eliminate the points block via the Schur complement:

\[\big(H_{CC} - H_{CP}H_{PP}^{-1}H_{CP}^\top\big)\Delta C = -b_C + H_{CP}H_{PP}^{-1}b_P.\]

The reduced camera system is dense but small. Then back-substitute for points.

Interview probe. "A bundle adjustment of 100 cameras and 50,000 points: estimate the size of the densest matrix you have to invert if you do everything right." Reduced camera system is \(6M\times 6M = 600\times 600\) if poses are 6-DoF, plus per-point \(3\times 3\) blocks. Without Schur complement, the full Hessian would be \(150{,}600\times 150{,}600\) — prohibitive.

1.5.8 RANSAC family

Vanilla RANSAC. 1. Sample \(s\) points (minimal sample for model). 2. Fit model. 3. Count inliers within threshold \(\tau\). 4. Repeat \(N\) times; return model with most inliers.

For inlier ratio \(w\), probability of an all-inlier sample is \(w^s\). To achieve confidence \(p\) that we sampled at least once an all-inlier set:

\[N = \frac{\log(1-p)}{\log(1 - w^s)}.\]

Adapt \(N\) on the fly as the \(w\) estimate improves.

Variants. - LO-RANSAC. Local optimization on inliers between RANSAC iterations. - MLESAC. Maximize log-likelihood of the inlier/outlier mixture model rather than inlier count. - MAGSAC++. Marginalize over the noise scale \(\sigma\) instead of choosing a hard threshold; smooth weighting more robust. - GC-RANSAC. Spatial coherence via graph-cut on inliers between iterations. - USAC framework. Modular template combining all of the above.

1.6 Modern visual SLAM and learned matching

1.6.1 Direct vs feature-based SLAM

Feature-based (ORB-SLAM3, VINS-Mono): detect/match features per frame, optimize reprojection error, and run loop closure with bag-of-words. Direct (DSO, LSD-SLAM): minimize photometric error directly across frames, joint with depth and pose.

Photometric residual:

\[r_{ij}(T, d) = I_2\big(\pi(T\cdot\pi^{-1}(x_i, d_i))\big) - I_1(x_i),\]

optimized over poses and depths jointly. Photometric is dense and information-rich but assumes brightness constancy.

1.6.2 Visual-inertial

Pre-integrated IMU residuals between keyframes give relative motion priors that make VO drift bounded. Optimize the joint cost:

\[\sum_{(i,j)}\big\|r^{\text{visual}}_{ij}\big\|^2 + \big\|r^{\text{IMU}}_{ij}\big\|^2.\]

1.6.3 Loop closure

Place recognition produces a candidate match between distant frames; geometric verification (PnP + RANSAC) confirms; pose graph optimization corrects accumulated drift via \(SE(3)\) edges. Modern open libraries: GTSAM, g2o, Ceres.

1.6.4 Differentiable rendering for SLAM

NeRF-SLAM and the GS-SLAM family (MonoGS, GS-SLAM, Splat-SLAM, Photo-SLAM) replace the explicit map with a learned radiance field or 3D Gaussians, optimized jointly with poses by photometric loss. The key benefit: instead of choosing a sparse representation up front, you let the rendering loss decide what to model. Trade-off: heavier compute, slower than feature SLAM at the limit, but much richer reconstruction.

1.6.5 Learned matchers

1.6.6 Sparse-to-dense reconstruction modern stack

  1. Detect and match features (SuperPoint+LightGlue) or directly run MASt3R.
  2. Initialize SfM (incremental or global) with COLMAP or hloc, or feed-forward via VGGT/Spann3R.
  3. Run dense MVS (PatchMatch in COLMAP, or learned MVSNet variants) for depth maps; or feed-forward with DUSt3R.
  4. Surface reconstruction (Poisson, TSDF fusion) or proceed directly to neural representation (NeRF, 3DGS).

1.7 What an interviewer is really probing in this chapter

The truthful summary is: can you do calculus and linear algebra fluently in 3D? Modern systems (3DGS, DUSt3R, neural-SLAM) are differentiable forward models stacked on top of the same geometry. If you understand pinhole projection, Lie groups, bundle adjustment, and the chain rule, you can read any 3D paper from the past decade in one sitting.

Interview probe. Defaults you should be able to whiteboard from memory: derive the essential matrix from rigid motion, derive the homography from a planar scene, derive the Schur complement, write the LM normal equations, write Rodrigues formula, write the photometric residual, derive the Hessian of a Gaussian negative log-likelihood, define the conditional Gaussian.

Part 2 — Deep Learning Core for Vision

This chapter is the substrate on which everything else rests. By 2026, every CV pipeline of consequence is either a Transformer, a hybrid, or a diffusion model on top of a Transformer. Understanding the architecture, the training dynamics, and the major task heads is non-negotiable.

2.1 From CNNs to Transformers: a working history

2.1.1 ResNet and the residual revolution

A residual block computes \(y = x + F(x)\). Why does this train? Three intuitions.

  1. Identity mapping is the default. If \(F\) is initialized small, the network starts as the identity and learns deviations. Gradients flow undisturbed through the skip path: \(\tfrac{\partial y}{\partial x} = I + \tfrac{\partial F}{\partial x}\).
  2. Ensemble interpretation. A network with \(N\) residual blocks computes a sum over \(2^N\) paths of varying depth (Veit et al.); deeper paths refine the predictions of shallower ones. Removing single blocks at test time only marginally hurts accuracy — evidence the network behaves like an ensemble.
  3. Loss landscape smoothing. Empirically, residual networks have substantially smoother loss surfaces (Li et al., visualizations).

Bottleneck design: \(1\times1\) down-project \(\to 3\times3\) conv \(\to 1\times1\) up-project. Reduces FLOPs at the same capacity for deep networks. See ResNet.

Watch out

ResNet's batchnorm-after-conv-before-add is brittle in distributed settings (small per-device batches). Modern variants (ResNet-RS, NFNet) explore replacing BN with group/layer norm; ConvNeXt drops BN entirely.

2.1.2 ConvNeXt and the modernization argument

ConvNeXt asks: if you start from ResNet-50 and apply, one at a time, every modernization that the Swin Transformer paper assumed — larger kernels (\(7\times7\) depthwise), inverted bottleneck (à la MobileNetV2), GELU, fewer activations / norms per block, LayerNorm replacing BN, AdamW + augment + label smoothing — where does pure CNN performance end up?

Answer: matches Swin at the same FLOPs. ConvNeXt V2 then adds Fully Convolutional Masked Autoencoder (FCMAE) pretraining and Global Response Normalization (GRN, a feature-channel competition that rescues feature collapse seen with MAE on conv backbones).

The lesson is not that conv beats attention or vice versa. It is that training recipe matters as much as architecture, and that mixing-channel vs mixing-token primitives is more important than the precise mixer (depthwise conv, attention, MLP-mixer, Mamba/SSM).

2.1.3 Vision Transformer (ViT)

Patchify image \(X\in\mathbb{R}^{H\times W\times 3}\) into \(N = HW/p^2\) patches of \(p\times p\). Linear embed each into \(\mathbb{R}^d\). Prepend a learnable [CLS] token. Add positional embeddings. Apply \(L\) Transformer blocks. Classify from [CLS].

The block in detail. Pre-norm convention:

\[z' = z + \mathrm{MHSA}(\mathrm{LN}(z)), \qquad z'' = z' + \mathrm{MLP}(\mathrm{LN}(z')).\]

Multi-head self-attention (MHSA), \(H\) heads, head dim \(d_h = d/H\):

\[Q = XW_Q,\ K = XW_K,\ V = XW_V, \qquad \mathrm{Attn}(Q,K,V) = \mathrm{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_h}}\Big)V.\]

MLP block, expansion ratio 4: \(\mathrm{MLP}(z) = W_2\,\sigma(W_1 z)\) with \(W_1\in\mathbb{R}^{4d\times d}\), \(W_2\in\mathbb{R}^{d\times 4d}\). Modern blocks replace \(W_2\sigma(W_1 z)\) with SwiGLU: \((W_3 z)\odot\sigma(W_1 z)\) then \(W_2\).

If \(q_i, k_j\sim\mathcal{N}(0, I_{d_h})\) i.i.d., \(\mathrm{Var}(q^\top k) = d_h\). Logits proportional to \(q^\top k\) would saturate the softmax (most mass on argmax). Dividing by \(\sqrt{d_h}\) keeps logits at unit scale.

ViT vs CNN inductive biases. ViT has weak inductive biases (no locality, no translation equivariance). It needs more data or stronger augmentation to match conv-net accuracy at small scale; given enough data (300M+ images), the lack of bias becomes a feature, not a bug.

2.1.4 Swin and hierarchical attention

Swin computes attention within non-overlapping \(M\times M\) windows (linear in image size), then shifts the windows by \(M/2\) in the next layer to enable cross-window communication. Hierarchical: patch-merging halves spatial resolution and doubles channels every stage, mirroring the CNN pyramid pattern.

Swin V2: log-spaced relative positional bias for resolution generalization, post-norm with residual for stability at scale, scaled cosine attention.

2.1.5 Mamba and state-space models in vision

Selective SSMs compute, in linear time, a recurrence

\[h_t = \bar A\, h_{t-1} + \bar B\, x_t, \qquad y_t = C\, h_t,\]

where the matrices depend on the input via a selection mechanism. In vision: Vision Mamba and VMamba scan tokens in 2D (multi-direction scans). Strong at long sequences (e.g., high-res images, long videos) due to linear complexity.

2.1.6 Hybrids that ship

★ 2026 SOTA update — Deep Learning Core for Vision

2.2 Position encodings: the unsung hero

2.2.1 Why position matters

Self-attention is permutation-equivariant: \(\mathrm{Attn}(\pi X) = \pi\,\mathrm{Attn}(X)\). Without position info, a cat-on-couch image has the same representation as cat-pixels-and-couch-pixels-shuffled.

2.2.2 Sinusoidal absolute (original Transformer)

\[\mathrm{PE}(p, 2i) = \sin\!\big(p/10000^{2i/d}\big), \qquad \mathrm{PE}(p, 2i+1) = \cos\!\big(p/10000^{2i/d}\big).\]

Property: \(\mathrm{PE}(p+\Delta)\) is a linear function of \(\mathrm{PE}(p)\) and \(\Delta\), encouraging the model to learn relative offsets.

2.2.3 Learned absolute

A learnable table of size \(L\times d\). Hard to extrapolate beyond \(L\).

2.2.4 RoPE (rotary position embedding)

Pair adjacent feature dims \((2i, 2i+1)\) and rotate by angle \(p\theta_i\), \(\theta_i = 10000^{-2i/d}\):

\[q'_p = R_p q_p, \qquad k'_{p'} = R_{p'} k_{p'}, \qquad \langle q'_p, k'_{p'}\rangle = q_p^\top R_{p'-p} k_{p'}\]

depends only on relative position. Extension: 2D-RoPE splits the feature dim into row and column halves and applies 1D RoPE to each. Used in Qwen2-VL, InternVL, FLUX (image diffusion), and modern LLMs.

Long-context RoPE: NTK and YaRN. Naively extending RoPE beyond train context degrades. NTK-aware: scale the base \(b\) such that high-frequency components remain unchanged; YaRN piecewise-rescales by frequency band. Critical for video VLMs reasoning over hour-long inputs.

2.2.5 ALiBi (Attention with Linear Biases)

Add a linear bias \(-m\,|p_i - p_j|\) to attention logits, with head-specific slopes \(m\). No learnable position; extrapolates well.

2.3 Normalization, activation, initialization

2.3.1 Normalization layers

Pre-norm vs post-norm. Post-norm (original Transformer): \(y = \mathrm{LN}(x + f(x))\). Suffers gradient instability at depth > a few dozen layers; needs warmup. Pre-norm: \(y = x + f(\mathrm{LN}(x))\). Stable to hundreds of layers without warmup, but final layer outputs grow with depth (sum of post-norm residuals); some recipes add a final LayerNorm at the end. All large-scale modern stacks use pre-norm.

2.3.2 Activations

ReLU \(\to\) GELU (\(x\Phi(x)\)) \(\to\) SiLU/Swish (\(x\sigma(x)\)) \(\to\) gated variants like SwiGLU and GeGLU. Gated activations consistently outperform pointwise at the cost of one extra matmul; standard in modern LLMs and adopted in MM-DiT (SD3, FLUX).

2.3.3 Initialization

µP / µTransfer. Choose initialization variance, learning rate, and update multipliers as functions of fan-in width such that the optimal hyperparameters for a small width remain optimal as you grow. The result: tune at 200M, transfer to 70B, save 99% of the GPU budget for sweeps. Asked frequently at modern infra-leaning interviews.

2.4 Self-supervised learning, deeply

2.4.1 Why pretrain at all?

Pretraining gives a backbone whose features are useful across many downstream tasks with low-shot fine-tuning. The economic argument is overwhelming: a single pretrained DINOv3 ViT-7B trained once is reused for hundreds of downstream tasks. The scientific argument is that the pretext task structures the representation in ways that align with the manifold of natural images.

2.4.2 Contrastive: SimCLR, MoCo, supervised contrastive

InfoNCE objective. Two augmented views of the same image are positives; other images in the batch are negatives:

\[\mathcal{L}_i = -\log\frac{\exp(\langle z_i, z_i^+\rangle/\tau)}{\sum_k\exp(\langle z_i, z_k\rangle/\tau)}.\]

The temperature \(\tau\) controls sharpness; small \(\tau\) enforces hard negative mining. The loss is a lower bound on mutual information up to \(\log K\) (Poole et al.). See SimCLR.

Why does it work? Two equivalent views: (a) cluster augmentations of the same image and push apart different images; (b) maximize mutual information between two views of the same data. Both views suggest features should be invariant to nuisance augmentations and discriminative across instances.

MoCo's momentum queue. Maintain a queue of \(K\approx 65{,}536\) recent encoded keys and a slowly updated key encoder \(\theta_k \leftarrow m\theta_k + (1-m)\theta_q\) (\(m\approx 0.999\)). Decouples the negative count from the GPU batch size. SimCLR needs giant batches to compete. See MoCo.

2.4.3 Negative-free: BYOL, SimSiam

BYOL has online and target networks; train the online predictor to match the target encoder's projection of a different view, with EMA target update and stop-gradient on the target. Loss:

\[\mathcal{L} = 2 - 2\,\frac{\langle q_\theta(z_1),\, \mathrm{sg}(z_2')\rangle}{\|q_\theta(z_1)\|\,\|z_2'\|}.\]

SimSiam shows the EMA isn't strictly necessary — stop-gradient + predictor suffice. Why no collapse? The predictor breaks the symmetry; the optimization "chases" the target before it can collapse, and the predictor's degree of freedom prevents trivial solutions.

2.4.4 Self-distillation: DINO and DINOv2/v3

Two views, student \(g_\theta\) and teacher \(g_\xi\) (EMA). Soft targets via teacher softmax with low temperature (sharpening), and a running mean subtracted (centering) to prevent collapse onto a constant:

\[\mathcal{L}_{\text{DINO}} = -\,\mathrm{softmax}\!\Big(\frac{g_\xi(v) - c}{\tau_t}\Big)^\top \log\,\mathrm{softmax}\!\Big(\frac{g_\theta(v')}{\tau_s}\Big).\]

See DINO and DINOv2. DINOv2 additions: iBOT (predict masked teacher patch tokens), KoLeo regularizer (ensure diverse coverage of the unit sphere), LayerScale, careful curation of LVD-142M. DINOv3 additions: scale to ViT-7B with 1.7B parameters; Gram-matching loss to prevent dense feature degradation at scale (otherwise long training collapses dense quality even as classification holds).

2.4.5 Masked image modeling: MAE family

MAE: mask 75% of patches; encoder sees only the visible 25%; a lightweight decoder reconstructs all patches in pixel space (with per-patch normalized targets). Loss only on masked patches:

\[\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\|\hat p_i - p_i\|_2^2.\]

Why high mask ratio works: image patches are highly redundant; lower mask leaves an underdetermined task (just averaging neighbors).

Variants: SimMIM predicts raw pixels with a single linear layer; MaskFeat predicts HOG features; EVA / EVA-02 reconstruct frozen CLIP features (semantic targets stronger than pixels); FCMAE / ConvNeXt V2 adapt to convs by running the encoder on dense feature maps with sparse convolution.

2.4.6 JEPA: I-JEPA, V-JEPA, V-JEPA 2

Predict the embedding (via EMA target encoder) of target patches given a context, instead of pixels:

\[\mathcal{L}_{\text{JEPA}} = \sum_{i\in\mathcal{T}}\big\|\hat z_i - \mathrm{sg}\big(f_{\bar\theta}(x)_i\big)\big\|^2.\]

The argument: predicting in latent space lets you ignore irrelevant pixel detail (texture, exact noise) and focus on semantically predictable structure. See I-JEPA; V-JEPA 2 scales this to billions of video frames, predicting in latent space across time.

Key

The four families above are not interchangeable. Pixel reconstruction gives strong dense (segmentation) features, weaker linear-probe accuracy. Contrastive gives strong global features, weaker dense. Self-distillation (DINOv2/v3) gives both, but is finicky and needs careful regularization at scale. JEPA bets that latent prediction is a better optimization target than pixel-perfect reconstruction.

2.4.7 Hybrids and curriculum

DINOv2 is itself a hybrid (DINO loss + iBOT mask prediction + KoLeo). Modern recipes typically combine three signals: contrastive / global, dense reconstruction / mask, and a stability regularizer. This combinatorial design space is where most empirical progress in SSL has come from over the past three years.

2.5 Tasks: detection, segmentation, tracking

2.5.1 Anchors, anchor-free, query-based

2.5.2 Two-stage detectors (Faster R-CNN, Cascade R-CNN)

  1. Backbone (ResNet, ViT) extracts multi-scale features (FPN).
  2. RPN predicts objectness and box offsets per anchor; NMS produces \(\sim 1000\) proposals.
  3. ROIAlign extracts per-proposal features (bilinear, no rounding — avoids quantization).
  4. Heads predict class and refined box per proposal.

Cascade R-CNN: stack \(K\) heads with progressively higher IoU thresholds, each refining the previous, mitigates the IoU–quality mismatch. See also Faster R-CNN.

2.5.3 One-stage / dense detectors

Focal loss (RetinaNet). With easy/hard example imbalance (typically \(\sim 1000{:}1\) for foreground/background), CE drowns in easy negatives. Focal loss:

\[p_t = \begin{cases}p & y=1\\ 1-p & y=0\end{cases}, \qquad \mathrm{FL}(p_t) = -\alpha(1-p_t)^\gamma\log(p_t).\]

Easy examples (\(p_t\to 1\)) get \((1-p_t)^\gamma\to 0\) down-weighting. Defaults \(\alpha = 0.25\), \(\gamma = 2\).

FCOS / ATSS / GFL. FCOS predicts (centerness, class, l/r/t/b distances) per location. ATSS adaptively selects positives by IoU statistics. GFL replaces classification with quality-focal loss and box with distribution focal loss (DFL: predict a categorical distribution over discrete bins; expected value used as the prediction).

2.5.4 DETR and successors

DETR removes anchors and NMS by formulating detection as set prediction. Cost matrix between predictions \(\hat y_i\) and GT \(y_j\):

\[\mathcal{C}_{ij} = -\hat p_i(c_j) + \lambda_{\ell_1}\|\hat b_i - b_j\|_1 + \lambda_{\text{giou}}\big(1 - \mathrm{GIoU}(\hat b_i, b_j)\big).\]

Solve \(\arg\min_\sigma\sum_i\mathcal{C}_{i\sigma(i)}\) with the Hungarian algorithm in \(O(N^3)\). Backprop standard losses on the matched pairs.

Pain points and fixes. - Slow convergence. Deformable DETR replaces dense attention with deformable attention: each query attends to a small set of \(K\) predicted offsets per feature level. - Inefficient matching. DN-DETR / DINO-DETR add denoising of GT during training: feed noised GT boxes as additional queries to stabilize matching. - Final accuracy lag. Co-DETR uses auxiliary one-to-many heads (dense predictions) to provide more positive signal during training, then deactivates for the one-to-one matching at inference. - Real-time. RT-DETR / RT-DETRv2 / D-FINE / DEIM engineer fast variants with hybrid backbones, efficient encoders, and careful loss balancing. Now beating YOLOv9/v10 in many benchmarks at similar speed.

2.5.5 YOLO family in 2026

The line between YOLO and DETR is blurring.

2.5.6 Segmentation

Semantic, instance, panoptic via the universal mask paradigm. Pre-2021 separate stacks; Mask2Former (and OneFormer, kMaX-DeepLab) unify them: predict \(N\) masks \(m_i\in[0,1]^{H\times W}\) and class probabilities \(p_i\in\Delta^{C+1}\) (+1 for "no object"). Final per-pixel semantic class:

\[P(c\mid x) = \sum_{i=1}^{N} p_i(c)\, m_i(x).\]

Train with mask-focal + DICE losses, Hungarian-matched to GT regions.

DICE loss.

\[\mathrm{DICE}(p, y) = 1 - \frac{2\sum_i p_i y_i + \epsilon}{\sum_i p_i + \sum_i y_i + \epsilon}.\]

Differentiable approximation of F1; resilient to class imbalance.

2.5.7 SAM family

SAM accepts prompts (points, boxes, masks, free-form text via CLIP) and produces segmentation masks. Trained on SA-1B (1.1B masks). Key contribution: ambiguity-aware multi-mask outputs (3 candidates, scored by mask IoU prediction).

SAM 2 extends to video with a memory bank: cross-attention to features and predicted masks of past frames; supports point/box prompts on any frame and propagates through the clip. Memory bank typically 8 most recent frames + the prompted anchor frame.

2.5.8 Open-vocabulary detection and segmentation

Replace the closed-set classifier with a text-conditioned open-set classifier:

\[\mathrm{score}_i(c) = \sigma\!\big(\langle f_i, T(c)\rangle/\tau\big),\]

where \(T(\cdot)\) is a CLIP-style text encoder. Train with grounded image-text pairs (GLIP, Grounding DINO).

Grounding DINO 1.5/1.6/Pro. Cross-modal feature enhancer in the encoder, query selection conditioned on text, decoder cross-attends to text + image. Performs phrase grounding (referring expressions) and open-vocabulary detection in one model.

2.5.9 Tracking

Tracking-by-detection (SORT family). Run detector per frame; associate detections to existing tracks via Hungarian matching on (Mahalanobis distance from Kalman prediction + appearance embedding cosine + IoU). DeepSORT adds appearance embeddings; ByteTrack uses low-confidence detections to recover missed tracks; OC-SORT corrects Kalman via observed positions; BoT-SORT adds camera motion compensation. See SORT.

Transformer trackers. TransTrack, MOTR, MeMOTR: track queries propagate across frames; new objects spawn detection queries each frame; Hungarian matching enforces identity continuity.

STM, AOT, DeAOT, Cutie, SAM 2: maintain a memory of past frames' features and masks; current frame queries cross-attend; mask decoder produces segmentation per object. SAM 2 has effectively become the default in 2026.

2.6 Training tricks that actually move metrics

2.6.1 Augmentation

2.6.2 Regularization

2.6.3 EMA model averaging

Maintain \(\bar\theta_t = \alpha\bar\theta_{t-1} + (1-\alpha)\theta_t\), evaluate with \(\bar\theta\). Decay \(\alpha\sim 0.999\)–\(0.9999\). Worth +1–3 mAP for free in many CV settings; required for diffusion sample quality.

2.6.4 Distillation

Three classes:

2.6.5 Mixed precision and FP8

BF16 default for training (8-bit exp, 7-bit mantissa, same range as FP32). FP16 needs loss scaling. FP8 (E4M3 forward, E5M2 backward) on H100/B200 doubles throughput but requires per-tensor or per-block scaling factors and careful kernel design. NVIDIA Transformer Engine handles much of this.

2.6.6 Gradient clipping and stability

Global norm clip \(g \leftarrow g\cdot\min(1, \tau/\|g\|)\), \(\tau\sim 1.0\). Skip-bad-batches: if gradient norm exceeds \(K\sigma\) above EMA, skip the step. Save model state every \(N\) steps; on persistent loss spike, rollback and resume from a recent checkpoint.

2.7 What an interviewer is really probing

The point of this chapter is the same one over and over: the modern Transformer is a meta-architecture. Whether you call it a vision Transformer, a U-Net with attention, a DiT, or a VLA, the pattern of (token, position-encode, attend, MLP, normalize) repeats. What distinguishes principal-level engineers is fluency: they can sketch any of these stacks in seconds, identify what changes, and reason about which trade-off is being made. If you can do that, you'll handle any architecture-design round.

Interview probe. "Walk me through the transformations applied to a single pixel from the moment an image enters a ViT-B/16 to the moment it influences the final classification logit." If you can't do this fluently in 5 minutes, this chapter's not done.

Part 3 — Foundation Models and Generative Computer Vision

This chapter covers the part of the field that has changed the most since 2022: vision-language foundation models, image and video generation, and the convergence of understanding and generation into single "everything" models. Expect at least one round of a principal interview to live here.

3.1 Vision-language foundation models (VLMs / MLLMs)

3.1.1 The dual-encoder line: CLIP, ALIGN, SigLIP, MetaCLIP, EVA-CLIP

CLIP loss. Image and text encoders produce \(\ell_2\)-normalized embeddings \(u_i, t_j\). Logits \(L_{ij} = \langle u_i, t_j\rangle/\tau\) with a learned temperature \(\tau\). Symmetric InfoNCE:

\[\mathcal{L}_{\text{CLIP}} = -\frac{1}{2B}\sum_{i=1}^{B}\Big[\log\frac{e^{L_{ii}}}{\sum_j e^{L_{ij}}} + \log\frac{e^{L_{ii}}}{\sum_j e^{L_{ji}}}\Big].\]

Why CLIP changed everything: zero-shot classification by computing the cosine of the test image with text embeddings of class names; zero-shot retrieval; a powerful frozen image encoder for downstream tasks (LLaVA prompts an LLM with CLIP features). See also ALIGN.

Pain points of CLIP. - Softmax over the full batch couples gradient computation to batch size; hard at small or huge batch. - Bad at counting, OCR, fine-grained reasoning. Heavy reliance on dataset scaling. - Single-vector global representation discards spatial info.

SigLIP / SigLIP 2. Replace softmax with a per-pair sigmoid loss:

\[\mathcal{L}_{\text{SigLIP}} = -\frac{1}{B}\sum_{i,j}\log\sigma\!\big(z_{ij}(\tau\langle u_i, t_j\rangle + b)\big), \qquad z_{ij} = \begin{cases}+1 & i=j\\ -1 & \text{else.}\end{cases}\]

Two learned scalars: \(t\) (temperature) and \(b\) (bias). No batch-wide normalization — scales to small or huge batches, and more sample-efficient. SigLIP 2 adds captioning and self-distillation auxiliaries for stronger dense features and zero-shot OCR. See SigLIP.

Curating better data: MetaCLIP, DFN. MetaCLIP balances metadata (text n-grams) over LAION-style scrapes to recover CLIP-quality with smaller, cleaner data. DFN (Data Filtering Networks) trains small filter models that score raw web pairs for inclusion, dramatically improving downstream metrics. See also EVA-CLIP.

3.1.2 The captioning / instruction-tuned line

BLIP-2 and the Q-Former. A Q-Former is a small Transformer with \(n_q\approx 32\) learnable queries, cross-attending to the frozen image encoder's tokens, producing a fixed-size token block that the LLM sees:

\[Q' = \mathrm{CrossAttn}\big(Q,\, f_\phi(x),\, f_\phi(x)\big).\]

Training: image-text contrastive + image-grounded text generation + image-text matching. Tradeoff: reduces visual tokens to \(n_q\) regardless of image size — great for context, lossy for dense reasoning.

LLaVA and the linear-projection minimalism. LLaVA simplifies: take CLIP features, apply a 2-layer MLP projector, prepend to LLM prompt. Train in two stages: (1) feature alignment on captions only, frozen LLM; (2) instruction tuning on synthetic GPT-4-generated multi-turn dialogs. LLaVA-NeXT introduced AnyRes: split a high-res image into tiles, encode each, prepend a global thumbnail, concat. Massive accuracy bump for OCR/dense tasks.

Q-Former vs MLP vs cross-attention. Three injection strategies and their trade-offs: - MLP projection (LLaVA, InternVL): simplest, every image patch becomes a token in the LLM. Flexible to dynamic resolution but token-hungry. - Q-Former (BLIP-2): fixed token budget, more parameters, needs more pretraining stages, weaker on dense tasks. - Gated cross-attention (Flamingo): insert cross-attention modules into the LLM with zero-init gates so the LM is undisturbed at start. Parameter-heavy; complex training.

Native dynamic resolution: Qwen2-VL, InternVL. Qwen2-VL processes images at their native resolution by computing 2D-RoPE on absolute (row, col) patch indices. Pixel unshuffle (reshape \((H, W, C)\to(H/r, W/r, Cr^2)\) then linearly project) compresses by \(r^2\). Token count then scales with image area, no aspect-ratio distortion. InternVL: tile high-res images into \(448\times448\) chunks (after dynamic aspect-ratio cropping), each tile becomes a sub-image; class token from each plus a global thumbnail. Family scales to InternVL3 (over 70B params).

3.1.3 Native multimodal architectures: the early-fusion bet

Three lines in 2024–2026 abandon the bolt-on adapter and train a single stack on interleaved image+text+audio tokens.

Chameleon (Meta). Image encoder is a VQ tokenizer producing 1024 discrete tokens per \(256\times256\) image. Single autoregressive Transformer trained on image and text tokens with the same next-token objective. Output: text or image (decoded back from tokens).

Show-o, Janus, Janus-Pro, Transfusion, Emu3. Show-o: same backbone for AR text and discrete diffusion image (parallel masking). Transfusion: same backbone, AR for text, continuous diffusion for image (separate loss heads on shared layers). Janus / Janus-Pro: decouple understanding and generation pathways into separate visual encoders sharing an LLM. Emu3: pure next-token prediction over text + image + video tokens, matching diffusion on text-to-image.

The big argument. Adapter-style VLMs are convenient and reuse existing LLMs, but they're a transition technology. By 2026, frontier proprietary models (GPT-5, Gemini 2.5, Claude Opus 4.6) are natively multimodal end-to-end; so are several open frontier models. Expect this trend to dominate.

3.1.4 Vision tokenizers: the unsung hero

Early-fusion native multimodal models live or die by their image/video tokenizers. The tokenizer's vocabulary size, codebook utilization, and reconstruction quality dictate downstream performance.

VQ-VAE recap. Encoder \(E(x) = z_e\in\mathbb{R}^{H'\times W'\times d}\); codebook \(\{e_1,\ldots,e_K\}\). Each spatial position \(z(i,j)\) is snapped to its nearest codebook entry \(z_q = e_{k^*}\). Loss with stop-grad straight-through estimator:

\[L = \|x - \mathcal{D}(z_q)\|^2 + \|\mathrm{sg}(z_e) - e_{k^*}\|^2 + \beta\|z_e - \mathrm{sg}(e_{k^*})\|^2.\]

Codebook collapse is the perpetual problem: many codes go unused. Mitigations: EMA-updated codebook, dead-code resampling, large-batch VQ.

LFQ (lookup-free quantization, MAGVIT-v2). Project latent \(z\in\mathbb{R}^d\) to dimension \(L\), then sign-quantize to \(\{-1, +1\}^L\): \(q = \mathrm{sgn}(z)\), giving \(2^L\) implicit codes. No codebook to maintain or learn. Avoids collapse by construction. Loss: reconstruction + an entropy regularizer that promotes uniform usage of the implicit codes.

FSQ (finite scalar quantization). Each scalar dim of \(z\) rounded to a small set \(\{-K,\ldots,K\}\) (e.g., 5 levels per dim, 8 dims \(\Rightarrow 5^8\approx 390{,}000\) codes). Even simpler than LFQ; no auxiliary entropy loss needed.

TiTok and 1D tokenization. TiTok (1D Image Tokenizer) produces \(\sim 32\) tokens per \(256^2\) image (vs \(\sim 1024\) in 2D). Aggressively compressed; surprisingly, downstream image generation quality holds with much shorter sequences.

Cosmos tokenizer (NVIDIA). Joint image + video tokenizer with continuous and discrete variants, supports up to 8K video. Causal 3D structure: temporal dim is causal, spatial dim is not.

3.1.5 Long-context video VLMs

A 1-hour video at 1 fps with 256-token-per-frame patching is \(\sim 920{,}000\) tokens — prohibitive without compression. Strategies:

3.1.6 Reasoning VLMs (multimodal o1 / R1)

The 2025 reasoning revolution carried into multimodal:

The empirical pattern: at fixed compute, scaling test-time chain-of-thought beats scaling model size for math/visual reasoning tasks — but only above a critical model scale where the base can already attempt the task.

★ 2026 SOTA update — Foundation Models and Generative Computer Vision

3.2 Diffusion models: derivations and design choices

3.2.1 Forward and reverse processes

Define a fixed Markov forward process that gradually adds noise:

\[q(x_t\mid x_{t-1}) = \mathcal{N}\big(x_t;\, \sqrt{1-\beta_t}\,x_{t-1},\, \beta_t I\big).\]

With \(\alpha_t = 1-\beta_t\) and \(\bar\alpha_t = \prod_{s\le t}\alpha_s\):

\[q(x_t\mid x_0) = \mathcal{N}\big(x_t;\, \sqrt{\bar\alpha_t}\,x_0,\, (1-\bar\alpha_t)I\big), \qquad x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ \ \epsilon\sim\mathcal{N}(0,I).\]

Reverse posterior, by Bayes:

\[q(x_{t-1}\mid x_t, x_0) = \mathcal{N}(\tilde\mu_t, \tilde\beta_t I), \quad \tilde\mu_t = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t,\quad \tilde\beta_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t.\]

The model \(p_\theta(x_{t-1}\mid x_t) = \mathcal{N}(\mu_\theta(x_t, t), \sigma_t^2 I)\) tries to match \(q\).

3.2.2 From ELBO to the simple objective

The ELBO decomposes per step:

\[\mathcal{L}_{\text{vlb}} = \mathbb{E}_q\Big[\mathrm{KL}(q(x_T\mid x_0)\,\|\,p(x_T)) + \sum_{t>1}\mathrm{KL}(q(x_{t-1}\mid x_t, x_0)\,\|\,p_\theta(x_{t-1}\mid x_t)) - \log p_\theta(x_0\mid x_1)\Big].\]

For Gaussians with the same variance, KL reduces to a mean-squared difference. Reparameterize \(\mu_\theta\) via the noise-prediction parameterization

\[\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\Big(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon_\theta(x_t, t)\Big),\]

and the per-step KL becomes (up to coefficients) \(\|\epsilon - \epsilon_\theta(x_t, t)\|^2\). Dropping the time-varying coefficient gives:

Key

The simplified DDPM objective:

\[\mathcal{L}_{\text{simple}} = \mathbb{E}_{t,x_0,\epsilon}\big\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t)\big\|^2.\]

Random \(t\in\{1,\ldots,T\}\), random \(\epsilon\sim\mathcal{N}(0,I)\), MSE on noise. See DDPM.

3.2.3 Parameterizations: ε, x₀, v

Three equivalent targets, related by:

\[x_0 = \frac{x_t - \sqrt{1-\bar\alpha_t}\,\epsilon}{\sqrt{\bar\alpha_t}}, \qquad v = \sqrt{\bar\alpha_t}\,\epsilon - \sqrt{1-\bar\alpha_t}\,x_0.\]

\(v\)-prediction (Salimans & Ho) is more stable across noise levels because the target's scale doesn't blow up at \(t\to 0\) or \(t\to T\). Standard for distillation and for video diffusion.

3.2.4 Score-based view (continuous time)

Forward SDE \(dx = f(x, t)\,dt + g(t)\,dw\). Reverse SDE (Anderson):

\[dx = \big[f(x, t) - g(t)^2\,\nabla_x\log p_t(x)\big]\,dt + g(t)\,d\bar w, \qquad \epsilon_\theta \propto -\sigma_t\,\nabla_x\log p_t(x).\]

Probability-flow ODE (deterministic):

\[dx = \big[f(x, t) - \tfrac12 g(t)^2\,\nabla_x\log p_t(x)\big]\,dt.\]

Equivalent to DDIM in the variance-preserving discretization.

3.2.5 Sampling families

3.2.6 Classifier-free guidance (CFG)

Train one network \(\epsilon_\theta(x_t, t, c)\) that randomly drops \(c\) to \(\varnothing\) with probability \(p_{\text{drop}}\) (commonly 0.1). At inference:

\[\tilde\epsilon(x_t, t, c) = \epsilon_\theta(x_t, t, \varnothing) + w\big(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\big).\]

\(w = 1\) recovers conditional. \(w > 1\) amplifies conditioning, sharper images, less diversity. Asymptotically equivalent to sampling from \(p(x\mid c) \propto p(x)\,p(c\mid x)^w\).

Modern variants. - CFG++: corrects the DDIM step using guidance only on the score component, not on the manifold projection step. - Autoguidance: instead of unconditional, use a smaller/worse model as the negative. - APG (Adaptive Projected Guidance) and PAG (Perturbed Attention Guidance): avoid the over-saturation artifacts of high-CFG.

3.2.7 Latent diffusion

Train an autoencoder \(E, D\) with KL-regularization (continuous latent) or VQ-regularization (discrete latent). Run diffusion in latent space:

\[\mathcal{L} = \mathbb{E}_{z,t,\epsilon,c}\big\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t}\,z + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t,\ c)\big\|^2.\]

Decode \(\hat x = \mathcal{D}(\hat z)\). Saves compute (often \(8\times\) spatial downsampling, so \(\sim 64\times\) fewer flops per layer). Stable Diffusion 1.5 / 2.x / SDXL / SD3 / SD3.5 all use latent diffusion. SDXL has a larger U-Net (2.6B) and refines images via a second-stage model. SD3 transitions to MM-DiT (multi-modal Transformer).

3.2.8 DiT and MM-DiT

DiT (Diffusion Transformer): replace the U-Net with a Transformer over noised latent tokens. Conditioning \(c\) (timestep + class/text) injected via adaLN (adaptive layer norm: scale and shift of LN). MM-DiT (SD3, FLUX): jointly model text and image tokens in one Transformer, with separate embedding projections and a single concatenated attention. Cleaner cross-modal coupling than U-Net cross-attention.

3.2.9 Flow matching and rectified flow

Continuous normalizing flow refresher. A vector field \(u_t:\mathbb{R}^d\to\mathbb{R}^d\) generates a probability path \(p_t\) via the ODE \(dx_t = u_t(x_t)\,dt\), \(x_0\sim p_0\). The path satisfies the continuity equation \(\partial_t p_t + \nabla\cdot(p_t u_t) = 0\).

Conditional Flow Matching (Lipman et al.). Given a marginal path between \(p_0\) (prior) and \(p_1\) (data), choose conditional paths \(p_t(x\mid x_1)\) for which we know the conditional vector field \(u_t(x\mid x_1)\). Train

\[\mathcal{L}_{\text{CFM}} = \mathbb{E}_{t, x_1, x\sim p_t(\cdot\mid x_1)}\big\|v_\theta(x, t) - u_t(x\mid x_1)\big\|^2.\]

For affine paths \(x_t = (1-t)x_0 + t x_1\) with \(x_0\sim\mathcal{N}(0,I)\): \(u_t(x\mid x_1) = x_1 - x_0\), so the loss simplifies dramatically (Flow Matching):

\[\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x_0, x_1}\big\|v_\theta((1-t)x_0 + t x_1,\ t) - (x_1 - x_0)\big\|^2.\]

Rectified flow. Iteratively re-pair \((x_0, x_1)\) to straighten the trajectories. After a few rectifications, \(v\) approximates the average direction along straight lines, enabling few-step or one-step sampling. Underlies SD3, FLUX, and several distillation pipelines (InstaFlow, Reflow).

Key

Diffusion vs flow matching: the same target architecture, different parameterization of the noise schedule. FM gives cleaner straight-line sampling, easier distillation, and is now the dominant choice for new model releases. Both are equivalent in expressive power.

3.2.10 Distillation: from many steps to a few

Progressive distillation. Halve the number of sampling steps each round; teacher runs 2 steps, student matches in 1. Repeat \(\log_2 T\) rounds.

Consistency models. Train \(f_\theta(x_t, t)\approx x_0\) for any \(t\) along the same ODE trajectory:

\[\mathcal{L}_{\text{CM}} = \mathbb{E}\,d\big(f_\theta(x_{t_{n+1}}, t_{n+1}),\ f_{\theta^-}(\hat x_{t_n}, t_n)\big),\]

with EMA target \(\theta^-\) and a perceptual or L2 metric \(d\). 1–4 step inference. Latent Consistency Models (LCM) apply this in latent diffusion. See Consistency Models.

DMD / DMD2 (Distribution Matching Distillation). Train a 1-step student by matching the score field of the teacher. DMD2 adds a GAN-style discriminator and removes the regression loss, enabling true one-step image generation at quality matching 50-step teachers.

Hyper-SD and phased-consistency. Phased Consistency Models (PCM) divide the ODE into segments and apply consistency within each, recovering quality for very few steps. Hyper-SD blends consistency, GAN, and ODE-trajectory losses for SDXL/FLUX.

3.2.11 Conditioning and control

ControlNet. Clone the encoder of the diffusion model into a trainable copy that takes an extra condition (canny, depth, pose, segmentation), zero-init the connections so the base model is undisturbed at start. Train the copy + connection layers; keep base frozen.

T2I-Adapter. Lightweight: a few conv blocks predict feature offsets added to the diffusion model's intermediate features. Less power than ControlNet, much fewer parameters.

IP-Adapter. Decouple text and image cross-attention: add a parallel image cross-attention path conditioned on a CLIP-encoded reference image. Personalization without fine-tuning.

LoRA, OFT, BOFT for personalization. Low-rank deltas \(W' = W + BA\), \(A\in\mathbb{R}^{r\times d}\), \(B\in\mathbb{R}^{d\times r}\), \(r\ll d\). OFT applies orthogonal rotations instead, preserving spectrum. BOFT block-diagonalizes the rotations for compute savings.

DreamBooth, Textual Inversion. DreamBooth: fine-tune the full model on a few subject images with a unique identifier and prior-preservation loss. Textual Inversion: only learn an embedding for a new token, freezing the model. Customization at very different cost / quality trade-offs.

InstantID, PhotoMaker, PuLID. Single-shot face personalization without fine-tuning. Encode the face with an ID encoder; inject via cross-attention or feature modulation. Common in modern image apps.

3.2.12 Editing

SDEdit. Add noise to the input image up to time \(t\), then denoise with the new prompt. Simple and effective; trades off prompt fidelity vs source preservation by choice of \(t\).

Prompt-to-Prompt and Null-text inversion. Manipulate cross-attention maps to localize edits to specific text tokens. Null-text inversion fixes DDIM inversion drift by fine-tuning the unconditional embedding per timestep so deterministic forward + reverse perfectly reconstructs.

InstructPix2Pix. Train a conditional diffusion model on (source, edit-instruction, target) triplets generated by GPT-3 + Stable Diffusion. At inference: condition on source image and an edit instruction.

FLUX Kontext, OmniEdit, AnyEdit. 2025–2026 generation: image editing is now first-class, end-to-end (single model, no inversion gymnastics). FLUX Kontext from Black Forest Labs; OmniEdit/AnyEdit are open, trained on synthetic + curated edit pairs.

3.2.13 Diffusion-DPO

Lift DPO to diffusion: preferences over images \((x_w, x_l\mid c)\). Replace log-prob with a per-step diffusion loss surrogate (Diffusion-DPO):

\[\mathcal{L}_{\text{D-DPO}} = -\mathbb{E}\,\log\sigma\!\Big(-\beta\big(\mathcal{L}_\theta(x^w, c) - \mathcal{L}_{\theta_{\text{ref}}}(x^w, c) - \mathcal{L}_\theta(x^l, c) + \mathcal{L}_{\theta_{\text{ref}}}(x^l, c)\big)\Big),\]

where \(\mathcal{L}_\theta(x, c)\) is the standard diffusion loss on \(x\). Used to align SDXL/SD3/FLUX to aesthetic preferences.

3.2.14 Eval: FID, CLIP-Score, ImageReward, GenAI-Bench

FID: \(\|\mu_r - \mu_g\|^2 + \mathrm{tr}\big(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2}\big)\) on Inception features. Limitations: not aligned with human preference, sensitive to backbone choice, broken at high resolution.

CLIP-Score: cosine similarity between image and text embeddings. Easy to game with generic outputs.

ImageReward, HPSv2, PickScore: reward models trained on human preferences. Better correlation with humans, but reward hacking remains.

VQAScore: VQA-on-generated; measures whether a frozen VLM can answer compositional questions about the image consistent with the prompt.

GenEval, T2I-CompBench, GenAI-Bench: programmatic compositional benchmarks (counting, attribute binding, spatial relations).

Reality: human preference (Chatbot Arena style, Image Arena) is the ground truth.

3.3 Video generation

3.3.1 The architecture pattern

Causal 3D VAE encodes video to latents (typically \(4\)–\(8\times\) temporal compression, \(8\times\) spatial). DiT (or MM-DiT) operates on latent tokens with 3D positional encodings. Loss is the same diffusion or flow matching target.

3.3.2 Joint image-video training

A still image is the \(T = 1\) case. With packing, the same model trains on both. Often weighted:

\[L = \lambda_I\,\mathcal{L}_{\text{img}} + \lambda_V\,\mathcal{L}_{\text{video}}\]

to avoid neglecting per-frame quality. The packed token format (variable \(T, H, W\)) has to be supported end-to-end.

3.3.3 Where things go wrong

3.3.4 Frontier systems (2025–2026)

Sora 2, Veo 3, Kling 2, MovieGen, Hunyuan Video, Wan 2.1, Mochi 1, Step-Video, Allegro, LTX-Video. Common patterns:

3.3.5 World models for video

GAIA-1/GAIA-2 (Wayve, driving), Cosmos World Foundation Models (NVIDIA, general), Genie / Genie 2 (DeepMind, playable). The unifying idea: a generative video model conditioned on actions becomes a learned simulator. Used to train RL policies, evaluate AVs in closed loop, and explore counterfactuals.

3.4 Promptable, open-vocabulary, "everything" models

The taxonomy in 2026:

What unifies these is the move from task-specific architectures with closed label spaces to language-conditioned models with open vocabulary, often shared backbones with task-specific heads, all trained on massive synthetic/programmatic supervision data.

3.5 Strong opinions to defend

A principal interview will probe what you actually believe. Some defensible 2026 positions:

Part 4 — Autonomous Driving, 3D, and Robotics

This is the part of CV that touches the physical world. The math is geometry-heavy; the systems engineering is brutal; the safety considerations make every design choice consequential. Senior interviewers in this space probe whether you've actually shipped — whether you understand sensor calibration drift, label noise on long-tail classes, and the closed-loop testing problem.

4.1 Autonomous driving perception, end-to-end

4.1.1 Sensor modalities and trade-offs

The architectural question every AV team faces: which sensor is the source of truth? Tesla's vision-only bet, Waymo's lidar-first stack, Mobileye's hybrid — different answers, all defensible.

4.1.2 The classical AV perception stack (decomposed)

  1. Sensor synchronization + calibration.
  2. Per-sensor detection (2D for camera, 3D for lidar/radar).
  3. Sensor fusion to a unified ego-frame representation (BEV).
  4. Tracking + ID association across frames.
  5. Lane / road geometry estimation; HD map matching or online mapping.
  6. Motion prediction for tracked objects.
  7. Planning + control.

The 2024–2026 trend is to collapse 2–6 into a single end-to-end neural net (UniAD, VAD, GenAD, Tesla FSD v12+).

4.1.3 Camera-to-BEV: Lift, Splat, Shoot

LSS (Philion & Fidler) predicts a categorical depth distribution per pixel:

\[\alpha\in\Delta^{D-1}, \qquad F(d) = \alpha_d\cdot f \quad\text{(weighted feature lift)},\]

where \(f\) is a per-pixel feature and \(D\) is the number of depth bins. Each lifted point projects to the BEV grid via \(X_w = R^\top(K^{-1}[u, v, 1]^\top d - t)\). Splat: scatter+sum into BEV cells. Shoot: BEV-conv head to produce maps and detections.

Pain points. - Depth ambiguity: monocular cues are fundamentally limited. - Calibration drift: small rotation errors are huge at range. - Long tail: rare classes get few BEV votes.

4.1.4 BEVFormer and deformable cross-attention

BEVFormer maintains a BEV query grid and attends from each query into the image features of multiple cameras at the projected pixels:

\[\mathrm{DA}(q, p) = \sum_{m=1}^{M} W_m \sum_{k=1}^{K} A_{mk}\, W_m'\, f(p + \Delta p_{mk}),\]

with attention offsets \(\Delta p_{mk}\) and weights \(A_{mk}\) predicted from \(q\). Temporal self-attention pulls from \(\text{BEV}_{t-1}\) for motion. BEVFormer V2 adds perspective-view auxiliaries.

4.1.5 Sparse query detectors: SparseBEV, StreamPETR, Far3D

Replace the dense BEV grid with a small set of sparse 3D queries (one per candidate object). Lower compute, cleaner gradient flow at the cost of more complex query selection. StreamPETR adds streaming temporal modeling; Far3D pushes detection range out to 150m+.

4.1.6 LiDAR detection

VoxelNet, SECOND, PointPillars. VoxelNet partitions the point cloud into 3D voxels, encodes each with PointNet-like features, applies sparse 3D convs. SECOND uses sparse 3D convs more efficiently. PointPillars replaces 3D convs with 2D convs on a pseudo-image: each pillar (vertical column) becomes a 2D "pixel." Faster, simpler, surprisingly strong; standard production baseline.

CenterPoint and friends. CenterPoint predicts objects as heatmap peaks in BEV (anchor-free). Two-stage refinement uses ROIAlign equivalent on the BEV pseudo-image. Common in Waymo/nuScenes leaderboards through 2023.

Transformer-based: TransFusion, DSVT. DSVT (Dynamic Sparse Voxel Transformer) processes sparse voxels with sliding windows; matches state-of-the-art with low latency.

4.1.7 BEV fusion

Several strategies for combining camera + lidar + radar into BEV:

4.1.8 Online HD map estimation: MapTR, MapTRv2, StreamMapNet

Predict a vectorized lane/road graph from sensor inputs. MapTR uses permutation-invariant point-set prediction per polyline, Hungarian-matched to GT polylines. StreamMapNet adds streaming temporal aggregation. Critical for "mapless" driving stacks (Tesla's bet).

4.1.9 Occupancy networks

Predict per-voxel occupancy + semantic class in a 3D grid around the ego vehicle. Catches general obstacles (debris, weird shapes) that detection-by-class misses.

TPVFormer: tri-plane representation (XY, YZ, XZ planes) instead of full voxel grid. OccFormer / FB-OCC / SurroundOcc: variants on dense voxel prediction. SparseOcc / OPUS: only predict non-empty voxels (massive memory savings).

Loss:

\[L_{\text{occ}} = \mathrm{CE}_{\text{voxel}} + \lambda_g\,\mathrm{Lov\acute{a}sz} + \lambda_a\,\mathrm{Affinity},\]

where Lovász handles class imbalance and affinity preserves local structure (geometry + semantic consistency between neighboring voxels).

4.1.10 Motion prediction

Vector representations. VectorNet, TNT, MultiPath++: encode all map and agent context as polyline vectors with attention; predict \(K\) trajectory modes per agent with mixture probabilities. Loss combines:

\[\mathcal{L}_{\text{traj}} = \min_k\|\hat\tau_k - \tau^*\|_2 + \lambda_c\,\mathrm{CE}(\hat\pi, k^*),\]

the winner-takes-all (closest mode trains for displacement) plus mode-classification (closest mode also trained for selection). Avoids mode-averaging.

Wayformer, MTR, MTR++, QCNet. Wayformer scales attention; MTR adds anchor trajectories as queries; MTR++ adds multi-agent joint prediction; QCNet uses query-centric encoding for invariance to ego frame.

4.1.11 End-to-end driving

UniAD (CVPR 2023 best paper). Unified architecture with shared BEV features driving multiple task heads (detection, tracking, mapping, occupancy, motion, planning). Planning head consumes all upstream features and outputs ego trajectory; loss is summed across tasks:

\[\mathcal{L}_{\text{e2e}} = \sum_{\text{tasks}}\lambda_\tau\mathcal{L}_\tau + \lambda_p\mathcal{L}_{\text{plan}}, \qquad \mathcal{L}_{\text{plan}} = \|\hat\tau - \tau^*\|_2 + \lambda_c\,\mathrm{CollisionPenalty}(\hat\tau).\]

VAD and successors. VAD (Vectorized Autonomous Driving) replaces dense BEV with vectorized representations throughout, faster and lighter. VAD V2 + GenAD blur into world-modeling. Hydra-MDP: Multi-Diversity Planning with multiple anchor trajectories; SparseDrive uses sparse queries throughout.

Tesla FSD v12+, Wayve, Mobileye. Tesla's FSD v12 reportedly drops C++ planner code and runs end-to-end neural net ("photons in, controls out"). Wayve trains world-model + policy from human-driven videos. Mobileye still favors rule-based safety overlays atop neural perception.

4.1.12 World models for driving

GAIA-1/GAIA-2 (Wayve): video diffusion conditioned on past frames + actions, generates futures. Used as a closed-loop simulator for evaluation and policy learning. DriveDreamer-2: text-controlled scenario generation. Vista, MagicDrive3D, EmerNeRF: scene-level video synthesis with controllability.

4.1.13 Closed-loop simulation

CARLA: classical simulator with full sensor models. Bench2Drive, NAVSIM, DriveArena: benchmarks for end-to-end stacks. Neural simulators: NeuRAD, UniSim, S-NeRF, EmerNeRF, StreetGaussians, OmniRe — reconstruct real driving logs as differentiable scenes you can re-render under perturbation. The promise: "test against your own data, perturbed."

4.1.14 Auto-labeling and data engines

The biggest leverage in AV is data quality. Patterns:

★ 2026 SOTA update — Autonomous Driving, 3D, and Robotics

4.2 NeRF and 3D Gaussian Splatting

4.2.1 NeRF, the original

A scene is a function \(F_\Theta: (x, y, z, \theta, \phi)\to(c, \sigma)\) with color \(c\in\mathbb{R}^3\) and density \(\sigma\ge 0\). Render along a ray \(r(t) = o + td\):

\[C(r) = \int_{t_n}^{t_f} T(t)\,\sigma(r(t))\,c(r(t), d)\,dt, \qquad T(t) = \exp\!\Big(-\int_{t_n}^{t}\sigma(r(s))\,ds\Big).\]

Discretize with stratified sampling:

\[\hat C(r) = \sum_{i=1}^{N} T_i\big(1 - e^{-\sigma_i\delta_i}\big)c_i, \qquad T_i = \exp\!\Big(-\sum_{j<i}\sigma_j\delta_j\Big).\]

Loss: \(\mathcal{L} = \sum_r\|C - \hat C\|_2^2\). Hierarchical sampling: a coarse network produces weights \(w_i = T_i(1 - e^{-\sigma_i\delta_i})\); the fine network samples additional points from the normalized PDF.

Positional encoding.

\[\gamma(p) = \big[\sin(2^k\pi p),\ \cos(2^k\pi p)\big]_{k=0}^{L-1}\]

(networks have a low-frequency bias that PE breaks).

4.2.2 NeRF variants

4.2.3 3D Gaussian Splatting (Kerbl, SIGGRAPH 2023)

Representation. Each Gaussian: - Position \(\mu_i\in\mathbb{R}^3\). - Anisotropic covariance \(\Sigma_i = R_i S_i S_i^\top R_i^\top\), where \(R_i\in SO(3)\) from a quaternion and \(S_i = \mathrm{diag}(s_x, s_y, s_z)\). - Opacity \(\alpha_i\in(0, 1)\). - Color \(c_i(d)\) via spherical harmonics (SH degree 0–3).

Number of Gaussians: 1M–10M for a typical scene. See 3D Gaussian Splatting.

Projection to image plane. Approximate the 3D Gaussian by a 2D Gaussian in screen space:

\[\Sigma' = J W \Sigma W^\top J^\top,\]

where \(W\) is the world-to-camera linearization at \(\mu_i\) and \(J\) is the Jacobian of the perspective projection. The z-dimension is collapsed (we render in screen space).

Differentiable rasterization. Per pixel \(p\), depth-sort Gaussians intersecting the pixel, then alpha-composite front to back:

\[C(p) = \sum_{i\in\mathcal{N}(p)} c_i\,\alpha_i'\prod_{j<i}(1 - \alpha_j'), \qquad \alpha_i' = \alpha_i\cdot\exp\!\Big(-\tfrac12(x - \mu_i')^\top\Sigma_i'^{-1}(x - \mu_i')\Big).\]

Differentiable w.r.t. all Gaussian parameters.

Training loss.

\[\mathcal{L}_{\text{3DGS}} = (1-\lambda)\mathcal{L}_1 + \lambda\,\mathcal{L}_{\text{D-SSIM}}(\hat I, I), \qquad \lambda\approx 0.2.\]

Adaptive density control. Periodically: - Clone Gaussians with small position gradients but small scale (under-reconstruction). - Split Gaussians with large position gradients (over-reconstruction): replace with 2 Gaussians sampled from the Gaussian itself, each with smaller scale. - Prune Gaussians with \(\alpha < \tau\) or huge screen-space size. - Periodically reset all opacities to a small value to prune redundant Gaussians.

4.2.4 3DGS variants

Interview probe. "NeRF or 3DGS for our use case?" Ask: real-time render needed (3DGS wins), large dynamic scenes (4D-GS or NeRF with deformation MLPs), relighting (early days for both, GS-IR / 3D-NeuS leading), mesh integration (3DGS with extraction or hybrid), training time (3DGS faster), storage (NeRF often wins compressed). Don't say "3DGS always." That's a junior-level answer.

4.2.5 Feed-forward 3D reconstruction (the new wave)

LRM and large reconstruction models. LRM (Large Reconstruction Model): given a single image, predict triplane neural representation in one forward pass. Trained on Objaverse + Objaverse-XL.

InstantMesh, MeshLRM, GS-LRM, MeshAnything. Family of feed-forward 3D models that predict mesh / Gaussians from one or a few images. Quality is improving fast; commercial-grade for many use cases by 2025.

DUSt3R, MASt3R, Spann3R, VGGT. The breakthrough idea: predict pixel-aligned 3D pointmaps directly from \(N\) images without solving SfM.

DUSt3R: given two images, predict \(X^{1,1}, X^{2,1}\in\mathbb{R}^{H\times W\times 3}\) (pointmaps in camera 1's frame) plus per-pixel confidence. Loss (confidence-weighted, scale-normalized):

\[\mathcal{L} = \sum_{v\in\{1,2\}}\sum_{i\in\mathcal{V}_v}\Big\|\tfrac{1}{z}X_i^{v,1} - \tfrac{1}{\bar z}\bar X_i^{v,1}\Big\|_2,\]

with normalization to handle scale ambiguity. Camera intrinsics, extrinsics, depth, point cloud all decode from the pointmap predictions; matches between pixels are simply nearest neighbors in 3D.

MASt3R: refines DUSt3R with explicit feature heads for matching, scaling to large-scale SfM (MASt3R-SfM does global SfM via hundreds of pairwise predictions + global optimization). Spann3R: incremental, processes one new view at a time. VGGT (Visual Geometry Grounded Transformer): large transformer that predicts depth, camera, and per-pixel 3D from \(N\) unposed images in a single forward pass. By 2025 the dominant feed-forward 3D model.

Key

The trend: classical SfM/MVS is being absorbed into feed-forward learned 3D. The implication is profound — bundle adjustment and triangulation become fallback / refinement steps, not the core. Within 3 years, COLMAP will be a comparison baseline, not a default.

4.2.6 3D generation

Optimization-based: SDS, VSD. Score Distillation Sampling (DreamFusion): differentiate a parametric scene \(\theta\) (NeRF or 3DGS) by passing renders through a frozen 2D diffusion model:

\[\nabla_\theta\mathcal{L}_{\text{SDS}} = \mathbb{E}_{t,\epsilon}\big[w(t)\big(\epsilon_\phi(x_t, t, c) - \epsilon\big)\,\partial x/\partial\theta\big].\]

The CFG-weighted gradient pushes the rendered image toward the diffusion prior. Slow (hours per scene) and prone to mode collapse / Janus problem (multiple front-faces). Variational SDS (ProlificDreamer): replace the noise target \(\epsilon\) by a learned variational distribution \(\hat q\), reducing mode collapse.

Multi-view diffusion: MVDream, Wonder3D, Zero123, Zero123++. Train a diffusion model that generates multiple consistent views of an object given one input image (or text). Use the multi-view outputs to optimize a 3D representation, dramatically reducing the SDS Janus problem.

Native 3D diffusion: Trellis, CLAY, Hunyuan3D-2, Direct3D. Diffuse directly in a 3D latent space (sparse-voxel + structural latents in Trellis; volumetric in CLAY; etc.). Fast (seconds), high quality, no SDS overhead. The dominant approach for new releases by 2025.

Mesh generation: MeshGPT, MeshAnything, MeshXL, EdgeRunner. Autoregressive mesh face generation: tokenize a mesh as a sequence of face-vertex triples; predict next face conditioned on history (and on a 3D shape latent or text). Native mesh topology output, no isosurface extraction needed.

4.2.7 Avatars

4.3 Robotics and Vision-Language-Action models

4.3.1 The data problem

Robotics has long been bottlenecked by data. Solutions in 2024–2026:

4.3.2 Imitation learning baselines

Behavior Cloning and DAgger. BC: \(\min_\theta\mathbb{E}_{(s,a)\sim\mathcal{D}}[-\log\pi_\theta(a\mid s)]\). The classic problem: distribution shift. Tiny errors compound across timesteps until the agent visits states the expert never saw.

DAgger: at iteration \(k\), run \(\pi_{\theta_k}\), query the expert on visited states, add to dataset:

\[\mathcal{D}_{k+1} = \mathcal{D}_k\cup\big\{(s, \pi^*(s)) : s\sim\pi_{\theta_k}\big\}.\]

Reduces compounding error to linear in horizon (vs quadratic for BC). In robotics, the expert query is usually a human teleop or a more expensive teacher policy.

Action Chunking Transformer (ACT). Predict a chunk of \(H\) future actions instead of one:

\[L_{\text{ACT}} = \mathbb{E}\big[\|\hat a_{t:t+H} - a_{t:t+H}\|_1 + \beta\,\mathrm{KL}(q(z\mid a_{t:t+H}, o)\,\|\,p(z))\big].\]

At inference, temporal ensembling: average overlapping chunk predictions across timesteps for smooth actions. Dramatic improvement over single-step BC on bimanual manipulation.

Diffusion Policy. Generate the action sequence with a conditional diffusion model on observation history:

\[\mathcal{L} = \mathbb{E}_{k,\epsilon,a_0,o}\big\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_k}\,a_0 + \sqrt{1-\bar\alpha_k}\,\epsilon,\ k,\ o)\big\|^2.\]

Multi-modal action distributions (which the unimodal Gaussian heads in BC/ACT can't represent) are critical for real manipulation. Receding-horizon execution: sample \(H\) actions, execute the first \(h < H\), replan.

3D Diffuser Actor: condition on a 3D scene token (point cloud or feature volume) for spatial generalization. RDT-1B (Robotics Diffusion Transformer): 1B-parameter diffusion policy pretrained across 46 datasets and embodiments, then fine-tuned per embodiment.

4.3.3 Vision-Language-Action models

The basic recipe. Take a pretrained VLM, add an action head, fine-tune on robot trajectories.

RT-1, RT-2, RT-X. RT-1: Transformer over image + instruction tokens, discretized actions. RT-2: VLM (PaLI-X or PaLM-E) with action tokens embedded into the LLM vocabulary. Co-trained on web data + robot data so VLM web knowledge transfers to action space. RT-X: same architecture trained on Open-X-Embodiment, demonstrating cross-embodiment transfer.

OpenVLA. Open-source 7B VLM (Llama 2 + DINOv2 + SigLIP) fine-tuned on Open-X with discretized actions. Strong baseline.

\(\pi_0\) (Pi-Zero, Physical Intelligence). VLM backbone with a small flow-matching action head for continuous actions. Why FM? Continuous control needs continuous outputs; discretization hurts precision for fine manipulation. FM head trained with the standard FM objective on action chunks. Pretrained on \(\sim 10\text{k}\) hours of robot data, fine-tuned per task. \(\pi_{0.5}\) extends to long-horizon tasks with hierarchical control.

GR00T (NVIDIA), Helix (Figure). Humanoid foundation models. GR00T: open VLM-action stack with NVIDIA's Cosmos world models for sim. Helix: Figure's proprietary humanoid VLA, two networks (high-level slow, low-level fast).

4.3.4 Sim-to-real

Domain randomization. Per episode, sample \(\xi\sim p(\xi)\) over visual + dynamics nuisances (lighting, textures, friction, mass, gravity, noise levels). Train a robust \(\pi(s, \xi)\). The policy that survives is invariant to exactly the nuisances you randomized over.

Asymmetric actor-critic and teacher-student. Critic uses privileged info (full state \(s\), terrain map \(\xi\)) for variance reduction; actor sees only egocentric observations \(o\). After training, distill the privileged actor into a vision-only student:

\[\mathcal{L}_{\text{distill}} = \mathbb{E}\big\|\pi_S(o) - \pi_T(s, \xi)\big\|^2.\]

Eureka and LLM-designed rewards. Eureka: LLM proposes reward function code; train RL with massive parallel sim (\(10^4+\) envs in Isaac Lab); evaluate task success on a held-out evaluator; provide signal back to LLM to refine. DrEureka extends to dynamics randomization design. Worked stunningly for shadow-hand pen-spinning, ANYmal locomotion, humanoid walking.

4.3.5 Locomotion

ANYmal-style: privileged teacher via PPO in Isaac Gym/Lab with terrain randomization, distilled to a vision-only student. Now standard for quadrupeds. Humanoids: OmniH2O, HumanPlus, ExBody2, Berkeley Humanoid — typically motion-capture-conditioned RL (track human reference motions with reward), then transfer.

4.3.6 Spatial reasoning VLMs for robotics

SpatialVLM: fine-tune VLM with synthetic 3D-grounded spatial QA data ("where is X relative to Y in 3D"). Massively improves spatial reasoning over base VLMs. RoboPoint, SpatialBot, SpatialVLA: similar idea for action grounding.

4.4 What an interviewer is really probing

The principal-level question in this space is: do you understand the data problem and the closed-loop testing problem as deeply as the modeling problem? Anyone can name BEVFormer; few can articulate why the long tail of edge cases dominates the production budget, why labeled data is the moat, why a slightly worse model with better evals beats a slightly better model without them, and why end-to-end neural nets are only safe when paired with a well-designed eval and rollback story.

Interview probe. "You're inheriting an AV perception team that's been chasing benchmark mAP for two years. What do you do in the first 90 days?" — expect this kind of strategic question. Have a coherent answer about evals, long-tail mining, deprecation, and rollout discipline.

Part 5 — AR/VR, Video Understanding, and On-Device

This chapter merges three adjacent worlds: video understanding (the largest unsolved supervision frontier in CV), AR/VR (where latency, power, and ergonomics dominate every architectural decision), and on-device inference (where compression and compiler co-design make or break a product). All three intersect in the consumer-AR roadmap of every major platform company.

5.1 Video understanding

5.1.1 From 3D conv to space-time attention

I3D, SlowFast, X3D. Earlier video models inflated 2D conv kernels to 3D (I3D), or used two streams at different temporal rates (SlowFast: slow pathway captures semantic content at low fps, fast pathway captures motion at high fps).

ViViT, TimeSformer, MViT, Video Swin. Transformer variants on space-time tokens. TimeSformer factorizes attention into space-only and time-only blocks (cheaper than full space-time). MViTv2 adds pooling attention for hierarchical features. Video Swin extends Swin's shifted window attention to 3D windows. See ViViT.

Hiera. Hierarchical without bells and whistles — pure ViT with simple pooling layers between stages. Surprisingly competitive; the lesson again is that careful training matters more than architectural cleverness.

5.1.2 Self-supervised video pretraining

VideoMAE / VideoMAE V2. Mask very high tube ratios (90%+) and reconstruct in pixel space. Tube masking (mask the same spatial location across all frames) prevents trivial temporal interpolation. See VideoMAE and VideoMAE V2.

V-JEPA, V-JEPA 2. Predict in latent space (EMA target encoder), not pixels. V-JEPA 2 scales to billion-frame pretraining. Argument: video is an even noisier supervision signal than images for pixel prediction; latent prediction extracts the predictable structure.

InternVideo / InternVideo 2. 1B+ multimodal video encoder trained with masked video reconstruction + video-text contrastive + multimodal next-token prediction. The combination of three losses gives both strong dense and strong global features. State of the art for many video benchmarks through 2025.

5.1.3 Video VLMs and long-form understanding

Standard recipe. 1. Sample frames at fixed fps (1–4 fps typically; up to 16 fps for short clips). 2. Encode each frame with a vision encoder. 3. Apply temporal pooling, Q-Former, or simple concat to produce a token sequence. 4. Prepend to LLM prompt.

Long-form challenges. A 1-hour video at 1 fps with 256 patch tokens per frame is \(\sim 920\text{k}\) tokens. Strategies: - LongVA, LongVU, LongVILA: training-time long-context support (ring attention). - MovieChat, MA-LMM: explicit short-term + long-term memory bank with summarization. - LLaMA-VID: 2 tokens per frame (one context, one content). - Goldfish: agent-style, retrieve relevant clips first, then reason. - Token compression: ToMe, similarity merging, hierarchical pooling.

Frontier 2026. Qwen2.5-VL handles 1-hour videos natively. Gemini 2.5 handles multi-hour. The race is on for both context length and reasoning quality over long videos.

5.1.4 Action recognition, temporal localization, dense captioning

5.1.5 Video object segmentation and tracking

SAM 2 is now the dominant video segmentation backbone (covered in Part 2). Sa2VA fuses SAM 2 with LLaVA-style chat for grounded video Q&A: "segment the bottle that the man passes to the woman" returns both the answer and the mask.

★ 2026 SOTA update — AR/VR, Video Understanding, and On-Device

5.2 AR/VR specific challenges

5.2.1 Latency budgets

Motion-to-photon latency of < 20 ms is the gold standard for VR comfort; the entire pipeline (sensor capture, IMU integration, tracking, render, scan-out) must fit. AR pass-through has even tighter budgets because real-world background is unstabilized. This drives:

5.2.2 Tracking and SLAM at AR latency

Visual-inertial SLAM. ORB-SLAM3, VINS-Fusion, OKVIS-2 ground the literature. Modern device stacks (Quest 3 Insight, Vision Pro, ARKit, ARCore) couple camera + IMU + (sometimes) depth with hand-tuned, latency-optimized engines.

Re-localization and persistent AR. For shared / persistent AR experiences, the device must re-localize against a previously-built map. - NetVLAD and successors (MixVPR, AnyLoc): global place descriptors. - Niantic Lightship VPS, ARKit Location Anchors: cloud VPS using street-level imagery. - Niantic LGM (Large Geospatial Model): vision-only large-scale localization.

5.2.3 Hand and body tracking

Hand: MediaPipe Hands, HaMeR, WiLoR: landmark + mesh regression with MANO model. Body: SMPL/SMPL-X-based regressors (CLIFF, OSX, SMPLer-X, NLF). NLF (Neural Localizer Fields) predicts per-vertex visibility and 3D position with neural fields.

5.2.4 Eye tracking and gaze

Foveated rendering needs fast gaze estimation (10+ Hz). Modern device stacks ship calibration-free gaze regressors trained on large datasets of (eye image, gaze direction).

5.2.5 Pass-through video / mixed reality

Vision Pro popularized full-pass-through MR. Engineering challenges include: - Depth-aware video processing for hand and keyboard occlusion. - Marigold-style monocular depth or stereo from device cameras. - Real-time matting / segmentation for keyboard / hand visibility. - Color and exposure matching between virtual content and pass-through video.

5.2.6 Avatars for AR/VR

5.2.7 Scene understanding for AR/VR

Real-time semantic + instance segmentation, depth completion, plane detection, occlusion meshing. Modern devices fuse a learned scene-understanding net with LiDAR/depth where present (Vision Pro, iPad Pro), or rely on monocular cues otherwise.

5.3 On-device inference

5.3.1 Architectures designed for mobile

5.3.2 Quantization

Symmetric per-tensor quantization.

\[q = \mathrm{clip}\big(\mathrm{round}(x/s),\, -Q,\, Q\big), \qquad s = \frac{\max|x|}{Q}, \qquad x\approx s\cdot q.\]

\(Q = 127\) for INT8. Per-channel for weights, per-tensor for activations is a common compromise.

Activation outliers and the per-tensor problem. LLMs and Transformers show outlier features in a small number of activation channels with \(> 100\times\) the typical magnitude. Per-tensor quantization stretches the scale to fit outliers, destroying precision for everyone else.

Solutions: - SmoothQuant: migrate scale from activations to weights via \(W' = W\,\mathrm{diag}(s)\), \(X' = X\,\mathrm{diag}(s^{-1})\). - QuaRot, SpinQuant: orthogonal rotations of weight/activation pairs to spread outliers. - LLM.int8(): keep outlier channels in FP16, rest in INT8. - Per-block (per 128-element block) quantization, common in FP4/FP8 stacks.

Post-training quantization (PTQ). GPTQ: layer-by-layer optimization with the inverse Hessian. Quantize a column of \(W\), then update remaining columns to compensate:

\[\Delta W_{:,j>k} = -\frac{(W_q - W)_{:,k}\,[H^{-1}]_{k,j>k}}{[H^{-1}]_{kk}}, \qquad H = 2XX^\top + \lambda I.\]

AWQ: salient-channel-aware per-channel scaling \(s\in\mathbb{R}^d\) to minimize \(\|W\,\mathrm{diag}(s^{-1})\,\mathrm{diag}(s)X - WX\|\) on calibration data, then symmetric quantize \(W\,\mathrm{diag}(s^{-1})\).

Quantization-aware training (QAT). LSQ: learnable step size per layer, gradients flow through the round operation via straight-through estimator. Better than PTQ for very low-precision (W4A4 and below) but adds training cost.

5.3.3 Diffusion-specific quantization

Q-Diffusion, PTQ4DM: quantize diffusion U-Nets. Activations vary dramatically across timesteps, so per-step calibration is needed. Recent work uses per-block scaling and rotations to make 8-bit and 4-bit inference work.

5.3.4 Pruning

5.3.5 Distillation for deployment

The right distillation choice depends on the target. For VLMs, distill larger model's logits / hidden states into a smaller student (e.g., DINOv2-Giant \(\to\) DINOv2-Base via feature distillation). For diffusion, distill many-step teacher into few-step student via consistency / DMD / phased consistency. For mobile detection, distill DETR into RT-DETR or YOLO-style.

5.3.6 Compilers, runtimes, and deployment stacks

5.3.7 Decode and data acceleration

Often the inference bottleneck isn't inference — it's video decode. NVDEC on NVIDIA GPUs can decode 30+ HD streams in parallel; pair with NVIDIA DALI for batched preprocessing on GPU; pair with FFCV for CPU-bottleneck breaking. For diffusion, latent caching (encode once with VAE, store latents) saves re-encoding.

5.3.8 System-level inference optimization

Part 6 — Reinforcement Learning, RLHF, GRPO

RL has become unavoidable in modern CV and multimodal: it's the alignment substrate for VLMs, the diffusion fine-tuning method of choice, and the only viable training signal for many robotics and game-playing systems. This chapter is the deepest technical chapter; if you're going for a research-leaning principal role, expect to defend it.

6.1 Foundations: MDPs, value functions, policy gradients

6.1.1 MDP setup

A Markov Decision Process \((S, A, P, r, \gamma)\) with transition \(P(s'\mid s, a)\), reward \(r(s, a)\), discount \(\gamma\in[0, 1)\). Policy \(\pi(a\mid s)\). Returns \(G_t = \sum_{k\ge 0}\gamma^k r_{t+k+1}\).

6.1.2 Value functions and Bellman equations

\[V^\pi(s) = \mathbb{E}_\pi[G_t\mid s_t = s], \qquad Q^\pi(s, a) = \mathbb{E}_\pi[G_t\mid s_t = s, a_t = a].\]

Bellman expectation:

\[V^\pi(s) = \mathbb{E}_{a\sim\pi}\big[r(s, a) + \gamma\,\mathbb{E}_{s'}[V^\pi(s')]\big].\]

Bellman optimality:

\[V^*(s) = \max_a\mathbb{E}_{s'}[r + \gamma V^*(s')], \qquad Q^*(s, a) = \mathbb{E}\big[r + \gamma\max_{a'}Q^*(s', a')\big].\]

6.1.3 Policy gradient theorem

\[\nabla_\theta J(\theta) = \mathbb{E}_{s\sim d^\pi, a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a\mid s)\, Q^\pi(s, a)\big] = \mathbb{E}\big[\nabla_\theta\log\pi_\theta(a\mid s)\, A^\pi(s, a)\big],\]

with advantage \(A = Q - V\) that subtracts a state-dependent baseline (variance reduction without bias).

Derivation sketch. Express \(J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}R(\tau)\), expand \(\nabla_\theta\log p_\theta(\tau) = \sum_t\nabla_\theta\log\pi_\theta(a_t\mid s_t)\) (transition probs are independent of \(\theta\)), apply the Markov property and reward-to-go, recover the \(Q^\pi\) form. Subtract baseline: zero in expectation.

6.1.4 GAE: Generalized Advantage Estimation

TD residual \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\). GAE:

\[\hat A_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l\,\delta_{t+l}.\]

\(\lambda = 0\) recovers TD(0) (high bias, low variance); \(\lambda = 1\) recovers full Monte Carlo (low bias, high variance). Typical values \(\lambda = 0.95, \gamma = 0.99\).

★ 2026 SOTA update — Reinforcement Learning, RLHF, GRPO

6.2 Algorithms: TRPO, PPO, SAC, TD3, model-based

6.2.1 TRPO and PPO

TRPO. Maximize the surrogate \(\mathbb{E}[r_\theta \hat A]\) with \(r_\theta = \pi_\theta/\pi_{\text{old}}\), subject to a trust-region constraint:

\[\mathbb{E}\big[\mathrm{KL}(\pi_{\text{old}}(\cdot\mid s)\,\|\,\pi_\theta(\cdot\mid s))\big] \le \delta.\]

Solved by conjugate gradient on the natural gradient direction. Theoretically clean, practically heavy.

PPO. Replace the constraint with a clipped surrogate:

\[\mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_t\big[\min(r_t(\theta)\hat A_t,\ \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat A_t)\big].\]

Total objective: \(L = \mathcal{L}^{\text{CLIP}} - c_v\mathcal{L}^{\text{VF}} + c_e\mathcal{H}[\pi_\theta]\). PPO is the workhorse: it matches TRPO's stability with vastly simpler implementation and is the default for both classical RL and the early RLHF era.

Watch out

Common PPO implementation pitfalls: failing to normalize advantages, missing the Adam epsilon adjustment, missing orthogonal initialization, unscaled value loss, premature aggregation across rollouts. The "PPO implementation matters" (ICLR 2020) paper enumerates 17 such details that account for orders of magnitude in performance.

6.2.2 SAC: Soft Actor-Critic

Maximum-entropy RL:

\[J(\pi) = \sum_t\mathbb{E}\big[r(s_t, a_t) + \alpha\,\mathcal{H}(\pi(\cdot\mid s_t))\big].\]

Soft Bellman:

\[Q^*(s, a) = r + \gamma\,\mathbb{E}_{s'}\mathbb{E}_{a'\sim\pi}\big[Q^*(s', a') - \alpha\log\pi(a'\mid s')\big].\]

Policy update minimizes \(\mathrm{KL}\big(\pi_\theta(\cdot\mid s)\,\big\|\,\exp(Q/\alpha)/Z\big)\). Two Q-networks (clipped double Q), target networks, automatic temperature tuning (\(\alpha\) adjusted to maintain target entropy). Off-policy; sample-efficient; the gold standard for continuous-control RL when an off-policy critic is feasible.

6.2.3 TD3 / DDPG

Deterministic policy \(\mu_\theta\). DDPG: \(L_Q = \big(r + \gamma Q_{\bar\theta}(s', \mu_{\bar\theta}(s')) - Q(s, a)\big)^2\). TD3 fixes DDPG's instability:

  1. Clipped double-Q (use min of two critics).
  2. Target-policy smoothing: \(\mu(s') + \mathrm{clip}(\mathcal{N}(0, \sigma), -c, c)\).
  3. Delayed policy updates (every \(d\) critic updates).

6.2.4 Model-based RL

Dreamer V3 sketch. RSSM (Recurrent State-Space Model): latent state \(h_t\) (deterministic) + \(z_t\) (stochastic).

\[h_t = f_\phi(h_{t-1}, z_{t-1}, a_{t-1}), \qquad z_t\sim q_\phi(z_t\mid h_t, o_t), \qquad \hat z_t\sim p_\phi(\hat z_t\mid h_t).\]

Losses: reconstruction \(-\log p_\phi(o_t\mid h_t, z_t)\), \(\mathrm{KL}(q\,\|\,\hat z)\) with free-bits, and reward / continue heads with two-hot symlog targets:

\[\mathrm{symlog}(x) = \mathrm{sgn}(x)\log(|x| + 1).\]

Actor optimizes a \(\lambda\)-return on imagined rollouts of horizon \(\sim 16\); critic regressed to the same target.

IRIS, DIAMOND. World models with discrete latents (IRIS uses a tokenizer + Transformer) or with diffusion world models (DIAMOND). Both impressive on Atari at low data.

TD-MPC, TD-MPC2. Model-based policy optimization with a learned latent dynamics model and sampling-based MPC at planning. Strong on continuous-control benchmarks.

6.2.5 Decision Transformer and offline RL

Decision Transformer. Conditioning on returns-to-go: model \(\pi_\theta(a_t\mid R_t, s_t, a_{<t}, R_{<t}, s_{<t})\) trained with cross-entropy on action tokens over expert / suboptimal trajectories with their actual returns. At test, prompt with target return \(R_0\).

Offline RL methods. BCQ: constrain policy to actions seen in dataset. CQL: regularize Q-function to under-estimate out-of-distribution actions: \(L = L_{\text{TD}} + \alpha\big(\mathbb{E}_{a\sim\mu}Q - \mathbb{E}_{a\sim\pi^*_\mathcal{D}}Q\big)\). IQL: learn V and Q via expectile regression, never query Q at OOD actions. Diffusion-QL, IDQL: use diffusion as the policy / behavior cloning regularizer.

6.3 RLHF: from PPO recipe to DPO and beyond

6.3.1 Bradley–Terry reward modeling

Pairwise preferences \((x, y_w, y_l)\) with \(y_w\) preferred. Reward model \(r_\phi\):

\[\mathcal{L}_{\text{RM}} = -\mathbb{E}\big[\log\sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big].\]

At scale, additional best practices: ranking loss (multiple responses ranked, not just pairs), reward model ensembles to reduce overoptimization, length normalization (reward correlated with length is a giveaway).

6.3.2 KL-regularized RLHF (PPO recipe)

\[\max_\pi\ \mathbb{E}_{x\sim\mathcal{D}, y\sim\pi(\cdot\mid x)}\big[r_\phi(x, y)\big] - \beta\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big).\]

Reduce to per-token: \(\tilde r_t = -\beta\big(\log\pi(a_t\mid s_t) - \log\pi_{\text{ref}}(a_t\mid s_t)\big)\) at every step plus the terminal reward; optimize with PPO. The KL term prevents drift from the reference (a frozen SFT model), avoiding reward hacking and degenerate policies.

The \(\pi\) objective \(\to\) closed-form \(\pi^*\):

\[\pi^*(y\mid x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y\mid x)\exp\!\Big(\frac{r(x, y)}{\beta}\Big), \qquad Z(x) = \sum_y\pi_{\text{ref}}(y\mid x)\exp(r(x, y)/\beta).\]

6.3.3 DPO derivation, in full

Solve for \(r\) in terms of \(\pi^*\):

\[r(x, y) = \beta\log\frac{\pi^*(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \beta\log Z(x).\]

Substitute into the BT log-likelihood. The \(\log Z(x)\) term cancels in differences:

Key

DPO loss (no reward model, no PPO, no value function):

\[\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}\,\log\sigma\!\Big(\beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\Big).\]

See DPO.

What DPO assumes (and where it breaks). - Bradley–Terry preferences (transitive); often violated in real data. - Optimal solution under perfect reward; in practice over-fits, can reduce \(\pi_\theta\) on both responses. - Sensitive to reference policy; if \(\pi_{\text{ref}}\) is poor, DPO inherits it.

6.3.4 The post-DPO zoo

IPO. Identity Preference Optimization: squared-margin loss, robust to deterministic preferences:

\[\mathcal{L}_{\text{IPO}} = \mathbb{E}\Big(h_\theta(y_w, y_l; x) - \tfrac{1}{2\beta}\Big)^2, \qquad h_\theta = \log\frac{\pi_\theta(y_w\mid x)\,\pi_{\text{ref}}(y_l\mid x)}{\pi_{\text{ref}}(y_w\mid x)\,\pi_\theta(y_l\mid x)}.\]

KTO. Kahneman–Tversky utility, only needs single-response thumbs-up / thumbs-down (no pairs). Loss:

\[\mathcal{L}_{\text{KTO}} = \mathbb{E}[1 - v_{\text{KTO}}], \qquad v = \begin{cases}\sigma\big(\beta(\log\pi_\theta/\pi_{\text{ref}} - \mathrm{KL}_{\text{ref}})\big) & y\succ\\ \sigma\big(\beta(\mathrm{KL}_{\text{ref}} - \log\pi_\theta/\pi_{\text{ref}})\big) & y\prec\end{cases}\]

plus loss-aversion weighting. Practical when collecting pair labels is expensive.

ORPO. Odds-ratio penalty appended to SFT loss:

\[\mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}}(y_w) - \lambda\log\sigma\!\Big(\log\frac{\mathrm{odds}(\pi_\theta(y_w\mid x))}{\mathrm{odds}(\pi_\theta(y_l\mid x))}\Big).\]

Removes the SFT \(\to\) DPO two-stage pipeline; combines into one.

SimPO. Length-normalized log-prob, drops the reference policy:

\[\mathcal{L}_{\text{SimPO}} = -\log\sigma\!\Big(\frac{\beta}{|y_w|}\log\pi_\theta(y_w\mid x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l\mid x) - \gamma\Big).\]

Why drop the reference? Avoids the constant memory cost of \(\pi_{\text{ref}}\) and length bias. Margin \(\gamma\) prevents over-fitting.

6.3.5 GRPO and the reasoning revolution

Motivation. Per-token PPO on long sequences has high variance; the value function is hard to fit and consumes memory. GRPO (DeepSeek): remove the value function entirely, use group-relative advantage as the baseline.

The GRPO objective. For prompt \(x\), sample a group of \(G\) responses \(\{y_i\}\) with rewards \(r_i\) (programmatic verifier, RM, or rule-based). Group-relative advantage:

\[\hat A_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\}) + \epsilon}.\]

Key

Per-token clipped objective:

\[\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\Big[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\big(\rho_{i,t}\hat A_i,\ \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat A_i\big)\Big] - \beta\,\mathrm{KL}[\pi_\theta\,\|\,\pi_{\text{ref}}],\]

with \(\rho_{i,t} = \pi_\theta(y_{i,t}\mid x, y_{i,<t}) / \pi_{\text{old}}(y_{i,t}\mid x, y_{i,<t})\).

Variants. - DAPO: decoupled clip (asymmetric upper/lower thresholds), dynamic sampling (re-sample for failed prompts). - Dr. GRPO: drop length normalization to remove the bias toward longer responses; conceptually cleaner. - REINFORCE++ / RLOO: per-token rewards + KL, no clipping ratio. - ReMax, VinePPO: variance reduction via additional baselines.

DeepSeek-R1 and what it demonstrated. Pure-RL on a base LLM (no SFT) with verifiable math / code rewards led to spontaneous emergence of long CoT, self-correction, "aha moments," and doubling the average response length over training. Replicated by community (TinyZero, Open-R1, SimpleRL), then ported to multimodal: Vision-R1, R1-V, MM-EUREKA, VLM-R1, LMM-R1, Video-R1.

6.3.6 Process Reward Models (PRMs)

Per-step labels \(y_t\in\{0, 1\}\) for partial reasoning correctness:

\[\mathcal{L}_{\text{PRM}} = -\sum_t\big[y_t\log p_\phi(y^t) + (1 - y_t)\log(1 - p_\phi(y^t))\big].\]

Auto-labeling via Math-Shepherd / OmegaPRM rollouts: from each prefix, sample \(K\) continuations, label step \(t\) as good if \(>\tau\) fraction succeed. PRMs guide MCTS / tree search at inference, often beating outcome-only training in math.

6.3.7 Inference-time scaling: best-of-N, MCTS, debate

Best-of-N: sample \(N\) responses, pick the one with highest reward. Expected improvement \(\mathbb{E}[\max_{i\le N} r_i]\approx\mu + \sigma\sqrt{2\ln N}\) in a Gaussian approximation. Diminishing returns; practical sweet spot \(N\in[8, 64]\).

MCTS / tree search: rStar, AlphaProof, Mulberry. Particularly effective with PRM as heuristic.

Self-consistency: generate \(K\) CoTs, take majority vote. Cheap, effective for multiple-choice or extractable answers.

Reflexion / debate: agent reviews its own answer; multi-agent debate. Mixed empirical evidence at scale.

6.4 RL for diffusion and generative models

6.4.1 Why RL for diffusion?

Diffusion models trained with MLE produce realistic samples but not necessarily preferred samples. RL fine-tunes for human preference (aesthetics, prompt fidelity, safety) without changing the architecture.

6.4.2 DDPO, DPOK

DDPO: cast diffusion as a multi-step MDP (each denoising step is an action). Use PPO with reward = aesthetic / preference score on the final image. Per-step policy gradient through the sampler. DPOK: simpler, per-batch PPO with KL to base model.

6.4.3 Diffusion-DPO

Lift DPO to diffusion (covered in Part 3): preferences over images, surrogate via diffusion losses. Now standard for aesthetic alignment of SDXL / SD3 / FLUX.

6.4.4 DRaFT, AlignProp, ReFL

Reward backpropagation through the sampler. The full diffusion ODE / SDE is differentiable (with checkpointing); compute \(\partial r(\hat x)/\partial\theta\) end-to-end. Memory-heavy but signal-rich; works for short samplers (e.g., 25-step DDIM).

6.5 RL for robotics, in practice

6.5.1 Sim-to-real with massive parallelism

Isaac Lab (replaces Isaac Gym), MuJoCo MJX, Genesis, ManiSkill 3 enable \(10^4+\) parallel environments on a single GPU. Combined with PPO + domain randomization, trains quadruped locomotion in hours and humanoids in days.

6.5.2 Eureka and DrEureka

Eureka: LLM proposes reward function code; train RL; evaluate task success on a held-out evaluator; refine reward via evolutionary loop. Worked stunningly for shadow-hand pen-spinning (and other tasks where humans struggle to write the reward). DrEureka: extends to dynamics randomization design. The LLM picks both rewards and randomization ranges.

6.5.3 Residual RL on top of pretrained VLA

Given a pretrained VLA (BC-trained), use RL to fine-tune a small residual policy. Reduces sample complexity dramatically vs RL from scratch.

6.5.4 Q-chunking and DPPO

Recent work extends DPO / GRPO ideas to diffusion policies in robotics, treating action chunks as the unit of preference comparison. Early but promising.

6.6 Multimodal RL: VLM-R1, MM-EUREKA, Vision-R1

Same machinery as text-only GRPO. Rewards for multimodal tasks:

The 2025–2026 frontier: extending pure-RL reasoning to video VLMs (Video-R1) and to embodied agents.

6.7 What an interviewer is really probing

RL has become the most consequential technique in modern AI: it's how every frontier model is aligned and how every reasoning capability is unlocked. A principal-level interviewer in this space wants to see (a) you can derive DPO from KL-RLHF and explain why GRPO removes the value head, (b) you can articulate when RL is the right tool vs SFT vs preference data, and (c) you've thought about reward hacking, evaluation contamination, and the open problems (process rewards, offline RL with foundation-model priors, sim-to-real for VLAs).

Interview probe. "Walk me through a complete RLHF pipeline you'd set up to align a 30B-parameter VLM for a customer-support task. Cover data, RM, training algorithm, evals, and the failure modes you're most worried about." — if you can deliver this in 15 minutes, you're at principal level.

Part 7 — Hot Topics 2025–2026

This is the chapter where you defend your view of where the field is going. A principal-level interviewer rewards conviction grounded in evidence, even (especially) for views the interviewer disagrees with.

7.1 Native multimodal: the early-fusion future

GPT-4o, Gemini 1.5/2.0/2.5, Claude 3.5/4 (with vision) are natively multimodal: a single model trained jointly on image, text, audio, and (for Gemini) video. The architectural advantage is not subtle:

The open ecosystem follows: Chameleon, Show-o, Janus, Janus-Pro, Transfusion, Emu3 demonstrate this is not a closed-source-only trick.

Implication. If you're building a CV stack from scratch in 2026 with a 24-month horizon, your default should be to train (or fine-tune) a native multimodal model, not a CLIP+LLaVA adapter. Justify the choice with the specific dense-task / OCR / grounding requirements that adapter-style models struggle with at production accuracy.

★ 2026 SOTA update — Hot Topics 2025-2026

7.2 Test-time compute scaling for visual reasoning

The o1 / DeepSeek-R1 paradigm proved that for a fixed parameter budget above a threshold, test-time chain-of-thought beats additional pretraining for math, code, and reasoning. The critical 2025–2026 finding: this transfers to multimodal, with Vision-R1, MM-EUREKA, VLM-R1, Mulberry, LLaVA-CoT.

The scaling law: at fixed inference cost, longer (or more, or tree-searched) CoT improves accuracy on hard visual reasoning, often by 10–30 absolute points. Compute spent at inference is now a first-class axis alongside parameters and pretraining tokens.

Open problems. - How to elicit good visual CoT without RL on huge programmatic verifiers. - How to verify visual claims (PRM equivalents for images: "is this step's spatial reference correct?"). - How to combine MCTS with multimodal — search branches are expensive when each branch involves vision.

7.3 3D Gaussian Splatting maturity, large-scale scene reconstruction

3DGS is well past the novelty phase. By 2026 the production toolchain looks like:

The next frontier: physical fidelity. 3DGS is geometric and radiometric; physically-grounded extensions (relighting, materials, dynamics) are early but rapidly progressing.

7.4 Feed-forward 3D from images: the SfM-eclipse

DUSt3R / MASt3R / Spann3R / VGGT mark a phase transition. For two-view to ten-view reconstruction, a single transformer forward pass beats classical SfM in both quality and runtime.

This is moving up the stack: - MASt3R-SfM stitches dozens of pairs into a global SfM. - LRM family extends to 3D / mesh / Gaussians. - Dense MVS (PatchMatch) is being absorbed into the same model.

Implication. If your team owns a classical SfM/MVS pipeline, plan its sunset. Within 3 years, a learned feed-forward 3D model will dominate the top of the funnel; classical solvers become refinement / sanity-check fallbacks.

7.5 World models as a unifying frame

Sora, Veo 3, Kling 2, MovieGen, GAIA-2, Cosmos, Genie 2 — different framings of the same idea: a generative video model conditioned on actions / text becomes a learned simulator.

Implications: - Closed-loop AV simulation against real-data distributions. - RL training inside generative simulators (rather than hand-built physics sims). - Embodied agent pretraining via passive video prediction (the V-JEPA bet).

Open problems: long-horizon physical consistency, controllability without sacrificing realism, action tokenization for VLAs.

7.6 Robotic foundation models

\(\pi_0/\pi_{0.5}\), GR00T, Helix, Octo, OpenVLA — the analog of GPT-3 for robotics arrived in 2024–2025. All share:

The open question is whether simulation-data scaling (synthetic) or teleoperation scaling (real) wins. Real data is gold but expensive; sim data scales but suffers reality gap. Most viable bet: hybrid, with sim scaling for breadth and real scaling for fidelity.

7.7 GRPO and pure-RL reasoning ported to vision-language

Covered in Part 6; here, the bigger picture. The 2025 reasoning revolution showed:

The principal-level question: which CV tasks have verifiable rewards we haven't exploited yet? Possible answers: detection (IoU), segmentation (mask-IoU), pose (PCK / MPJPE), classification accuracy, retrieval rank, code-from-image matching, layout reconstruction.

7.8 Diffusion to flow matching: parameterization migration

SD3, FLUX, MovieGen, Cosmos all use flow matching. Pure diffusion is being replaced by FM for new releases. Reasons:

This is not a paradigm shift; it's a parameterization upgrade. But conversationally, you should refer to "flow matching" for new systems and "diffusion" for legacy.

7.9 Mixture-of-experts in vision

DeepSeek-VL2, Aria, MoE-LLaVA, CuMo, Mixtral-VL: MoE applied to VLMs. Per-token routing to \(k\) of \(N\) expert MLPs. Effective parameter count grows; activated parameter count stays constant. For VLMs, expert specialization has emerged for OCR vs general vs document.

Trade-off: routing instabilities, load-balancing, all-to-all communication overhead in distributed training.

7.10 Long video and long context

Gemini 2.5 handles multi-hour video natively. Open models (LongVA, LongVU, LongVILA) close the gap with ring attention + careful training data curation. Applications: surveillance summarization, long-form content moderation, sports analytics, education / lecture video search.

7.11 Vision agents and "computer use"

Claude Computer Use, OS-Atlas, ShowUI, UI-TARS, CogAgent: VLMs grounded in screen pixels that can navigate UIs (click, scroll, type). The recipe: VLM with screen-coordinate grounding, RL or SFT on UI navigation traces. Production deployment by late 2026 is expected for many tools.

7.12 Safety, watermarking, and provenance

7.13 FP8 / FP4 training and Blackwell economics

NVIDIA's Blackwell (B100/B200/GB200) and Hopper (H100/H200) hardware support FP8 training natively. FP4 with double quantization is emerging. Training cost per token has dropped by \(\sim 5\times\) in three years. Implication for org strategy: at fixed budget, you can train \(5\times\) more or train a \(5\times\) larger model.

Part 8 — ML System Design at Scale

Principal-level system design questions are open-ended ("design a CV training/serving system for X"). They reward (a) a structured framework, (b) numerical estimation, (c) trade-off articulation, and (d) operational realism. This chapter gives you the frame and the substance.

8.1 A reusable system-design framework

8.1.1 Five steps

  1. Clarify the problem. Task, scale (QPS, dataset size, image vs video), latency budget, accuracy target, cost ceiling, team / org constraints, geographic distribution, regulatory.
  2. End-to-end data flow. Sources \(\to\) ingest \(\to\) train \(\to\) eval \(\to\) serve \(\to\) monitor. Draw it on the whiteboard before talking about any one box.
  3. Deep dive 2 components. Whichever the interviewer picks. Be ready on every box.
  4. Failure modes. Distribution shift, label noise, long-tail, hardware failures, model regressions, thermal throttling, cold-start, throttle-and-degrade.
  5. Org and rollout. Who owns what, on-call, model release process, kill switch, A/B and shadow traffic.

8.1.2 Numerical estimation that signals seniority

Always come back to numbers. Examples:

8.2 Training infrastructure

8.2.1 Hardware (2026 baseline)

8.2.2 Memory math

For a model with \(P\) parameters trained in mixed precision:

\[\mathrm{Memory} \approx \underbrace{2P}_{\text{BF16 weights}} + \underbrace{4P}_{\text{FP32 master}} + \underbrace{8P}_{\text{Adam } m,v}.\]

Total \(\approx 14P\) bytes for state. For a 70B model, that's \(\sim 980\) GB just for state. ZeRO-3 / FSDP shards across \(D\) devices: state per device \(\approx 14P/D\). Add activations (significant; checkpoint to manage).

8.2.3 Parallelism, composed

Data parallelism (DP). Replicate model across \(D\) devices; split batch. All-reduce gradients per step. Communication scales with model size.

Tensor parallelism (TP, Megatron). Split single layer across devices. For \(W = [W_1, W_2]\) in the MLP, each device computes one column block of the first matmul, all-gather to compute the second matmul. Communication per layer \(\approx \mathrm{batch}\times\mathrm{seq}\times d/TP\).

Pipeline parallelism (PP). Partition layers across devices. "Bubble" fraction \(\approx (PP - 1)/(M + PP - 1)\) for \(M\) microbatches. 1F1B and interleaved schedules reduce bubble; Zero Bubble PP achieves zero bubble in some configurations.

Sequence parallelism (SP). Along the sequence dim for activations of LayerNorm / dropout (the parts not affected by the tensor-parallel split). Complements TP.

Expert parallelism (EP). For MoE: assign each expert to a subset of devices; route tokens to the appropriate device via all-to-all.

Composing. Total devices \(N = DP\cdot TP\cdot PP\cdot SP\cdot EP\). For a large run: - Pick \(TP\le 8\) to keep within an NVLink island. - Pick \(PP\) such that bubble \(\le 5\%\). - SP as needed for sequence \(> \sim 8\text{K}\). - DP to fill remaining devices.

8.2.4 Frameworks

8.2.5 Dataloading at scale

For video pretraining specifically: the bottleneck is rarely compute, it's decode. Solutions: pre-decode and store latents (encode once with VAE, store .pt files), or fully GPU-accelerated decode pipelines.

8.2.6 Numerical and stability

8.2.7 Checkpointing and fault tolerance

At 1024 GPU scale, hardware failures are routine. Strategies: - Hierarchical: local SSD (every \(\sim\) few min) \(\to\) NFS / object store (every \(\sim\) h). - Async / non-blocking: write checkpoint to local while training continues. - Elastic training: restart with different topology after failure. - Hot-swap / on-the-fly node replacement.

8.2.8 Eval harnesses and continuous monitoring during training

8.3 Inference infrastructure

8.3.1 Serving stacks

8.3.2 Optimizations

8.3.3 Diffusion inference specifics

8.3.4 CV-specific inference

8.3.5 On-device / edge

Apple Neural Engine via Core ML, Qualcomm via QNN, MediaTek via NeuroPilot. TFLite / ExecuTorch / ONNX Runtime Mobile. MLC-LLM, llama.cpp, mlx for Apple Silicon.

8.3.6 Cost controls

8.4 Data systems for CV

8.4.1 Dataset construction pipeline

  1. Acquisition: web crawl, licensed partnerships, captured, synthetic.
  2. Deduplication: SimHash / MinHash for near-dup; CLIP embedding NN for semantic dup.
  3. Safety filtering: NSFW classifier, watermark / copyright detection.
  4. Quality scoring: aesthetic models (LAION-aesthetic), CLIP-score, OCR for text, motion-quality for video.
  5. PII detection: face detection, license plate, text OCR + PII classifier.
  6. Captioning / labeling: synthetic captions from a large VLM; bootstrapping from earlier models.
  7. Quality QA: human review on samples.

8.4.2 Labeling

8.4.3 Versioning and lineage

You must be able to answer: "what data was this checkpoint trained on?" and "what changed in the data since the last checkpoint?" — both for debugging and for compliance. Tools: DVC, LakeFS, MLFlow, internal lineage systems.

8.4.4 Storage

8.5 Evaluation and monitoring

8.5.1 Pre-launch eval

8.5.2 Online

8.5.3 Drift and regression detection

8.5.4 Human eval at scale

8.6 Capacity planning sketch

Training: for \(C\) training FLOPs at \(F\) FLOPs/s/device with utilization \(u\) on \(N\) devices:

\[T_{\text{wall}} = \frac{C}{N\cdot F\cdot u}.\]

Typical \(u\in[0.4, 0.55]\) for H100/B200 large-scale runs; lower for non-Transformer workloads.

Compute-optimal scaling. Chinchilla-style: \(D\approx 20P\) training tokens for an LLM. For VLMs, image tokens count and the image-vs-text mix is a hyperparameter; common pretrain mixes are \(\sim 30\)–\(50\%\) image tokens.

Inference cost. For a request taking \(T\) tokens at \(K\) tokens/sec/device, cost \(= T/K\) device-seconds \(= (T/K)\cdot \$/\text{device-second}\). At scale, cache hit rate, batching efficiency, and KV cache size dominate.

Part 9 — Organizational Leadership and Strategy

The technical chapters get you to senior staff. This chapter is what makes you a principal: the ability to operate on multi-quarter, multi-team timescales, to set technical direction the organization can execute against, and to develop other engineers who can do the same.

9.1 Articulating tech vision

9.1.1 The 3–5 year bet

A principal owns at least one 3–5 year technical bet for the org. The bet should be:

Examples of principal-level CV bets you could defend in 2026:

9.1.2 Roadmap structure

Each item ties to a business outcome (revenue, retention, cost) or a strategic capability. Items without that linkage are tech debt or research, both important but different categories.

9.1.3 Tech radar

Maintain an internal radar with categories: adopt, trial, assess, hold. Revisit quarterly. Helps the org avoid both tech debt (sticking with deprecated tools) and shiny-object syndrome (chasing every novel paper). A principal owns the radar; they are accountable for the consequences of misclassification.

9.2 Building and scaling a CV organization

9.2.1 Org shape

Common shapes for a 50–150 person CV org:

The right shape depends on stage: early-stage favors vertical; mid-stage favors research/applied/platform; late-stage favors functional. Restructure when the current shape blocks more value than it creates.

9.2.2 Career frameworks and ladders

Crisp leveling reduces calibration drift and politicking. Principles:

A principal owns at least one engineering or research competency rubric in detail.

9.2.3 Hiring bar

The principal owns the bar. Operational mechanisms:

9.2.4 Mentorship and development

9.2.5 Headcount allocation

Explicit framework for exploration vs exploitation. Common 70/20/10 split:

A principal should be able to defend their org's allocation and articulate when to shift it.

9.3 Decision-making and influence

9.3.1 Tech reviews and design docs

Own the format and the bar. Principles:

A principal should be able to read 5 design docs in a morning and ask the question that saves a quarter of work. That question is almost always: what's the smallest verifiable experiment that would change your mind?

9.3.2 Disagree-and-commit

Healthy orgs make disagreement explicit and time-boxed. Patterns:

Principals model this; they don't snipe at decisions they didn't get.

9.3.3 Build vs buy vs adopt

Default for commodities: buy or adopt open source. Default for differentiated capabilities: build.

9.3.4 Influence without authority

Cross-team work is where principals create or destroy value. Patterns:

9.3.5 Killing projects

Principals are often the people who must call it. The question is rarely "is this technically possible" — usually it is. The question is "is this the best use of the next 6 months for this team." Have a story for at least two projects you've killed.

9.4 Cross-functional and stakeholder management

9.4.1 Working with product

Translate ML capability into product features. Translate product needs into measurable ML targets. Beware of: PMs requesting 100% accuracy; PMs not understanding latency / cost trade-offs; ML engineers shipping features no user asked for.

9.4.2 Working with research

Embed researchers in applied teams when shipping a research bet. Maintain a translation interface (paper \(\to\) design doc \(\to\) prototype \(\to\) productionization \(\to\) launch).

9.4.3 Working with infra

Capacity planning, GPU budgets, network bandwidth, storage tiers, eval infrastructure, CI/CD for models. Principals should sit with infra leadership at least quarterly.

9.4.4 Working with safety and policy

Red-team your model; write a content policy with clear examples; provision for graduated response (warn, refuse, escalate). Document.

Data licensing (especially for image / video corpora), IP, GDPR/AI Act compliance, enterprise deals. The principal should know enough to file the right legal asks; they don't need to be a lawyer.

9.4.6 Executive communication

Always: TL;DR up top, visualized trade-offs, "what we are asking you to do." Never: bury the lede, use jargon without explaining, deliver bad news in person without a written follow-up.

Part 10 — Behavioral and Staff+ Scope Stories

The interviewer's job in a behavioral round is to pattern-match your stories to the level you're applying for. A principal-level story has scope, ambiguity, real trade-offs, sustained influence, and a counterfactual: what would have happened without you.

10.1 The story portfolio

Build 8–12 stories that cover the buckets below. Each story is its own card, which you can recombine to answer any prompt.

10.1.1 The card format

For each story:

Practice each card under 4 minutes; under 90 seconds for the headline + result if asked.

10.1.2 Story buckets

10.2 Common executive-style prompts

10.2.1 "Tell me about a strong technical opinion you hold that the field disagrees with."

You should have one. It should be specific, defendable, and falsifiable. Don't pick a non-controversial opinion ("I think transformers will continue to dominate" — nobody disagrees). Pick something with stake: "I believe the entire industry is over-investing in large diffusion models for video and the right answer is autoregressive video."

10.2.2 "Where will CV be in 5 years?"

Have a layered answer. Short term (12 months): native multimodal dominant; flow matching standard; reasoning VLMs mainstream. Medium (24–36): generative video as world models for AV; robotic foundation models in commercial deployment; on-device VLMs. Long (5y): the discipline "computer vision" likely won't exist as a hiring category; everything is "multimodal foundation models" plus thin domain-specific layers.

10.2.3 "Tell me about a time you changed your mind on something significant."

Specifics. "In 2022 I bet the team on convnets for our segmentation task. Within 9 months DETR-family transformers had made our backbone obsolete; I led the rewrite to a query-based model. The lesson: hold strong opinions weakly, and assign 10% of the team's bandwidth to revisiting your priors quarterly."

10.2.4 "What's the worst technical decision you've made?"

A specific, recent, real failure. Articulate the bias that produced it ("I assumed batch eval was a fair proxy for online; the long-tail showed up only in production"). Articulate what you do now to avoid the same mistake.

10.2.5 "How do you decide what NOT to work on?"

The principal answer: "I write the inverse roadmap. For every quarter I commit, I publish a list of things we explicitly de-prioritize, with a one-sentence explanation each. That makes the trade-off legible to the org and reduces stealth work on deprecated items."

10.2.6 "How would you spend $50M of GPU budget over 18 months?"

Have a layered answer. "50% on core training (one or two flagship models). 25% on data engine (auto-labeling at scale, synthetic generation, eval infrastructure). 15% on inference for product. 10% on exploratory bets (with clear kill criteria)."

10.2.7 "How do you keep up with the field without drowning?"

A concrete system. "Weekly internal reading group with a curated 10-paper list. Daily 30-min skim of arXiv. Monthly synthesis essay: I write 1 page summarizing what changed and what we should do about it. Quarterly recalibration of the team's tech radar."

10.3 The conversational shape of a behavioral round

A staff+ behavioral round usually flows: the interviewer asks an open-ended prompt; you tell a story; they probe the specifics. Their questions will increasingly drill into the decisions you made, not the events. "At that moment, what alternatives did you consider?" "Why did you not just X?" "How did you know Y was the right person to lead?" "What did you tell your skip-level when this happened?"

10.3.1 What good looks like

10.3.2 Anti-patterns

Part 11 — Coding Round Survival Kit

Even at principal level, expect one coding question. They tend to be CV-flavored. The bar is fluency, communication, and judgment — not memorization of obscure algorithms.

11.1 What you should be able to write from scratch in 25 minutes

11.1.1 Detection / segmentation primitives

IoU.

def iou(b1, b2):
    # b1, b2: (x1, y1, x2, y2)
    xa = max(b1[0], b2[0]); ya = max(b1[1], b2[1])
    xb = min(b1[2], b2[2]); yb = min(b1[3], b2[3])
    inter = max(0, xb - xa) * max(0, yb - ya)
    a1 = (b1[2] - b1[0]) * (b1[3] - b1[1])
    a2 = (b2[2] - b2[0]) * (b2[3] - b2[1])
    return inter / (a1 + a2 - inter + 1e-9)

Vectorized version (NumPy / PyTorch broadcasting): standard, write it cleanly.

NMS.

def nms(boxes, scores, thresh):
    order = np.argsort(scores)[::-1]
    keep = []
    while order.size > 0:
        i = order[0]; keep.append(i)
        ious = iou_batch(boxes[i:i+1], boxes[order[1:]])
        order = order[1:][ious[0] <= thresh]
    return keep

Soft-NMS variant: \(s_j \leftarrow s_j\exp(-\mathrm{IoU}^2/\sigma)\).

mAP. For each class, sort detections by score; sweep threshold; compute precision \(P = TP/(TP+FP)\), recall \(R = TP/(TP+FN)\). AP = area under interpolated PR curve. COCO mAP averages over IoU thresholds \(\{0.50, 0.55, \ldots, 0.95\}\) and all classes.

11.1.2 Image processing

Bilinear resize. For each output pixel \((u, v)\), find its float source coordinate \((x, y) = ((u + 0.5)\,s_x - 0.5,\ (v + 0.5)\,s_y - 0.5)\). Floor to \((x_0, y_0)\); weights are fractional parts \(a = x - x_0\), \(b = y - y_0\):

\[\hat I(u, v) = (1-a)(1-b)I(x_0, y_0) + a(1-b)I(x_0+1, y_0) + (1-a)b\,I(x_0, y_0+1) + ab\,I(x_0+1, y_0+1).\]

Be careful about half-pixel offsets; OpenCV and PyTorch differ in default conventions.

2D convolution from scratch. im2col the input into a \(C\cdot k_h\cdot k_w\times H'\cdot W'\) matrix; reshape kernel to \(C_{\text{out}}\times C\cdot k_h\cdot k_w\); matmul; reshape. Same as what cuDNN ultimately does for many sizes.

11.1.3 Deep-learning primitives

Minimal ViT block.

class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.ln1 = nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, h, batch_first=True)
        self.ln2 = nn.LayerNorm(d)
        self.mlp = nn.Sequential(nn.Linear(d, 4*d), nn.GELU(), nn.Linear(4*d, d))
    def forward(self, x):
        x = x + self.attn(self.ln1(x), self.ln1(x), self.ln1(x), need_weights=False)[0]
        x = x + self.mlp(self.ln2(x))
        return x

Minimal diffusion training step.

def diffusion_step(model, x0, t, alphas_cumprod):
    eps = torch.randn_like(x0)
    a = alphas_cumprod[t].view(-1, 1, 1, 1)
    xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
    eps_pred = model(xt, t)
    return F.mse_loss(eps_pred, eps)

DDIM sampling.

@torch.no_grad()
def ddim_sample(model, shape, alphas_cumprod, steps):
    x = torch.randn(shape, device=device)
    ts = torch.linspace(len(alphas_cumprod)-1, 0, steps).long()
    for i in range(steps - 1):
        t, t_next = ts[i], ts[i+1]
        a, a_next = alphas_cumprod[t], alphas_cumprod[t_next]
        eps = model(x, t.repeat(shape[0]))
        x0_hat = (x - (1-a).sqrt() * eps) / a.sqrt()
        x = a_next.sqrt() * x0_hat + (1 - a_next).sqrt() * eps
    return x

11.1.4 Geometry primitives

3D \(\to\) 2D projection.

def project(K, R, t, X):
    Xc = R @ X + t
    x = K @ Xc
    return x[:2] / x[2:3]

PnP outline (DLT then iterative). For each \(X_i\leftrightarrow x_i\), write 2 linear equations on \(P\in\mathbb{R}^{3\times 4}\). Stack, solve via SVD (\(\hat P\) = smallest right-singular vector reshaped). Decompose \(\hat P = K[R\mid t]\) via QR. Refine via Gauss–Newton on reprojection error.

Homography via DLT. Same pattern: 4 correspondences give 8 equations on \(H\in\mathbb{R}^{3\times 3}\). Solve via SVD; normalize.

11.1.5 Algorithmic patterns

11.2 How to communicate during a coding round

  1. Restate the problem in your own words. Confirm assumptions: input shape, dtype, batch.
  2. Clarify edge cases: empty input, NaN, zero-area boxes.
  3. Sketch the approach in 30 seconds before writing code.
  4. Annotate tensor shapes inline as comments: # x: (B, C, H, W).
  5. Default to vectorization; mention loop fallback.
  6. Talk through numerical stability points (log-sum-exp, sqrt clamps).
  7. Write a 1-line sanity test on toy input.
  8. If stuck: think out loud about what would constrain the solution.

Part 12 — Curated Reading List

You don't need to have read every paper. You need to be able to discuss any one of these at the level of: what's the contribution, why does it matter, what are the limitations, what would I do next.

12.1 Foundation models and generative

12.2 3D / AV / Robotics

12.3 AR/VR and video understanding

12.4 Systems and efficiency

Part 13 — Day-of-Interview Tactics

13.1 Logistics

Sleep. Two coffees max. Eat something with protein. Have water. Charge your laptop. Test the conferencing audio at least 30 minutes before. Have a hard line / hot-spot backup.

13.2 Whiteboard / virtual board

13.3 Time management

13.4 When stuck

Think out loud. Articulate the constraints that the solution must satisfy. Often the structure of the question reveals the answer.

13.5 The closing question

Have 3–5 sharp questions ready. At principal level the best ones probe org dynamics:

13.6 After the loop

Within 24 hours, send a tailored thank-you to each interviewer that references something specific from the conversation. (Generic thank-yous are worse than no thank-you.)

Appendix: Ten Derivations You Must Own Cold

  1. ELBO from Jensen \(\to\) VAE objective.
  2. ELBO \(\to\) DDPM simplified loss (the variance-cancelling step).
  3. DDPM noise prediction \(\Leftrightarrow\) score matching: \(\epsilon_\theta = -\sigma_t s_\theta\).
  4. DPO from KL-constrained RLHF: closed-form policy \(\to\) Bradley–Terry on log-ratios.
  5. Policy gradient theorem from \(J(\theta) = \sum_s d^\pi(s)\sum_a\pi(a\mid s)Q^\pi(s, a)\).
  6. PPO clipped surrogate \(\to\) trust-region intuition.
  7. GRPO advantage and why removing the value head reduces variance / cost.
  8. Flow matching from continuous normalizing flows: continuity equation and the conditional FM objective.
  9. Eckart–Young in two lines (SVD + orthogonal decomposition).
  10. NeRF discretization from the volume-rendering equation (alpha-compositing).

If you can do all ten without notes, you can survive any research-deep round at the principal level.