ML Python Debugging Interview Problems — Complete Prep Guide

A collection of 62 realistic debugging problems of the kind asked in ML engineer / applied scientist / research engineer interviews — organized in six parts: classic ML & PyTorch (Parts I & III, Problems 1–12 and 23–34), transformers, LLMs & pre-training (Parts II & IV, Problems 13–22, 35–42, and 51–54), generative models — diffusion, flow matching, VAEs, GANs, tokenizers (Part V, Problems 43–46 and 55–58), and post-training, evaluation & serving (Part VI, Problems 47–50 and 59–62). Each problem contains: the buggy code, the observed symptom, a step-by-step diagnostic analysis (how you should reason out loud in the interview), the fix, and the deeper lesson interviewers are probing for.

Frequency tiers — how to prioritize your prep

Every problem is tagged with a frequency tier:

• Tier 1 — Core screener. Shows up in virtually every ML coding/debugging interview regardless of role. You should solve these in under 5 minutes with a crisp narration. Miss one of these and the interview is likely over.
• Tier 2 — Role-standard. Very common in loops for the relevant specialty (LLM engineering, GenAI, ML infra, applied ML) and expected at mid/senior level. Interviewers use these to check you've actually trained real models.
• Tier 3 — Differentiator. Asked less often, but strongly separates senior/staff and research-engineer candidates. Nailing the diagnostic procedure on these — not just the fix — is what earns "strong hire."

Tiers are role-dependent: for an LLM/post-training role, Problems 13, 14, and 21 behave like Tier 1; for a diffusion/GenAI vision role, Problem 18 does. The tags below note these shifts.

Suggested prep order: all Tier 1 first (Problems 1–5, 7, 10, 13, 14), then the Tier 2 set matching your target role, then Tier 3 as differentiators.

How interviewers evaluate ML debugging

Before the problems, understand what is actually being graded:

1. Hypothesis-driven debugging. Strong candidates state a hypothesis ("loss is flat, so either gradients aren't flowing or the labels are decoupled from the inputs"), then propose the cheapest experiment to test it (print param.grad, overfit a single batch).
2. Knowing the classic failure taxonomy. Most ML bugs fall into a few families: data leakage, shape/broadcasting silent errors, optimizer/gradient plumbing, train-vs-eval mode mismatches, numerical instability, and randomness/reproducibility.
3. The "overfit one batch" reflex. A healthy model must be able to memorize 10 samples. If it can't, the bug is in the plumbing, not the data or the hyperparameters.
4. Reading symptoms like a clinician. Loss = exactly ln(num_classes) at init and never moving, loss = NaN at step 137, val accuracy of 99.8% that collapses in production — each symptom implicates a specific family of bugs.

Part I — Classic ML & PyTorch fundamentals (Problems 1–12)

Problem 1 — The loss that never moves (optimizer plumbing)

Category: PyTorch training loop / gradient flow Difficulty: Warm-up, but with a twist most candidates miss Frequency tier: Tier 1 — core screener (asked in virtually every ML coding/debugging interview, all roles)

Buggy code

import torch
import torch.nn as nn

class MLP(nn.Module):
    def __init__(self, in_dim=20, hidden=64, out_dim=3):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden, out_dim)

    def forward(self, x):
        return self.fc2(self.relu(self.fc1(x)))

model = MLP()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
criterion = nn.CrossEntropyLoss()

for epoch in range(20):
    for xb, yb in train_loader:        # mini-batches of 64
        logits = model(xb)
        loss = criterion(logits, yb)
        loss.backward()
        optimizer.step()
    print(f"epoch {epoch}: loss={loss.item():.4f}")
# epoch 0: loss drops nicely... epoch 3: loss=6.2  epoch 19: loss=18.1

Symptom

Loss decreases for the first epoch or two, then climbs and oscillates wildly, ending far above where it started. (With Adam instead of SGD the failure is subtler: training doesn't blow up, it just stalls at a much worse loss than it should.)

Diagnostic reasoning (say this out loud)

• "Trained briefly, then diverged" points at the update rule, not the data: the effective step size is growing over time.
• First hypothesis: LR too high. But the LR is constant — something else is scaling the updates.
• Second hypothesis: gradients are wrong. Print model.fc1.weight.grad.norm() each step — it grows monotonically even when the loss is flat. Gradients shouldn't grow if the weights aren't moving much.
• Read the loop carefully: optimizer.zero_grad() is missing. PyTorch accumulates gradients across backward() calls, so each step applies the running sum of every gradient since the start of training. The effective learning rate grows without bound → divergence with SGD.
• Verified empirically (mini-batch SGD, lr=0.05, 20 epochs): final loss 18.18 without zero_grad vs 0.126 with it. With Adam the per-parameter normalization hides the blow-up, but the stale accumulated direction still degrades convergence (0.71 vs 0.084 in the same setup) — a nastier version of the bug because nothing obviously breaks.

Fix

for epoch in range(20):
    for xb, yb in train_loader:
        optimizer.zero_grad()          # <-- the fix
        loss = criterion(model(xb), yb)
        loss.backward()
        optimizer.step()

Deeper lesson

• PyTorch accumulates gradients by design (to support gradient accumulation for large effective batch sizes). zero_grad() is your responsibility.
• Modern idiom: optimizer.zero_grad(set_to_none=True) (default since PyTorch 2.0) — frees memory and is marginally faster.
• Interview follow-up you should volunteer: gradient accumulation done intentionally looks like calling zero_grad() every N steps and dividing the loss by N.
• Notice how the optimizer changes the symptom of the same bug: SGD diverges loudly, Adam quietly underperforms. This is why "it trains, just not great" deserves the same scrutiny as a crash.

Problem 2 — 99.5% validation accuracy, 62% in production (data leakage)

Category: Data preprocessing / leakage Difficulty: Medium — the single most common real-world ML bug Frequency tier: Tier 1 — core screener (the most common real-world ML bug; any applied role)

Buggy code

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X, y = load_patient_data()          # features include lab results
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # BUG 1: fit on ALL data

# BUG 2: duplicate rows from repeated patient visits
X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y, test_size=0.2, random_state=42
)

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print("val acc:", clf.score(X_val, y_val))   # 0.995 (!)

Symptom

Validation accuracy is suspiciously high; the deployed model performs far worse on new patients.

Diagnostic reasoning

• Rule of thumb: when validation looks too good, suspect leakage before celebrating.
• Leak #1 — preprocessing leakage: StandardScaler.fit was called on the full dataset, so the validation rows' means/variances influenced the training features. For scaling this is a mild leak, but the same pattern with target encoding, feature selection, or imputation is catastrophic. The interview point is the pattern, not the magnitude.
• Leak #2 — group leakage: the same patient appears in both train and validation (repeated visits, near-duplicate rows). The model memorizes patient-specific signatures instead of generalizable signal. This is usually the dominant leak and explains 99.5% → 62%.
• How to detect: check df.duplicated().sum(), check whether any grouping key (patient_id, user_id, session_id) spans both splits, and compare val performance against a grouped split.

Fix

from sklearn.model_selection import GroupShuffleSplit
from sklearn.pipeline import make_pipeline

# Split FIRST, and split by patient so no patient straddles the boundary
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, val_idx = next(gss.split(X, y, groups=patient_ids))

X_train, X_val = X[train_idx], X[val_idx]
y_train, y_val = y[train_idx], y[val_idx]

# Pipeline guarantees the scaler is fit only on training folds,
# including inside cross-validation
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
clf.fit(X_train, y_train)
print("val acc:", clf.score(X_val, y_val))

Deeper lesson

• The golden rule: anything learned from data must be learned from training data only — scalers, imputers, vocabularies, target encoders, feature selectors, PCA, even "which features looked promising in EDA."
• For time series, the analogue is temporal leakage: always split by time, never randomly.
• sklearn.pipeline.Pipeline exists precisely to make this correctness composable with cross_val_score / GridSearchCV.

Problem 3 — The double softmax (loss/activation mismatch)

Category: Loss function semantics Difficulty: Medium — extremely common in PyTorch Frequency tier: Tier 1 — core screener (top-3 most frequent PyTorch questions)

Buggy code

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10)
        )

    def forward(self, x):
        logits = self.backbone(x)
        return torch.softmax(logits, dim=1)   # BUG

criterion = nn.CrossEntropyLoss()
# ... training loop uses criterion(model(x), y)

Symptom

The model trains, but slowly, plateaus early, and confidence calibration is bizarre. Nothing crashes — this is a silent bug.

Diagnostic reasoning

• nn.CrossEntropyLoss = LogSoftmax + NLLLoss. It expects raw logits.
• Feeding it softmax outputs applies softmax twice. The composition softmax(softmax(z)) still produces a valid distribution, so nothing errors — but the effective function is heavily flattened: probabilities are squashed toward uniform, gradients shrink dramatically, and the model can never express high confidence (max achievable "logit" into the second softmax is 1.0).
• How to catch it: print the loss at initialization. With 10 classes it should be ≈ ln(10) = 2.303. With the double softmax it will start near 2.30 too, but after training it can never get close to 0 — the loss floor is elevated. Also inspect the model's outputs: if they already sum to 1 before the loss, that's the tell.

Fix

def forward(self, x):
    return self.backbone(x)        # return raw logits

# At inference time only:
probs = torch.softmax(model(x), dim=1)

Deeper lesson

• Same family of bugs: BCELoss with logits (should be BCEWithLogitsLoss — also more numerically stable via the log-sum-exp trick), NLLLoss fed probabilities instead of log-probabilities.
• Keras analogue: from_logits=True mismatch with a final softmax layer.
• Rule: losses own the activation. Keep models returning logits; apply activations only at inference/serving boundaries.

Problem 4 — NaN loss at step 137 (numerical instability)

Category: Numerical stability Difficulty: Medium-hard Frequency tier: Tier 1 — core screener (every deep-learning loop has a NaN question)

Buggy code

def custom_loss(pred, target, eps=0.0):
    # focal-style loss written by hand
    p = torch.sigmoid(pred)
    loss = -(target * torch.log(p) + (1 - target) * torch.log(1 - p))
    return loss.mean()

optimizer = torch.optim.SGD(model.parameters(), lr=1.0)  # aggressive LR

for step, (x, y) in enumerate(loader):
    optimizer.zero_grad()
    loss = custom_loss(model(x), y.float())
    loss.backward()
    optimizer.step()
    # step 0..136: loss decreasing... step 137: loss=nan, forever after: nan

Symptom

Training is healthy, then the loss becomes NaN and never recovers. Classic "it worked for a while" failure.

Diagnostic reasoning — the NaN debugging playbook

1. Localize when: NaN mid-training (not step 0) usually means values drifted into an unstable region — exploding logits, extreme learning rate, or an unstable formula — rather than bad input data (which would NaN immediately).
2. Localize where: enable torch.autograd.set_detect_anomaly(True) to get the exact op in the backward graph that produced the NaN, or register forward hooks checking torch.isfinite(out).all().
3. Read the math: torch.log(p) where p = sigmoid(pred). When pred is very negative, sigmoid underflows to exactly 0.0 in float32, and log(0) = -inf; 0 * -inf = nan. Symmetrically log(1-p) blows up for large positive pred. The aggressive LR pushed logits into the saturation zone by step 137.
4. Check the amplifiers: lr=1.0 with SGD on raw logits accelerates the drift; no gradient clipping means one bad batch can launch the weights.

Fix

# Best fix: use the numerically stable fused op — it applies log-sum-exp internally
criterion = nn.BCEWithLogitsLoss()
loss = criterion(model(x), y.float())

# If you must hand-roll, clamp the probabilities:
def custom_loss(pred, target, eps=1e-7):
    p = torch.sigmoid(pred).clamp(eps, 1 - eps)
    return -(target * torch.log(p) + (1 - target) * torch.log(1 - p)).mean()

# And defensive training hygiene:
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Deeper lesson

• Unstable formulas to recognize on sight: log(softmax(x)) (use log_softmax), log(sigmoid(x)) (use logsigmoid/BCEWithLogitsLoss), naive softmax without max-subtraction, sqrt(x) near 0 in backward, x / (norm + 0), exp() of large values, division in cosine similarity without eps.
• Mixed precision (fp16) shrinks the safe range further — this is why AMP uses a GradScaler. Under bf16/fp16, mention loss-scaling and keeping reductions in fp32.
• Once weights are NaN, every future step is NaN — checkpointing + NaN detection lets you restart from the last healthy state.

Problem 5 — Great training accuracy, terrible eval — and eval changes every run (mode mismatch)

Category: train/eval semantics — Dropout & BatchNorm Difficulty: Medium Frequency tier: Tier 1 — core screener (train/eval semantics are near-universal)

Buggy code

model = nn.Sequential(
    nn.Linear(100, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.5),
    nn.Linear(256, 10),
)

def evaluate(model, loader):
    correct = 0
    total = 0
    for x, y in loader:                      # BUG 2: no torch.no_grad()
        preds = model(x).argmax(dim=1)       # BUG 1: model still in train mode
        correct += (preds == y).sum().item()
        total += y.numel()
    return correct / total

Symptom

Evaluation accuracy is (a) worse than expected, (b) non-deterministic across identical calls, and (c) memory usage grows during evaluation, occasionally OOMing on large eval sets.

Diagnostic reasoning

• Non-deterministic eval on fixed data is the fingerprint of active Dropout — it's still randomly zeroing units because the model was never switched to eval mode.
• BatchNorm in train mode uses per-batch statistics and keeps updating its running mean/var during "evaluation" — so evaluating actually mutates the model, and small/skewed eval batches (e.g., a last batch of size 1) produce garbage normalization. BatchNorm1d with batch size 1 in train mode raises an error — a helpful clue.
• The growing memory: without torch.no_grad(), every forward builds an autograd graph that is never freed by a backward().

Fix

@torch.no_grad()                 # no autograd graph -> fast, low-memory
def evaluate(model, loader):
    model.eval()                 # dropout off; BN uses running stats
    correct, total = 0, 0
    for x, y in loader:
        preds = model(x).argmax(dim=1)
        correct += (preds == y).sum().item()
        total += y.numel()
    model.train()                # restore state for the caller
    return correct / total

Deeper lesson

• model.eval() and torch.no_grad() are orthogonal and you need both: one changes layer behavior, the other disables graph construction. A frequent interview follow-up.
• torch.inference_mode() is the stricter, faster modern alternative to no_grad() for pure inference.
• Related production bug: exporting a model without calling .eval() before tracing/ONNX export bakes train-mode behavior into the artifact.

Problem 6 — The shuffled features, unshuffled labels (silent data misalignment)

Category: NumPy/pandas data handling Difficulty: Medium — brutal because everything "runs" Frequency tier: Tier 2 — role-standard (common in data-heavy / applied-ML loops)

Buggy code

import numpy as np

X = np.load("features.npy")       # (N, D)
y = np.load("labels.npy")         # (N,)

# shuffle before splitting
np.random.shuffle(X)              # BUG: X and y shuffled independently...
np.random.shuffle(y)              # ...breaking row correspondence

split = int(0.8 * len(X))
X_train, y_train = X[:split], y[:split]
X_val,   y_val   = X[split:], y[split:]

A pandas variant of the same disease:

df = df.sort_values("timestamp")
features = df[FEATURE_COLS].reset_index(drop=True)
labels   = raw_labels_series          # still carries the OLD index
aligned  = pd.concat([features, labels], axis=1)   # aligns on index -> scrambled/NaN

Symptom

Training loss decreases only slightly then plateaus at roughly the entropy of the label distribution; accuracy ≈ majority-class rate. The model behaves as if labels were random — because they now are.

Diagnostic reasoning

• Accuracy pinned at the class prior (e.g., ~52% on a balanced-ish binary task) with a loss stuck near ln(K) screams "labels are independent of features."
• The definitive test: overfit a tiny subset. Take 16 (x, y) pairs and train to zero loss. With scrambled labels, the model can still memorize 16 random pairs — so also run the complement test: check label agreement directly, e.g., visualize samples with their labels, or verify a known-signal feature correlates with y.
• Root cause: np.random.shuffle shuffles in place, independently per call. Two calls = two different permutations.
• The pandas variant: pd.concat/assignment aligns on the index, not position. After reset_index on one side only, rows pair up wrong — or produce NaNs that a later .fillna(0) quietly buries.

Fix

# One permutation applied to both arrays
rng = np.random.default_rng(42)
perm = rng.permutation(len(X))
X, y = X[perm], y[perm]

# or shuffle at split time and keep pairs together:
from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# pandas: align positionally on purpose
labels = raw_labels_series.reset_index(drop=True)
aligned = pd.concat([features, labels], axis=1)

Deeper lesson

• Add an alignment assertion to pipelines: assert len(X) == len(y) is necessary but not sufficient — better is carrying an ID column end-to-end and asserting IDs match after every join/shuffle/filter.
• Know pandas index-alignment semantics cold: it is the #1 source of silent data corruption in pandas code.

Problem 7 — The broadcasting bug that halves your metric (silent shape mismatch)

Category: NumPy/PyTorch broadcasting Difficulty: Medium — a favorite short screener question Frequency tier: Tier 1 — core screener (favorite 5-minute filter question)

Buggy code

import torch

preds  = model(x)                # shape (B, 1)  — regression head keeps last dim
target = batch["label"]          # shape (B,)

loss = torch.nn.functional.mse_loss(preds, target)
# UserWarning: Using a target size (torch.Size([64])) that is different
# to the input size (torch.Size([64, 1])) ... (warning often scrolls by unread)

What actually happens:

diff = preds - target        # (B,1) - (B,) broadcasts to (B, B)!!
loss = (diff ** 2).mean()    # mean over B*B pairwise differences

Symptom

The loss value looks plausible and even decreases, but the model converges to predicting roughly the mean of the targets and R² is terrible. GPU memory is also mysteriously high for large batch sizes (the hidden (B, B) tensor).

Diagnostic reasoning

• (B,1) vs (B,) broadcast to (B,B): every prediction is compared against every target. Minimizing the mean of all pairwise squared differences drives predictions toward the global target mean — which is exactly the degenerate behavior observed.
• The framework did warn — treating warnings as errors in CI (python -W error::UserWarning) turns this from a silent bug into a loud one.
• Quadratic memory in batch size is the second clue: memory scaling ∝ B² instead of B.

Fix

loss = F.mse_loss(preds.squeeze(-1), target)      # shapes (B,) vs (B,)
# or: F.mse_loss(preds, target.unsqueeze(-1))     # shapes (B,1) vs (B,1)

# Defensive: assert shapes at the loss boundary
assert preds.squeeze(-1).shape == target.shape, (preds.shape, target.shape)

Deeper lesson

• Same disease appears as: cosine similarity between (B,D) and (D,), attention masks of (B,L) vs (B,1,L) applied wrong, argmax(dim=?) on the wrong axis, keepdim=True/False confusion in reductions.
• Habits that prevent it: shape-comment every tensor (# (B, L, D)), use einops.rearrange/reduce which make shapes explicit and assert them, and unit-test losses with asymmetric shapes (B=2, D=3, never square) so transposition bugs can't hide.

Problem 8 — In-place ops and a detached graph (autograd surgery)

Category: Autograd mechanics Difficulty: Hard Frequency tier: Tier 2 — role-standard (expected at mid/senior level for PyTorch-heavy roles)

Buggy code

class GatedBlock(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.lin = nn.Linear(d, d)
        self.gate = nn.Linear(d, d)

    def forward(self, x):
        g = torch.sigmoid(self.gate(x))
        h = self.lin(x)
        g *= h                    # BUG A: in-place on a tensor needed in backward
        return g

# elsewhere, a "warm-up trick" someone added:
def forward_features(model, x):
    with torch.no_grad():         # BUG B: entire feature extractor detached
        feats = model.backbone(x)
    return model.head(feats)      # only the head ever trains

# and a metric helper:
def accuracy(logits, y):
    preds = logits.detach().argmax(1)     # fine
    return (preds == y).float().mean()

running_loss += loss              # BUG C: accumulating the TENSOR, not loss.item()

Symptoms

• Bug A: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation — sometimes only after a refactor changes the graph.
• Bug B: no error at all; training "works" but the backbone's weights never change and final accuracy is mediocre.
• Bug C: no error; GPU memory climbs every step until OOM near the end of the first epoch.

Diagnostic reasoning

• A: g *= h mutates g in place. Sigmoid's backward is σ'(z) = σ(z)(1 − σ(z)) — it is computed from the saved output g, so mutating g invalidates the saved tensor and backward raises a version-counter mismatch error. set_detect_anomaly(True) points to the offending op. The subtle part (verified): h *= g on the Linear output would not error, because Linear's backward needs its input and weight, not its output. Whether an in-place op is legal depends on what each op's backward saved — which is why these bugs appear and disappear across refactors. In-place ops (*=, relu_(), tensor[mask] = 0) are only safe when the overwritten value is not needed for any backward computation.
• B: The canonical check for "is my model actually training?": after loss.backward(), inspect [(n, p.grad is None or p.grad.abs().sum().item()) for n, p in model.named_parameters()]. All backbone grads are None → the graph never reached them → search for no_grad / detach / .data on the forward path. Also compare parameter checksums between epochs.
• C: running_loss += loss keeps a reference to the whole graph of every step (each loss tensor retains its autograd history). Memory grows linearly with steps. The tell: memory grows within an epoch, resets at epoch boundaries if running_loss is reinitialized.

Fix

# A: allocate a new tensor
out = g * h

# B: remove no_grad from the training path (keep it for frozen-backbone
# inference only, and if freezing is intended, do it explicitly):
for p in model.backbone.parameters():
    p.requires_grad = False       # explicit, documented freezing

# C: accumulate a python float
running_loss += loss.item()       # .item() detaches AND moves to CPU

Deeper lesson

• .detach() vs .data: both give a graph-free view, but .data silently skips version tracking and can corrupt gradients — never use .data in modern code.
• Hidden detach sources interviewers love: converting to numpy and back (torch.tensor(t.numpy())), .item() inside the loss, integer casts, and torch.no_grad() decorators applied too broadly.
• Retained-graph OOM also occurs with: storing losses/logits in a list for logging, and appending loss to a metrics dict — always .item() or .detach() at the logging boundary.

Problem 9 — The DataLoader that feeds identical "random" augmentations (randomness & workers)

Category: Reproducibility, multiprocessing, NumPy seeding Difficulty: Hard — a well-known real PyTorch footgun Frequency tier: Tier 3 — differentiator (senior/staff depth; famous real-world footgun)

Buggy code

import numpy as np
from torch.utils.data import Dataset, DataLoader

class AugmentedDataset(Dataset):
    def __init__(self, images):
        self.images = images

    def __getitem__(self, idx):
        img = self.images[idx]
        # numpy-based augmentation
        noise_scale = np.random.uniform(0.0, 0.1)     # BUG (with workers > 0)
        crop_x = np.random.randint(0, 8)
        return augment(img, noise_scale, crop_x)

    def __len__(self):
        return len(self.images)

loader = DataLoader(AugmentedDataset(images), batch_size=32,
                    num_workers=4, shuffle=True)

Symptom

Augmentation "works" but the model overfits more than expected, and inspecting augmented samples reveals that groups of samples share identical augmentation parameters — every epoch, the same pattern.

Diagnostic reasoning

• With num_workers > 0, workers are created by forking the parent process (on Linux). Each fork inherits an identical copy of NumPy's global RNG state. In older PyTorch versions all workers then produce the same "random" sequence; in newer versions PyTorch reseeds torch's RNG per worker, but NumPy's and Python's random state handling still trips people up, and with persistent_workers=False the workers are re-forked every epoch from the same parent state → identical augmentations every epoch.
• How to catch: log the augmentation parameters with the worker id (torch.utils.data.get_worker_info().id) for one epoch and diff across workers/epochs.
• This generalizes: any library with global RNG state (NumPy, random, OpenCV) + process forking = correlated randomness.

Fix

def worker_init_fn(worker_id):
    # derive a distinct, epoch-varying seed for numpy & random in each worker
    seed = torch.initial_seed() % 2**32
    np.random.seed(seed)
    import random; random.seed(seed)

g = torch.Generator()
g.manual_seed(42)                          # reproducible shuffling

loader = DataLoader(dataset, batch_size=32, num_workers=4, shuffle=True,
                    worker_init_fn=worker_init_fn, generator=g)

# Best practice: avoid global RNGs entirely — use torch ops for augmentation
# randomness, or a per-sample np.random.default_rng(seed + idx + epoch * N).

Deeper lesson

• Full reproducibility checklist for interviews: torch.manual_seed, np.random.seed, random.seed, torch.use_deterministic_algorithms(True), torch.backends.cudnn.deterministic = True; benchmark = False, seed workers, seed the DataLoader generator, and set PYTHONHASHSEED. Also mention that some CUDA ops (e.g., atomicAdd-based scatter) are fundamentally nondeterministic.
• The mirror-image bug: seeding too aggressively inside __getitem__ (e.g., np.random.seed(idx)) gives every epoch identical augmentations — randomness must vary across epochs but be reconstructible.

Problem 10 — The mutable default and the config that mutates itself (Python semantics in ML code)

Category: Core Python — the ML-flavored classics Difficulty: Easy-medium individually; often combined into one review problem Frequency tier: Tier 1 — core screener (general Python rounds love these in ML dress)

Buggy code

# (a) mutable default argument
def make_optimizer(params, lr=1e-3, betas=[0.9, 0.999]):
    betas[0] *= 0.5 if warmup else 1.0     # mutates the SHARED default list
    return torch.optim.Adam(params, lr=lr, betas=tuple(betas))

# (b) late-binding closures in a sweep
schedulers = []
for factor in [0.1, 0.5, 0.9]:
    schedulers.append(lambda epoch: base_lr * factor)   # all use factor=0.9

# (c) shallow config copy
base_cfg = {"model": {"hidden": 256}, "lr": 1e-3}
cfg = dict(base_cfg)                # shallow!
cfg["model"]["hidden"] = 512        # ALSO changes base_cfg
run_a = train(base_cfg)             # silently runs with hidden=512

# (d) class-level mutable attribute
class MetricTracker:
    history = []                    # shared across ALL instances
    def log(self, v):
        self.history.append(v)

Symptoms

Sweeps where every run mysteriously uses the last hyperparameter; experiment B contaminating experiment A's config; metrics from different models interleaved in one list; a "default" that drifts over the process lifetime.

Diagnostic reasoning & fixes

• (a) Default arguments are evaluated once at function definition. Every call shares the same list; mutation persists across calls. Fix: betas=None then betas = (0.9, 0.999) if betas is None else betas.
• (b) Closures capture variables, not values; after the loop, factor is 0.9 for all three lambdas. Fix: lambda epoch, factor=factor: base_lr * factor or functools.partial.
• (c) dict(base_cfg) copies one level; nested dicts are shared references. Fix: copy.deepcopy(base_cfg) — or better, immutable config objects (frozen dataclasses, pydantic models).
• (d) history = [] in the class body is a class attribute shared by all instances. Fix: create it in __init__. (Note the trap: self.history.append(...) mutates the shared list, while self.history = [...] would create an instance attribute — which is why the bug appears only with mutation.)

Deeper lesson

The unifying concept is Python's object/reference model: names bind to objects; defaults, class bodies, and closures each evaluate at surprising times. Interviewers use these because ML config/sweep code is exactly where they bite hardest.

Problem 11 — The learning-rate scheduler stepped in the wrong place (training dynamics)

Category: Training loop orchestration Difficulty: Medium Frequency tier: Tier 2 — role-standard (training-orchestration roles; resume-from-checkpoint variants are common)

Buggy code

model = build_model()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=EPOCHS)

for epoch in range(EPOCHS):
    for x, y in train_loader:            # ~1000 batches/epoch
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        scheduler.step()                 # BUG 1: stepped per-BATCH with T_max in EPOCHS
        optimizer.step()                 # BUG 2: also, scheduler before optimizer

    if val_acc > best:
        torch.save(model, "best.pt")     # BUG 3: saving the whole module w/ pickle

Symptoms

• LR collapses to ~0 within the first epoch (cosine schedule designed for EPOCHS steps consumed 1000× too fast); training freezes after a promising start. A UserWarning about calling scheduler.step() before optimizer.step() is emitted and ignored.
• Later, torch.load("best.pt") breaks after a refactor moves the model class (AttributeError: Can't get attribute 'MyModel' on module ...).

Diagnostic reasoning

• Plot or log optimizer.param_groups[0]["lr"] every step — the single most underused debugging trace. Here it reveals LR hitting its minimum ~1 epoch in.
• Unit mismatch: T_max is in scheduler steps. Stepping per batch means T_max should be EPOCHS * len(train_loader) — or step per epoch. Neither is wrong per se; they must simply agree.
• Order matters since PyTorch 1.1: optimizer.step() then scheduler.step(). Reversed order skips the first LR value and (for some schedulers) shifts the whole schedule.
• torch.save(model) pickles the class by reference (module path + name), so loading depends on the code layout at load time. Save state_dict instead.

Fix

steps_per_epoch = len(train_loader)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    optimizer, T_max=EPOCHS * steps_per_epoch
)

for epoch in range(EPOCHS):
    for x, y in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()
        scheduler.step()                 # after optimizer.step()

    if val_acc > best:
        torch.save({
            "model": model.state_dict(),
            "optimizer": optimizer.state_dict(),
            "scheduler": scheduler.state_dict(),
            "epoch": epoch,
        }, "best.pt")

Deeper lesson

• Always checkpoint optimizer + scheduler + epoch + RNG state if you intend to resume; a model-only checkpoint silently restarts Adam's moments and the LR schedule, producing a loss spike on resume — itself a classic interview symptom ("why does my loss jump when I resume from a checkpoint?").
• Warm-up interactions: schedulers that expect initial_lr behave oddly when composed; know SequentialLR / LambdaLR for warmup-then-cosine.

Problem 12 — The 0.91 AUC model that predicts one class (metrics & imbalance)

Category: Evaluation methodology Difficulty: Medium-hard — tests judgment, not just code Frequency tier: Tier 2 — role-standard (Tier 1 in classic-ML / data-science loops)

Buggy code

from sklearn.metrics import accuracy_score
from sklearn.ensemble import RandomForestClassifier

# fraud data: 1.2% positive
clf = RandomForestClassifier().fit(X_train, y_train)

y_pred = clf.predict(X_val)
print("accuracy:", accuracy_score(y_val, y_pred))          # 0.988 — "great!"

# team later switches to AUC:
from sklearn.metrics import roc_auc_score
print("auc:", roc_auc_score(y_val, y_pred))                # BUG: hard labels, not scores

# threshold chosen on the VALIDATION set used for reporting:
best_t = max(np.arange(0, 1, 0.01),
             key=lambda t: f1_score(y_val, proba[:, 1] > t))   # BUG: threshold leakage
print("f1 at best threshold:", ...)                            # reported as final

Symptoms

98.8% accuracy on a 98.8%-negative dataset (the majority-class baseline); an "AUC" computed on hard 0/1 predictions that understates and distorts the true AUC; an F1 that won't reproduce on the test set because the threshold was tuned on the same data used to report it.

Diagnostic reasoning

• Always compare to the trivial baseline. Accuracy 0.988 with prevalence 0.012 means the model may be doing nothing. Check the confusion matrix first — here it shows ~0 predicted positives.
• roc_auc_score(y_val, y_pred) with hard labels computes AUC of a single-threshold classifier (a two-point ROC). It must receive scores: clf.predict_proba(X_val)[:, 1].
• Threshold selection is a form of model selection → doing it on the reporting set is leakage. Tune the threshold on a separate split (or nested CV) and report on untouched data.
• For 1.2% prevalence, discuss PR-AUC / average precision over ROC-AUC (ROC is insensitive to the false-positive rate denominator being huge), calibration, and cost-weighted thresholds.

Fix

proba = clf.predict_proba(X_val)[:, 1]

from sklearn.metrics import (average_precision_score, roc_auc_score,
                             precision_recall_curve, confusion_matrix)
print(confusion_matrix(y_val, clf.predict(X_val)))
print("roc-auc:", roc_auc_score(y_val, proba))             # scores, not labels
print("pr-auc :", average_precision_score(y_val, proba))   # better for 1.2% prevalence

# tune threshold on a dedicated tuning split, evaluate once on test
prec, rec, ts = precision_recall_curve(y_tune, proba_tune)
best_t = ts[np.argmax(2 * prec[:-1] * rec[:-1] / (prec[:-1] + rec[:-1] + 1e-12))]
final_f1 = f1_score(y_test, proba_test > best_t)

# and fix the training itself:
clf = RandomForestClassifier(class_weight="balanced")
# or resampling / focal loss / threshold-moving — discuss trade-offs

Deeper lesson

• The metric hierarchy for imbalance: confusion matrix → precision/recall at business-relevant operating points → PR-AUC → calibrated probabilities. Accuracy is last.
• predict_proba from tree ensembles is poorly calibrated — mention CalibratedClassifierCV if downstream consumers use the probabilities as probabilities.

Part II — Modern generative stacks: Transformers, Pre-training, Diffusion, Flow Matching, Post-training (Problems 13–22)

Problem 13 — The transformer that outputs NaN on the first token (attention masking)

Category: Transformers / attention masks Difficulty: Medium Frequency tier: Tier 2 overall; Tier 1 for any LLM/transformer role — mask bugs are the #1 transformer interview question

Buggy code

import torch, torch.nn as nn, torch.nn.functional as F

class SelfAttention(nn.Module):
    def __init__(self, d, n_heads):
        super().__init__()
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.n_heads, self.d_head = n_heads, d // n_heads

    def forward(self, x, pad_mask):            # pad_mask: (B, L), 1 = real token
        B, L, D = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)
        q = q.view(B, L, self.n_heads, self.d_head).transpose(1, 2)
        k = k.view(B, L, self.n_heads, self.d_head).transpose(1, 2)
        v = v.view(B, L, self.n_heads, self.d_head).transpose(1, 2)

        scores = q @ k.transpose(-2, -1)                       # BUG 1: no 1/sqrt(d_head)

        causal = torch.triu(torch.ones(L, L, dtype=torch.bool,
                                       device=x.device), diagonal=0)   # BUG 2: diagonal=0
        scores = scores.masked_fill(causal, float("-inf"))

        scores = scores.masked_fill(pad_mask[:, None, None, :], float("-inf"))
        # BUG 3: mask polarity flipped — this masks the REAL tokens (1s), keeps padding

        attn = scores.softmax(dim=-1)
        out = (attn @ v).transpose(1, 2).reshape(B, L, D)
        return self.proj(out)

Symptoms

• Forward pass produces NaN at position 0 for every sequence.
• After "fixing the NaN" by clamping, training is glacially slow and attention maps are near one-hot.
• Model outputs change when the same sentence is batched with different-length neighbors — a classic serving bug report: "the model gives different answers depending on batch composition."

Diagnostic reasoning

• NaN at exactly position 0 is a masking fingerprint. torch.triu(..., diagonal=0) includes the diagonal, so every token masks itself; row 0 has no unmasked entries → softmax over all -inf → NaN. Verified: diagonal=0 produces NaN rows; diagonal=1 doesn't. The convention to memorize: mask entries where j > i (strictly upper triangle), so each token sees itself and its past.
• Near one-hot attention + slow training → missing 1/sqrt(d_head) scaling. With unit-variance q,k the score variance grows ∝ d_head; verified with d=512: mean max attention weight 0.94 unscaled vs 0.11 scaled. Saturated softmax ⇒ vanishing gradients through attention.
• Batch-composition-dependent outputs → padding leaking into attention. Here the polarity is inverted: pad_mask==1 marks real tokens, and masked_fill fills where the mask is True — so real tokens are masked and padding attends. Even with correct polarity, remember downstream poolers: with right-padding, "take the last hidden state" grabs a pad position — use the last non-pad index, or left-pad for generation.

Fix

scale = self.d_head ** -0.5
scores = (q @ k.transpose(-2, -1)) * scale

causal = torch.triu(torch.ones(L, L, dtype=torch.bool, device=x.device),
                    diagonal=1)                         # strictly above diagonal
scores = scores.masked_fill(causal, float("-inf"))
scores = scores.masked_fill(~pad_mask[:, None, None, :].bool(), float("-inf"))
attn = scores.softmax(dim=-1)

# In practice, prefer the fused kernel and let it build the mask:
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)  # handles scale + causal

Deeper lesson

• Mask polarity conventions are inconsistent across libraries and interviewers test whether you check rather than assume: HF attention_mask uses 1 = attend; nn.MultiheadAttention's key_padding_mask uses True = ignore; SDPA's boolean attn_mask uses True = attend. State the convention out loud before writing the mask.
• A padded position that attends to nothing still emits a (garbage) output vector — harmless if downstream losses/poolers ignore pads, catastrophic if they don't. Trace where pad positions flow after attention.
• Follow-up you should expect: "why do we scale by 1/√d?" — keep score variance ~1 so softmax stays in its high-gradient regime at init.

Problem 14 — The language model with near-zero training loss and gibberish generations (causal LM labels)

Category: LLM training / label pipeline Difficulty: Medium Frequency tier: Tier 2 overall; Tier 1 for LLM roles — the label-shift question appears in some form in nearly every LLM loop

Buggy code

def compute_lm_loss(model, input_ids, pad_id):
    logits = model(input_ids)                        # (B, L, V)
    loss = F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        input_ids.view(-1),                          # BUG 1: labels not shifted
    )                                                # BUG 2: pad tokens included in loss
    return loss

# elsewhere, a teammate "fixed" it for a HuggingFace model:
outputs = model(input_ids=input_ids[:, :-1],
                labels=input_ids[:, 1:])             # BUG 3: double shift (HF shifts internally)

Symptoms

• Bug 1: training loss collapses to ~0 within a few hundred steps — suspiciously fast — yet generations are gibberish.
• Bug 2: the model learns to spam the padding token; generations degenerate into pad/EOS repetition.
• Bug 3: loss looks reasonable, but the model learns to predict the token two positions ahead; generations are subtly incoherent, always "skipping" a step.

Diagnostic reasoning

• Near-zero LM loss is never good news. Predicting token t from an input that contains token t is an identity map. Verified with a toy model: loss on unshifted labels reached 0.0014 after 300 steps (random tokens, so the true next-token loss should stay near ln V ≈ 3.9; the same model measured with properly shifted labels scores 11.7 — pure memorization of the copy task). Rule: for causal LMs, labels[i] = input_ids[i+1]; logits at position i are scored against the next token.
• The reflex check: compare training loss to the theoretical floor. Random 50-token vocab data cannot be predicted below ln 50 — a loss far below the data's entropy means the model has access to the answer.
• Padding in the loss: every pad position contributes a trivially learnable target, deflating the loss and teaching the model that pad/EOS is the most likely token everywhere. Use ignore_index=-100 (the HF convention) and set label positions for pads (and any non-target text) to -100.
• Double shift: HuggingFace ...ForCausalLM models shift internally when you pass labels. Pre-shifting yourself composes two shifts. The tell: inspect one example end-to-end — print tokenizer.decode of the input position and the label it's scored against. A 5-line sanity print beats an hour of loss-curve staring.

Fix

def compute_lm_loss(model, input_ids, pad_id):
    logits = model(input_ids)                        # (B, L, V)
    labels = input_ids.clone()
    labels[labels == pad_id] = -100                  # exclude pads from loss
    shift_logits = logits[:, :-1, :]                 # predict positions 1..L-1
    shift_labels = labels[:, 1:]
    return F.cross_entropy(shift_logits.reshape(-1, logits.size(-1)),
                           shift_labels.reshape(-1),
                           ignore_index=-100)

# HuggingFace: pass UNshifted labels; the model shifts for you
outputs = model(input_ids=input_ids, labels=labels)   # labels = input_ids with -100 on pads

Deeper lesson

• Loss-floor reasoning is a universal LLM debugging tool: loss below the data entropy ⇒ leakage into the input (unshifted labels, document leaks across the context window, eval data in train).
• The same "who shifts?" question recurs everywhere: HF shifts, most nanoGPT-style codebases shift in the dataloader, some kernels (fused CE) shift internally. Establish it once per codebase and write it down.
• Related Tier-2 follow-up: sequence packing without cross-document attention masking — tokens attend across document boundaries, quietly improving loss and hurting downstream quality.

Problem 15 — Great first answer, garbage second answer (KV-cache & decoding)

Category: LLM inference / KV cache Difficulty: Medium-hard Frequency tier: Tier 2 — standard for inference/serving-flavored LLM roles

Buggy code

class ChatServer:
    def __init__(self, model):
        self.model = model
        self.past_kv = None                         # BUG 1: cache is server state

    @torch.inference_mode()
    def generate(self, prompt_ids, max_new=100, temperature=0.0):
        ids = prompt_ids
        for _ in range(max_new):
            input_step = ids[:, -1:] if self.past_kv is not None else ids
            position_ids = torch.arange(input_step.shape[1])[None]   # BUG 2: always [0] for steps
            out = self.model(input_step, past_key_values=self.past_kv,
                             position_ids=position_ids)
            self.past_kv = out.past_key_values
            probs = torch.softmax(out.logits[:, -1] / temperature, -1)  # BUG 3: /0 when temp=0
            next_id = torch.multinomial(probs, 1)
            ids = torch.cat([ids, next_id], dim=1)
        return ids

Symptoms

• The first request generates fine; the second request returns tokens influenced by the first conversation, degenerating over time (Bug 1).
• With Bug 2, generations start coherent then become repetitive/degenerate — worse the longer the output.
• Setting temperature=0.0 (intended as "greedy") returns NaN-driven random tokens (Bug 3: division by zero → inf logits → NaN probs; multinomial then errors or samples garbage).

Diagnostic reasoning

• Request-order-dependent outputs ⇒ hidden state persisting across requests. The KV cache lives on self; request 2 attends over request 1's keys/values. Print past_kv[0][0].shape[2] (cached sequence length) at the start of each request — it should equal 0/None, not the previous conversation's length.
• Position bug: during incremental decoding the new token's position_ids must be past_length + arange(new_len). Computing arange(input_step.shape[1]) gives [0] for every step, so RoPE/positional encoding places every generated token at position 0. The model still "works" (attention over cache is intact) which is what makes it insidious — quality just quietly degrades. Test: generate with and without cache and assert identical outputs (use_cache=False as the reference implementation).
• temperature=0 must be special-cased to argmax, not passed to a division. The clue is the stack trace location (softmax returning NaN) plus "only when temperature is 0."

Fix

@torch.inference_mode()
def generate(self, prompt_ids, max_new=100, temperature=0.0):
    past_kv = None                                   # cache is per-request, not server state
    ids = prompt_ids
    for _ in range(max_new):
        input_step = ids[:, -1:] if past_kv is not None else ids
        past_len = 0 if past_kv is None else past_kv[0][0].shape[2]
        position_ids = (past_len + torch.arange(input_step.shape[1]))[None]
        out = self.model(input_step, past_key_values=past_kv, position_ids=position_ids)
        past_kv = out.past_key_values
        logits = out.logits[:, -1]
        if temperature == 0.0:
            next_id = logits.argmax(-1, keepdim=True)          # greedy
        else:
            next_id = torch.multinomial(torch.softmax(logits / temperature, -1), 1)
        ids = torch.cat([ids, next_id], dim=1)
    return ids

Deeper lesson

• The gold-standard KV-cache test: cached and uncached decoding must produce identical tokens (greedy). Any divergence is a cache/position/mask bug. Same trick validates batched-vs-single and left-padded-vs-unpadded inference.
• Related bugs in this family: attention mask not extended along with the cache; left-padding required for batched generation but dataset right-padded; sampling with repetition_penalty applied to logits after softmax; forgetting eos handling so finished sequences keep generating inside a batch.

Problem 16 — The pre-training run with loss spikes and a useless grad-clip (mixed precision & optimizer hygiene)

Category: Pre-training at scale / AMP / optimizer grouping Difficulty: Hard Frequency tier: Tier 2 — expected for anyone claiming pre-training experience; the AMP-clip ordering is a favorite senior question

Buggy code

scaler = torch.cuda.amp.GradScaler()
optimizer = torch.optim.AdamW(model.parameters(),
                              lr=3e-4, weight_decay=0.1)   # BUG 3: decay on EVERYTHING

for step, batch in enumerate(loader):
    with torch.autocast("cuda", dtype=torch.float16):
        loss = model(**batch).loss
    scaler.scale(loss / ACCUM).backward()

    if (step + 1) % ACCUM == 0:
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # BUG 1: clipping SCALED grads
        scaler.step(optimizer)
        scaler.update()
        optimizer.zero_grad()

    if step % 100 == 0:
        log(loss=loss.item(), grad_norm=grad_norm)   # BUG 2: grad_norm is in scaled units

Symptoms

• Grad-clip "doesn't work": occasional loss spikes and divergences persist even with max_norm=1.0.
• Logged grad norms are enormous (∼10⁴–10⁵) and jump discontinuously when loss spikes occur — but the numbers look meaningless.
• Weights in LayerNorm drift toward zero over long runs; embedding quality degrades late in training.

Diagnostic reasoning

• Clipping scaled gradients: GradScaler multiplies the loss (hence gradients) by a dynamic scale factor (often 2¹⁴–2¹⁶). Clipping to max_norm=1.0 in scaled units is effectively clipping to 1/65536 in true units — or, more commonly, effectively never clipping because the comparison is against the scaled norm. The correct order is scaler.unscale_(optimizer) before clip_grad_norm_, then scaler.step. The logged grad norms being ~the scale factor is the giveaway.
• fp16 vs bf16: fp16 has a tiny dynamic range (max ~65504) — hence the scaler; bf16 has fp32's range and normally needs no GradScaler. If you see GradScaler + bf16 in review, flag it as harmless-but-confused; if you see fp16 with no scaler, flag it as a NaN generator. Loss spikes at scale also implicate data (bad shards), so bisect: replay the exact global step's batch (this is why you checkpoint dataloader state).
• Blanket weight decay: AdamW decay should exclude LayerNorm/RMSNorm weights, all biases, and (commonly) embeddings. Decaying norm gains pulls them toward zero and destabilizes activations late in training. This is a "did you actually read a real pre-training codebase" check.

Fix

decay, no_decay = [], []
for n, p in model.named_parameters():
    if p.ndim < 2 or "norm" in n.lower() or "embed" in n.lower():
        no_decay.append(p)
    else:
        decay.append(p)
optimizer = torch.optim.AdamW(
    [{"params": decay, "weight_decay": 0.1},
     {"params": no_decay, "weight_decay": 0.0}], lr=3e-4)

for step, batch in enumerate(loader):
    with torch.autocast("cuda", dtype=torch.bfloat16):   # bf16: no scaler needed
        loss = model(**batch).loss
    (loss / ACCUM).backward()

    if (step + 1) % ACCUM == 0:
        grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)
        log(loss=loss.item(), grad_norm=grad_norm.item(), lr=...)
# (If fp16 is required: keep GradScaler and call scaler.unscale_(optimizer)
#  immediately before clip_grad_norm_.)

Deeper lesson

• Grad-norm logging is only meaningful in unscaled units; also log it pre-clip — the spike pattern (isolated spikes vs sustained growth) distinguishes bad-data events from genuine instability.
• Other pre-training spike suspects worth naming: LR warmup too short, Adam eps too small for bf16, attention logit growth (fix: QK-norm or logit soft-capping), and a data shard with pathological content (fix: skip-batch + data bisect).
• Gradient accumulation subtlety: loss must be divided by ACCUM (done here) and anything computed per-micro-batch (like logged loss) interpreted accordingly; with DDP, use no_sync() on non-boundary micro-steps to avoid ACCUM× communication.

Problem 17 — The 8-GPU run that's worse than 1 GPU (distributed training)

Category: DDP / distributed data loading Difficulty: Hard Frequency tier: Tier 2 — standard for ML-infra and scale-adjacent roles

Buggy code

sampler = DistributedSampler(dataset)
loader = DataLoader(dataset, batch_size=64, sampler=sampler)
model = DistributedDataParallel(model.cuda(rank), device_ids=[rank])

for epoch in range(EPOCHS):
    # BUG 1: sampler.set_epoch(epoch) never called
    for batch in loader:
        train_step(batch)

    acc = evaluate(model, my_val_shard)          # BUG 2: each rank evaluates its shard...
    if rank == 0:
        print("val acc:", acc)                   # ...but only rank 0's shard is reported
        torch.save(model.module.state_dict(), "ckpt.pt")
    # BUG 3: no barrier — other ranks race ahead into the next epoch / load a half-written file

Symptoms

• The distributed run converges to a worse loss than the single-GPU baseline given the same number of samples seen (Bug 1: every epoch replays the identical shuffle order, so the model sees batches in the same order every epoch — less effective SGD noise, and with augmentation-free data, effectively a repeated curriculum).
• Reported validation accuracy is noisy and disagrees with post-hoc full-dataset evaluation (Bug 2: it's the accuracy of 1/8th of the data, whichever shard rank 0 got).
• Intermittent crashes or silently stale checkpoints when other ranks torch.load the checkpoint file that rank 0 is still writing (Bug 3).

Diagnostic reasoning

• "Distributed is worse" checklist, in order of likelihood: (1) set_epoch missing → same permutation every epoch (the sampler seeds its RNG with seed + epoch; without it, epoch is always 0 — log the first batch's indices at epoch 0 and 1 and diff them); (2) learning rate not adjusted for the 8× effective batch size; (3) loss reduction semantics — DDP averages gradients across ranks, so per-rank mean losses compose correctly, but a per-rank sum loss silently scales gradients by world_size; (4) BatchNorm statistics computed per-rank (use SyncBatchNorm if BN matters).
• Metric aggregation: any metric computed on a DistributedSampler shard must be all-reduced (dist.all_reduce on correct/total counts) before reporting. Also beware: DistributedSampler pads the dataset to divide evenly across ranks — duplicated samples inflate eval metrics; use drop_last thinking or a dedicated (unsharded, rank-0) eval pass.
• Checkpoint discipline: rank-0-writes + dist.barrier() after; or write to temp file + atomic rename.

Fix

for epoch in range(EPOCHS):
    sampler.set_epoch(epoch)                     # different shuffle each epoch
    for batch in loader:
        train_step(batch)

    correct, total = evaluate_counts(model, my_val_shard)
    stats = torch.tensor([correct, total], device=rank)
    dist.all_reduce(stats)                       # aggregate over all ranks
    if rank == 0:
        print("val acc:", (stats[0] / stats[1]).item())
        torch.save(model.module.state_dict(), "ckpt.tmp")
        os.replace("ckpt.tmp", "ckpt.pt")        # atomic
    dist.barrier()                               # everyone waits for the checkpoint

Deeper lesson

• DDP invariant worth stating in interviews: after backward(), gradients are identical on all ranks; parameters must therefore stay identical forever. Any rank-dependent randomness in the model (e.g., unseeded dropout noise applied to weights, rank-dependent data-dependent control flow) breaks this silently — symptoms only appear at checkpoint-load or when find_unused_parameters errors surface.
• Know the classic hang: a conditional branch where some ranks call backward() on parameters others didn't use → NCCL collective waits forever. Debug with TORCH_DISTRIBUTED_DEBUG=DETAIL and find_unused_parameters=True (and know its perf cost).

Problem 18 — The diffusion model whose loss converges but samples are pure noise (parameterization mismatch)

Category: Diffusion models / DDPM training Difficulty: Hard Frequency tier: Tier 2 overall; Tier 1 for GenAI-vision / diffusion roles

Buggy code

# Training (DDPM-style)
def train_step(model, x0, alphas_cumprod):
    t = torch.randint(0, T, (x0.shape[0],), device=x0.device)
    noise = torch.randn_like(x0)
    a = alphas_cumprod[t].view(-1, 1, 1, 1)
    x_t = a.sqrt() * x0 + (1 - a).sqrt() * noise

    pred = model(x_t, t)
    loss = F.mse_loss(pred, x0)          # BUG 1: trains model to predict x0 ...
    return loss

# Sampling (copied from a reference epsilon-prediction implementation)
@torch.no_grad()
def sample(model, shape):
    x = torch.randn(shape)
    for t in reversed(range(T)):
        eps = model(x, torch.full((shape[0],), t))     # ... but sampler treats output as EPSILON
        x = ddpm_step_epsilon_param(x, eps, t)          # BUG 1 (cont.)
    return x

# Latent diffusion wrapper
z = vae.encode(images).latent_dist.sample()             # BUG 2: missing 0.18215 scaling
# ... train diffusion on z ...
images_out = vae.decode(z_sampled)                      # decode also unscaled

Symptoms

• Training loss decreases smoothly and plateaus at a healthy-looking value — the loss gives no warning — yet samples are pure noise or vague color blobs (Bug 1).
• In the latent-diffusion variant, samples are washed-out / low-contrast / desaturated, and the diffusion loss behaves oddly because latents have ~5× the expected standard deviation (Bug 2).

Diagnostic reasoning

• Training loss cannot detect a train/sample contract mismatch. MSE-to-x0 is a perfectly learnable objective; the network happily learns it. The bug lives in the interface: the sampler algebra assumes model(x_t,t) ≈ ε and reconstructs x0 = (x_t − √(1−ᾱ)·ε)/√ᾱ. Feed it an x0-prediction and every step's mean is wrong. This is the diffusion analogue of Problem 3 (loss owns the activation): the sampler owns the parameterization.
• Diagnostic that settles it in minutes: one-step reconstruction test. Take a real image, noise it to a middling t, run the model once, apply the sampler's x0-recovery formula, and look at the result. If the recovered x0 is garbage while mse(pred, target_you_trained_on) is small, the parameterization contract is broken.
• Know the three parameterizations and their conversions: ε-prediction, x0-prediction, and v-prediction (v = √ᾱ·ε − √(1−ᾱ)·x0). Any can train; trainer, sampler, and loss weighting must simply agree. (v-prediction is the modern default for stability at high noise levels.)
• Latent scaling: SD-family VAEs produce latents with std ≈ 1/0.18215; the scale_factor normalizes them to ~unit variance so the diffusion noise schedule matches. Skipping it at encode (or decode) breaks the SNR the schedule assumes. The tell: z.std() far from 1 on a training batch — a one-line check worth doing on any latent pipeline.
• Also verify in review: t passed to the model matches the timestep the noising used (off-by-one between 0-indexed alphas_cumprod and 1-indexed schedule tables is a classic — see Problem 19), and that the model actually receives t (a missing timestep embedding degrades to a blurry average denoiser).

Fix

# Make trainer and sampler agree — epsilon-parameterization end to end:
def train_step(model, x0, alphas_cumprod):
    t = torch.randint(0, T, (x0.shape[0],), device=x0.device)
    noise = torch.randn_like(x0)
    a = alphas_cumprod[t].view(-1, 1, 1, 1)
    x_t = a.sqrt() * x0 + (1 - a).sqrt() * noise
    return F.mse_loss(model(x_t, t), noise)       # predict the NOISE

# Latent diffusion: apply and invert the scale factor symmetrically
z = vae.encode(images).latent_dist.sample() * 0.18215
images_out = vae.decode(z_sampled / 0.18215)

Deeper lesson

• Generative-model debugging inverts the usual signal: the training loss is nearly useless; the samples are the test. Sample early, sample often, and keep a fixed noise seed so sample quality is comparable across checkpoints.
• Standard follow-ups: why ε-prediction loss ≈ uniform weighting in a particular SNR sense; min-SNR loss weighting; why classifier-free guidance requires training with condition dropout (10% null-condition) — forgetting that dropout is its own classic bug: CFG at inference then extrapolates between two conditionals the model never learned to separate.

Problem 19 — The diffusion samples that got worse after "cleanup" (EMA & schedule off-by-one)

Category: Diffusion models / sampling & maintenance Difficulty: Hard Frequency tier: Tier 3 — differentiator; probes whether you've shipped a diffusion model, not just read the paper

Buggy code

# A refactor "simplified" the trainer:
class Trainer:
    def __init__(self, model):
        self.model = model
        self.ema_model = copy.deepcopy(model)

    def step(self, batch):
        loss = diffusion_loss(self.model, batch)
        loss.backward(); self.opt.step(); self.opt.zero_grad()
        # BUG 1: EMA update was deleted in the refactor ("it wasn't referenced anywhere")

    def sample(self):
        return ddim_sample(self.model, ...)        # BUG 2: samples from RAW weights anyway

# DDIM sampler, ported by hand from the paper's 1-indexed math:
for i, t in enumerate(timesteps):                  # e.g. [999, 979, ..., 19]
    a_t    = alphas_cumprod[t]
    a_prev = alphas_cumprod[t - 20]                # BUG 3: assumes uniform spacing; breaks at
    ...                                            # the last step (t-20 = -1 wraps to index 999!)

Symptoms

• Samples are subtly but consistently worse than a colleague's older checkpoint — more high-frequency artifacts, less coherent global structure (Bugs 1–2). FID quietly regressed ~15% and nobody could point to a diff that "touched the model."
• With Bug 3, the final denoising step occasionally adds massive noise: alphas_cumprod[-1] wraps to the most-noised entry, so the last step's target SNR is catastrophically wrong — images look almost right but carry a global haze/noise floor.

Diagnostic reasoning

• EMA is part of the model, not an optimization nicety. Diffusion sample quality depends heavily on exponentially-averaged weights (decay ~0.9999); raw online weights sit in a sharper, noisier region. When sample quality regresses with no model-code diff, ask "what weights are we sampling from?" first. Detection: checksum the sampled-from state_dict against both raw and EMA checkpoints.
• Negative indexing is a silent wrap in PyTorch/NumPy. alphas_cumprod[-1] is a valid read of the wrong element — no exception, just wrong math on exactly one step. The general lesson: any hand-ported index arithmetic from 1-indexed paper math deserves an explicit boundary test (first step, last step). Fix by carrying explicit (t, t_prev) pairs, with a_prev = 1.0 when t_prev < 0 (the fully-denoised boundary).
• Interview-grade habit: plot the schedule you actually execute — t values, ᾱ_t, SNR per sampling step — before debugging the network. Most sampler bugs are visible in that plot (wrapped index, skipped final step, reversed order, trained-vs-sampled schedule mismatch).

Fix

def step(self, batch):
    loss = diffusion_loss(self.model, batch)
    loss.backward(); self.opt.step(); self.opt.zero_grad()
    with torch.no_grad():                          # EMA update, every step
        for p_ema, p in zip(self.ema_model.parameters(), self.model.parameters()):
            p_ema.lerp_(p, 1 - 0.9999)

def sample(self):
    return ddim_sample(self.ema_model, ...)        # ALWAYS sample from EMA

# Sampler: explicit prev-timestep pairs, safe boundary
timesteps = list(reversed(range(0, 1000, 20)))
for t, t_prev in zip(timesteps, timesteps[1:] + [-1]):
    a_t    = alphas_cumprod[t]
    a_prev = alphas_cumprod[t_prev] if t_prev >= 0 else torch.tensor(1.0)
    ...

Deeper lesson

• "No diff touched the model" is never true — EMA updates, weight tying, buffer registration, and checkpoint-loading strict=False are all model-affecting code that doesn't look like model code. strict=False deserves special fear: it silently skips mismatched keys, so a renamed module loads random init without error.
• Related sampler-family bugs to name-drop: guidance scale applied to the wrong quantity (ε vs x0 space), eta misuse in DDIM, float32 schedule computed in float16 (ᾱ underflow at high t), and mismatch between training noise schedule (e.g., cosine) and the sampler's hardcoded linear betas.

Problem 20 — The flow-matching model that trains beautifully and samples nothing (sign & time conventions)

Category: Flow matching / continuous-time generative models Difficulty: Hard Frequency tier: Tier 3 — differentiator for research-engineer and frontier-lab loops (increasingly common since 2024–25)

Buggy code

# Training — conditional flow matching with linear interpolation path
def fm_loss(model, x1):                    # x1: data
    x0 = torch.randn_like(x1)              # noise
    t = torch.rand(x1.shape[0], 1)
    x_t = (1 - t) * x0 + t * x1            # t=0 noise -> t=1 data
    v_target = x0 - x1                     # BUG 1: sign flipped (should be x1 - x0)
    return F.mse_loss(model(x_t, t), v_target)

# Sampling — Euler integration
@torch.no_grad()
def sample(model, n, steps=100):
    x = torch.randn(n, D)
    dt = 1.0 / steps
    for i in range(steps):
        t = torch.full((n, 1), 1.0 - i * dt)   # BUG 2: time runs 1 -> 0, but the
        x = x + model(x, t) * dt               # path was trained with t=0 at NOISE
    return x

Symptoms

• Training loss converges smoothly to a sensible plateau — the target x0 − x1 is a perfectly well-defined regression target, so the loss cannot reveal the bug.
• Samples never reach the data distribution. Verified on a 1-D bimodal toy (modes at ±2): correct sign/time puts 98% of samples near the modes; the sign-flipped model leaves 3% near the modes with samples statistically indistinguishable from the starting noise (the learned field pushes away from data, and the two bugs partially "cancel" into a field that transports nowhere useful).

Diagnostic reasoning

• Flow matching has exactly three conventions that must agree, and each is a coin-flip across papers/codebases: (1) does t=0 mean noise or data? (2) is the velocity target x1 − x0 (pointing toward data under t=0-noise) or the reverse? (3) does the sampler integrate t from 0→1 or 1→0, with x += v·dt matching that direction? Any consistent triple works; any mismatch fails silently. (Diffusion papers use t=0 = data, which is exactly why ported code mixes conventions.)
• The decisive test costs 20 lines: run the pipeline on a 1-D or 2-D toy dataset (two Gaussians, checkerboard) where you can see the samples. This localizes convention bugs in minutes, versus days on images. Interviewers specifically reward "I'd shrink the problem to a visualizable toy."
• Second decisive test: integrate the ground-truth field. For linear paths the true velocity is known in closed form (x1 − x0 for paired endpoints); plugging the analytic field into your sampler validates the sampler independently of the network. Separate "trainer bug" from "sampler bug" — the two-component contract is the same lesson as Problem 18.
• Sanity checks worth narrating: at t≈data-end, samples should already look like data; the norm of the learned velocity should ≈ E‖x1 − x0‖ (≈ √(2D/π)-ish for Gaussian pairs), not ~0 (a ~0 field means targets averaged out — e.g., unpaired/reshuffled (x0, x1) within the batch, another classic).

Fix

def fm_loss(model, x1):
    x0 = torch.randn_like(x1)
    t = torch.rand(x1.shape[0], 1)
    x_t = (1 - t) * x0 + t * x1
    v_target = x1 - x0                      # velocity points from noise toward data
    return F.mse_loss(model(x_t, t), v_target)

@torch.no_grad()
def sample(model, n, steps=100):
    x = torch.randn(n, D)                   # start at t=0 (noise) ...
    dt = 1.0 / steps
    for i in range(steps):
        t = torch.full((n, 1), i * dt)      # ... integrate FORWARD to t=1 (data)
        x = x + model(x, t) * dt
    return x

Deeper lesson

• Write the convention as a docstring contract at the top of the module (t=0: N(0,I); t=1: data; v = dx/dt = x1 − x0; sampler: Euler 0→1) and add a unit test that trains 30 seconds on two Gaussians and asserts sample means. Convention bugs recur every time someone ports code.
• Expect follow-ups: why the conditional FM objective (regressing per-pair velocities) has the same minimizer as the intractable marginal objective; independent coupling vs minibatch-OT coupling (straighter paths, fewer sampling steps); relationship to rectified flow and to v-prediction diffusion (linear-path FM is closely related — good candidates connect them).

Problem 21 — The fine-tuned assistant that never stops talking (SFT data pipeline)

Category: Post-training / supervised fine-tuning Difficulty: Medium-hard Frequency tier: Tier 2 overall; Tier 1 for post-training / applied-LLM roles

Buggy code

def build_example(tokenizer, prompt, response):
    text = prompt + response                    # BUG 3: no chat template, no separator
    ids = tokenizer(text, truncation=True, max_length=1024)["input_ids"]
    # BUG 2: no EOS appended
    labels = list(ids)                          # BUG 1: loss computed on PROMPT tokens too
    return {"input_ids": ids, "labels": labels}

# inference (deployed):
out = model.generate(tokenizer("User: " + q, return_tensors="pt").input_ids,
                     max_new_tokens=512)        # BUG 3 (cont.): different format than training

Symptoms

• The model never emits EOS — every generation runs to max_new_tokens and trails off mid-sentence (Bug 2).
• The model often responds by asking another user-style question or continuing the prompt rather than answering — it has partially learned to imitate prompts (Bug 1).
• Offline eval loss looks great, but deployed quality is far worse than the eval suggested (Bug 1 deflates loss with easy prompt tokens; Bug 3 means the deployed input distribution doesn't match training at all).

Diagnostic reasoning

• Prompt-loss masking: SFT should optimize P(response | prompt), so label positions covering the prompt must be −100. Training on prompt tokens (a) wastes capacity teaching the model to mimic users, and (b) corrupts loss-based evals (prompt tokens are easy → loss looks better than the model is). Detection: decode one example's (input_id, label) pairs and read them — the fastest, most underused post-training debug step.
• EOS: a model that never saw <eos> after responses assigns it near-zero probability forever. Symptom is unmistakable (always hits the token cap). Also check the opposite failure: padding with eos without masking pad-labels teaches the model to spam EOS immediately — the same underlying issue as Problem 14's Bug 2.
• Template train/serve skew: exact string format is part of the model's input distribution — role markers, whitespace, newlines, BOS handling. apply_chat_template at train but raw concatenation at serve (or vice versa) is a distribution shift you inflicted on yourself. The classic subtle version: tokenizing prompt and response separately and concatenating token IDs produces different tokens than tokenizing the joined string (tokenizer merges across the boundary); mask offsets computed from the separate tokenization then misalign by a token or two.
• Bonus check for interviews: truncation must never cut the response to fit max_length while keeping the whole prompt (truncate the prompt from the left instead) — otherwise you train on amputated targets.

Fix

def build_example(tokenizer, messages):         # messages: [{role, content}, ...]
    prompt_ids = tokenizer.apply_chat_template(
        messages[:-1], add_generation_prompt=True)
    full_ids = tokenizer.apply_chat_template(messages)   # includes response + eos per template
    labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]
    assert full_ids[:len(prompt_ids)] == prompt_ids      # guard against boundary re-tokenization
    return {"input_ids": full_ids, "labels": labels}

# inference uses the SAME template:
ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": q}], add_generation_prompt=True, return_tensors="pt")
out = model.generate(ids, max_new_tokens=512,
                     eos_token_id=tokenizer.eos_token_id)

Deeper lesson

• Post-training debugging rule: read your actual tokenized examples, decoded, with label masks visualized, before trusting any loss curve. Nearly every SFT bug is visible in one printed example.
• Know the adjacent family: pad_token = eos_token (common default) silently masking real EOS from the loss when pads are set to −100 by matching on token id — mask by position, not by id. Multi-turn masking (train on all assistant turns, mask user turns) is the standard follow-up.

Problem 22 — The DPO run where reward margins soar and the model gets worse (post-training / preference optimization)

Category: Post-training / DPO & RLHF Difficulty: Hard Frequency tier: Tier 3 — differentiator for alignment / post-training specialist roles

Buggy code

ref_model = model                               # BUG 1: "reference" IS the policy (no copy/freeze)

def dpo_loss(model, ref_model, batch, beta=0.1):
    # log-probs summed over ALL positions, including prompt & padding
    pi_w  = model(batch.chosen).logits.log_softmax(-1)
    pi_l  = model(batch.rejected).logits.log_softmax(-1)
    logp_w  = gather_token_logps(pi_w,  batch.chosen).sum(-1)     # BUG 2: prompt+pad included
    logp_l  = gather_token_logps(pi_l,  batch.rejected).sum(-1)

    with torch.no_grad():
        ref_w = sum_logps(ref_model, batch.chosen)
        ref_l = sum_logps(ref_model, batch.rejected)

    margin = beta * ((logp_w - ref_w) - (logp_l - ref_l))
    return -F.logsigmoid(margin).mean()
# Logged: reward margins climb beautifully; accuracy of margin > 0 hits 95%+.
# Actual generations: shorter-lived coherence, repetition, then degeneration.

Symptoms

• All DPO dashboard metrics improve (margins ↑, implicit-reward accuracy ↑, loss ↓) while side-by-side human/LLM-judge evals show the model getting worse — the signature divergence of optimizing a proxy.
• With Bug 1, the "KL leash" is gone: since ref == policy, the reference log-probs track the policy exactly (the no_grad only stops gradients, not weight sharing), so implicit rewards are computed against a moving target and the loss degenerates toward pushing logp_w − logp_l apart without any anchor — unbounded drift from the initial model.
• With Bug 2, sequences win by being long: summed log-probs over prompt+padding make length and prompt-likelihood confounders of "preference." The model learns verbosity, not quality.

Diagnostic reasoning

• First question for any preference-optimization pathology: "what exactly anchors the policy to its initialization, and is it actually frozen?" Check id(ref_model) != id(model), all(not p.requires_grad for p in ref_model.parameters()), and — the subtle one — that no optimizer param group or weight tying reaches the reference copy. deepcopy + .eval() + requires_grad_(False).
• Second: what tokens are inside the log-prob sum? DPO's implicit reward should cover response tokens only (mask prompt positions and padding with the same −100 discipline as SFT). Diagnostic: correlate margin with response-length difference across the eval set — high correlation ⇒ length confound. (Length-normalized variants exist precisely because even correct masking leaves some length bias.)
• Third — the conceptual layer interviewers push on: margins measure relative log-prob movement, and DPO can achieve them by decreasing the chosen response's absolute log-prob (just decreasing the rejected one faster). Monitor absolute logp_w on held-out data: if it falls persistently, the model is unlearning fluent text while "winning" the margin game. Always pair proxy metrics with actual generation evals.
• The RLHF-side sibling bugs to name: KL penalty sign/coefficient errors (policy pushed away from ref → fluent-looking reward-hacked gibberish), reward model queried on a different template than the policy generates, and advantage normalization across a batch containing multiple prompts of very different difficulty.

Fix

ref_model = copy.deepcopy(model).eval().requires_grad_(False)   # true frozen snapshot

def masked_response_logps(model, ids, response_mask):
    logits = model(ids).logits[:, :-1]
    logps = logits.log_softmax(-1).gather(-1, ids[:, 1:, None]).squeeze(-1)
    return (logps * response_mask[:, 1:]).sum(-1)   # response tokens only, pads/prompt excluded

# Monitor, alongside margins:
#   - absolute chosen logp on held-out data (should not collapse)
#   - KL(policy || ref) on sampled generations
#   - periodic generation evals vs a fixed judge — the only metric that counts

Deeper lesson

• Post-training is where Goodhart's law becomes a debugging category: the loss can be implemented perfectly and the run still "fails" because the proxy diverged from quality. Strong candidates volunteer the monitoring suite (absolute log-probs, KL, length distributions, judge win-rates), not just the code fix.
• This problem generalizes Problem 12's lesson (metric ≠ objective) to the frontier setting — connecting those two explicitly is a strong interview move.

Part III — More classic ML & PyTorch (Problems 23–34)

Problem 23 — The cleaning step that never happened (pandas chained assignment)

Category: pandas semantics Difficulty: Easy-medium Frequency tier: Tier 1 — core screener (any role touching pandas; a favorite code-review question)

Buggy code

df = pd.read_csv("train.csv")
adults = df[df.age >= 18]
adults["income"] = adults["income"].fillna(adults["income"].median())  # SettingWithCopyWarning

df[df.label == "spam"]["weight"] = 2.0        # chained assignment: silently does NOTHING

train_df = df                                  # not a copy!
train_df.drop(columns=["id"], inplace=True)    # also mutates df for every other consumer

Symptom

NaNs survive into training despite the "fillna"; the spam weights are all still 1.0; a completely different module crashes because id vanished from df. Verified: after sub['b'] = 0 on a filtered frame, the original is untouched.

Diagnostic reasoning

• df[mask] may return a copy; assigning into it modifies the copy, which is immediately discarded. df[mask]["col"] = x is two operations — __getitem__ then __setitem__ — and the warning tells you pandas can't guarantee which object you wrote to.
• The reflex check when a transformation "doesn't take": immediately after the line, assert the effect (assert df["income"].isna().sum() == 0). Cheap assertions after each cleaning step convert silent no-ops into loud failures.
• The reverse hazard is aliasing: train_df = df binds a second name to the same object, so inplace=True mutations propagate to every holder of the reference.

Fix

adults = df[df.age >= 18].copy()               # explicit copy when you want a copy
adults["income"] = adults["income"].fillna(adults["income"].median())

df.loc[df.label == "spam", "weight"] = 2.0     # single .loc — one operation, guaranteed write

train_df = df.copy()                            # explicit copy when consumers must not interact
train_df = train_df.drop(columns=["id"])        # prefer non-inplace, reassign

Deeper lesson

• pandas ≥ 2.x with Copy-on-Write (pd.options.mode.copy_on_write = True, default in pandas 3) makes chained assignment consistently a no-op rather than sometimes-working — know both behaviors, because production codebases straddle versions.
• Style rule that prevents the whole class: every write goes through a single .loc[rows, cols] = value; every subset that will be mutated gets an explicit .copy().

Problem 24 — The pretrained model that's blind (vision preprocessing skew)

Category: Computer vision / preprocessing Difficulty: Medium Frequency tier: Tier 2 — role-standard for vision roles; the train/serve skew variant appears in ML-systems interviews

Buggy code

import cv2, numpy as np, torch

def load_image(path):
    img = cv2.imread(path)                     # BUG 1: OpenCV loads BGR, model expects RGB
    img = img + 30                              # BUG 2: brightness "augmentation" on uint8 wraps
    img = torch.from_numpy(img).permute(2, 0, 1)
    return img.float()                          # BUG 3: [0,255] floats fed to a model trained
                                                #        on [0,1] then ImageNet-normalized

Symptom

A pretrained ImageNet backbone that should get ~76% top-1 scores ~30%; fine-tuning "fixes" it (the network relearns around the skew), which hides the bug until someone evaluates the frozen backbone. The brightness op produces bizarre speckled artifacts on bright regions.

Diagnostic reasoning

• Pretrained-model accuracy far below the published number with no training involved is always preprocessing skew. Diff your pipeline against the model card line by line: color order, value range, normalization constants, resize interpolation, center-crop.
• Bug 2 verified: uint8 200 + 100 = 44 — modular wraparound, not saturation. Any arithmetic on uint8 images must be lifted to float (or use cv2.add, which saturates). The speckle pattern — bright pixels turning dark — is the fingerprint.
• The decisive 2-minute test: run one known image (a dog photo) through the exact deployed pipeline and check the top-5 predictions. Interviewers reward concrete "golden input" tests over staring at code.
• Channel-order check: swap img[..., ::-1] and see if accuracy jumps. BGR-vs-RGB costs a characteristic ~10–20 points on ImageNet models — degraded but not destroyed, which makes it easy to ship.

Fix

def load_image(path):
    img = cv2.imread(path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)                 # fix color order
    img = img.astype(np.float32)                                # lift to float BEFORE math
    img = np.clip(img + 30, 0, 255)                             # saturating brightness
    img = img / 255.0
    img = (img - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225] # ImageNet stats
    return torch.from_numpy(img).permute(2, 0, 1).float()

Deeper lesson

• Preprocessing is part of the model. Serialize it with the model (torchvision transforms in the checkpoint, or a processing config), and add a unit test asserting the pipeline's output stats (mean ≈ 0, std ≈ 1 post-normalization) on a fixture image.
• Same family: PIL vs OpenCV resize interpolation differences, JPEG decode differences across libraries, EXIF rotation ignored, and normalizing with the wrong dataset's mean/std after switching backbones (e.g., CLIP's stats differ from ImageNet's).

Problem 25 — The churn model that flags your best customers (predict_proba column order)

Category: sklearn API semantics Difficulty: Easy-medium Frequency tier: Tier 2 — role-standard; quick screener in applied-ML and DS loops

Buggy code

clf = LogisticRegression().fit(X_train, y_train)   # y is strings: "churn" / "stay"

churn_prob = clf.predict_proba(X_val)[:, 0]        # BUG: assumes column 0 = "churn"
top_risk = X_val[np.argsort(-churn_prob)[:1000]]   # send retention offers to these users
print("AUC:", roc_auc_score(y_val == "churn", churn_prob))   # 0.18 (!!)

Symptom

AUC of 0.18 — far below 0.5. The campaign targets exactly the wrong users.

Diagnostic reasoning

• AUC substantially below 0.5 is a label/score inversion, not a bad model. A truly uninformative model gives 0.5; 0.18 means the score is strongly informative but flipped — the very first thing to check is which column of predict_proba you grabbed.
• predict_proba columns follow clf.classes_, which sklearn sorts (alphabetically for strings, numerically for ints). Verified: classes ['neg','pos'] → column 1 is 'pos'. Here classes_ = ["churn", "stay"] — alphabetical — so column 0 is churn… but had the labels been "active"/"churn", it would silently flip. Never assume; index by lookup.
• The generalized lesson: any code containing [:, 1] on predict_proba output is only correct for a specific label encoding. It breaks when labels change from {0,1} to strings, or when a binary problem becomes multiclass.

Fix

churn_col = list(clf.classes_).index("churn")
churn_prob = clf.predict_proba(X_val)[:, churn_col]

# CI guard:
assert set(clf.classes_) == {"churn", "stay"}

Deeper lesson

• Sibling bugs: roc_auc_score(y, clf.predict(X)) (hard labels — Problem 12), pos_label defaults in precision_score when labels aren't {0,1}, and LabelEncoder fit on train mapping differently than on serving data (Problem 34).
• AUC ≈ 1 − true AUC, accuracy ≈ 1 − expected, correlations of the wrong sign: "anti-performance" always means an inversion somewhere — labels, scores, or a comparison operator.

Problem 26 — The oversampled validation set (resampling leakage)

Category: Imbalanced learning / evaluation methodology Difficulty: Medium Frequency tier: Tier 2 — role-standard; standard follow-up to any imbalance discussion

Buggy code

from imblearn.over_sampling import SMOTE

X_res, y_res = SMOTE().fit_resample(X, y)          # BUG: resample BEFORE split
X_train, X_val, y_train, y_val = train_test_split(X_res, y_res, test_size=0.2)

clf = RandomForestClassifier().fit(X_train, y_train)
print("val F1:", f1_score(y_val, clf.predict(X_val)))   # 0.93 — "solved it!"
# production F1: 0.31

Symptom

Validation F1 of 0.93 on a problem where production delivers 0.31.

Diagnostic reasoning

• Two distinct leaks compound. First, SMOTE synthesizes minority samples by interpolating between real minority neighbors — do it before splitting and synthetic points built from validation rows land in train (and vice versa): the model has effectively seen blurred copies of the validation set.
• Second, even without interpolation leakage (plain random oversampling), duplicating minority rows before splitting puts exact copies of validation rows in train.
• Third — independent of leakage — the validation set is now 50/50 balanced, so its F1 answers a question about a distribution that doesn't exist in production. Evaluation must happen on the true prevalence.
• Interview framing: "resampling is a training-time trick; evaluation data must be untouched in both membership and distribution."

Fix

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

X_train_res, y_train_res = SMOTE().fit_resample(X_train, y_train)  # train only
clf = RandomForestClassifier().fit(X_train_res, y_train_res)
print("val F1:", f1_score(y_val, clf.predict(X_val)))   # honest number, true prevalence

# with cross-validation, use imblearn's Pipeline so resampling happens inside each fold:
from imblearn.pipeline import Pipeline
pipe = Pipeline([("smote", SMOTE()), ("clf", RandomForestClassifier())])

Deeper lesson

• The general form of Problem 2's rule: split first, then everything — scaling, resampling, feature selection, augmentation — happens inside the training fold only. imblearn.pipeline.Pipeline exists because sklearn's own Pipeline can't express "fit-time-only" resampling.
• Also question whether resampling is needed at all: class weights or threshold-moving often match SMOTE without synthetic-sample pathologies (SMOTE in high dimensions interpolates through empty space).

Problem 27 — The trading signal that knew tomorrow's price (time-series lookahead)

Category: Time series / feature engineering Difficulty: Medium Frequency tier: Tier 2 — role-standard; Tier 1 in quant/forecasting loops

Buggy code

df["ma_5"] = df["price"].rolling(5, center=True).mean()   # BUG 1: window includes FUTURE rows
df["ret"]  = df["price"].pct_change()
df["target"] = df["ret"].shift(1)          # BUG 2: shift(+1) puts YESTERDAY's return as target
                                            # (author intended tomorrow's: shift(-1))
df = df.dropna()
df["vol_norm"] = df["volume"] / df["volume"].mean()       # BUG 3: full-series mean (future info)

X_train, X_test = train_test_split(df, test_size=0.2, shuffle=True)   # BUG 4: random split

Symptom

Backtest Sharpe ratio is spectacular; live trading loses money immediately. Alternatively (Bug 2 as written): the model achieves near-perfect "prediction" because predicting yesterday's return from today's features is trivial.

Diagnostic reasoning

• Any time a temporal model looks amazing, hunt for lookahead. Four channels here: (1) center=True windows average future prices into "current" features; (2) shift direction — shift(1) moves data forward (past→present), shift(-1) pulls the future; writing the intended alignment as a comment and testing it on a 5-row toy frame settles it in a minute; (3) normalizing by full-series statistics injects the future's scale; (4) random splitting trains on the future to predict the past.
• The decisive audit: pick one row, list every feature, and ask "was this value knowable at this row's timestamp?" — a knowability audit. Interviewers care that you have a procedure, not that you spot all four instantly.
• Detection signal in the wild: performance that degrades as you move the evaluation window further from training data, or a feature-importance ranking dominated by one suspiciously perfect feature.

Fix

df["ma_5"] = df["price"].rolling(5).mean()                # trailing window only
df["target"] = df["ret"].shift(-1)                        # tomorrow's return
train = df.iloc[: int(0.8 * len(df))]                     # temporal split
test  = df.iloc[int(0.8 * len(df)) :]
train_vol_mean = train["volume"].mean()                   # stats from train era only
train["vol_norm"] = train["volume"] / train_vol_mean
test["vol_norm"]  = test["volume"] / train_vol_mean       # frozen at train value

# for CV: sklearn TimeSeriesSplit (expanding-window), optionally with a purge gap

Deeper lesson

• Add a purge/embargo gap between train and test when features use windows (a 5-day feature straddles the boundary otherwise) — this is standard in financial ML for exactly this reason.
• The subtlest variant: target leakage through labels computed with future revisions (restated economic data, delayed outcome labels). Ask when the label became known, not just when the event occurred.

Problem 28 — Early stopping that stopped nothing (checkpoint & selection bugs)

Category: Training orchestration / model selection Difficulty: Medium Frequency tier: Tier 2 — role-standard; nearly every "review this training script" exercise hides one of these

Buggy code

best_loss, patience, bad_epochs = float("inf"), 5, 0
for epoch in range(100):
    train_loss = train_one_epoch(model, train_loader)
    val_loss   = validate(model, val_loader)

    if train_loss < best_loss:                 # BUG 1: monitoring TRAIN loss
        best_loss = train_loss
        torch.save(model.state_dict(), "best.pt")
        bad_epochs = 0
    else:
        bad_epochs += 1
    if bad_epochs >= patience:
        break

test_acc = evaluate(model, test_loader)        # BUG 2: evaluates LAST model, never reloads best
# BUG 3 (silent): best.pt is chosen on val_loss in a fixed-seed run —
# the same val set later reused to pick lr, patience, architecture, ...

Symptom

Early stopping essentially never triggers (train loss almost always improves), so the model overfits for 100 epochs; then the final overfit weights — not the best checkpoint — get evaluated and shipped. Reported test accuracy is noticeably below the best val epoch, and nobody can reproduce "the good checkpoint."

Diagnostic reasoning

• Early stopping's entire purpose is to detect generalization degradation; train loss monotonically improves under sufficient capacity, so monitoring it disables the mechanism. The tell: bad_epochs never increments in the logs.
• Bug 2 is the more damaging and more common one: the loop saves the best model but evaluates whatever the last epoch left in memory. Diff test-time metrics against the best-epoch val metrics — a gap larger than noise means you're not evaluating what you think.
• Bug 3 is the methodological layer: every decision made by peeking at val (stopping epoch, LR, architecture) is a fit to val. With enough decisions, val becomes train. Report final numbers on a test set touched exactly once.
• Bonus review point: val_loss vs val_metric — for imbalanced tasks, loss and task metric can disagree about the best epoch; monitor the metric you'll be judged on.

Fix

best_val, bad_epochs = float("inf"), 0
for epoch in range(100):
    train_one_epoch(model, train_loader)
    val_loss = validate(model, val_loader)
    if val_loss < best_val - MIN_DELTA:        # monitor VAL, with a minimum improvement
        best_val, bad_epochs = val_loss, 0
        torch.save(model.state_dict(), "best.pt")
    else:
        bad_epochs += 1
        if bad_epochs >= PATIENCE:
            break

model.load_state_dict(torch.load("best.pt"))   # RELOAD the winner
test_acc = evaluate(model, test_loader)        # touch test once

Deeper lesson

• min_delta matters: without it, noise-level improvements reset patience forever on plateaued runs.
• Related orchestration bugs to name: LR scheduler continuing to step during patience epochs (so the reloaded best model meets a mismatched LR), and saving state_dict without the epoch/optimizer needed to resume (Problem 11's lesson).

Problem 29 — The loss that changed meaning with the batch size (reduction semantics)

Category: Loss reduction / optimization coupling Difficulty: Medium Frequency tier: Tier 2 — role-standard; probes whether you understand what the gradient actually is

Buggy code

criterion = nn.CrossEntropyLoss(reduction="sum")     # BUG 1: sum, tuned lr assumes mean

# later, someone adds variable-length sequence support:
loss = 0
for i in range(B):
    loss += F.cross_entropy(logits[i, :lens[i]], targets[i, :lens[i]], reduction="mean")
loss = loss / B          # BUG 2: mean-of-means — short sequences weighted same as long ones

# and a "weighted loss" for imbalance:
w = torch.tensor([1.0, 50.0])
loss = F.cross_entropy(logits, y, weight=w)           # OK — but the team ALSO oversampled 50x
                                                      # BUG 3: imbalance corrected twice

Symptoms

• Bug 1: training that was stable at batch size 32 diverges at 256 with the "same" hyperparameters (gradient magnitude scaled 8×); works again when LR is divided by 8 — nobody knows why.
• Bug 2: per-token perplexity on long documents is worse than expected; the model underweights exactly the sequences with the most signal.
• Bug 3: the model wildly over-predicts the minority class; predicted prevalence ≈ 50× the true rate.

Diagnostic reasoning

• reduction="sum" makes the gradient proportional to batch size, so LR and batch size become coupled: every batch-size change silently retunes the effective LR. mean decouples them. Neither is wrong, but the choice must be deliberate and consistent with LR tuning. Fingerprint: hyperparameters that mysteriously stop transferring across batch sizes.
• Bug 2 is the token-weighting question that recurs throughout LLM training: mean-of-per-sequence-means weights each sequence equally; a global token-level mean weights each token equally. For language modeling the standard is token-level: sum the per-token losses and divide by total non-pad tokens. (This same choice silently differs across gradient-accumulation implementations — micro-batch means averaged ≠ global token mean when micro-batches have different token counts.)
• Bug 3: class weights, oversampling, and threshold-moving are three levers for the same correction — stacking them overcorrects. Predicted prevalence vs true prevalence is the 1-line diagnostic.

Fix

# 1: pick mean (decoupled), tune lr once
criterion = nn.CrossEntropyLoss(reduction="mean")

# 2: token-level mean over the whole batch
losses = [F.cross_entropy(logits[i, :lens[i]], targets[i, :lens[i]], reduction="sum")
          for i in range(B)]
loss = torch.stack(losses).sum() / lens.sum()

# 3: choose ONE imbalance lever; verify with predicted-prevalence check

Deeper lesson

• "What, exactly, is this gradient the gradient of?" is the interview question behind all three bugs. Being able to state the objective as a weighted sum over examples/tokens — and defend the weights — is the skill being probed.
• Linear-scaling rule connection: when you intentionally grow batch size, LR typically scales up proportionally (with warmup) — the opposite direction of Bug 1's accidental coupling, and a good place to show you know both.

Problem 30 — Softmax over the batch (wrong-dimension reductions)

Category: Tensor semantics Difficulty: Easy — but under time pressure, misses happen Frequency tier: Tier 1 — core screener; the fastest bug-spotting question in the deck

Buggy code

logits = model(x)                       # (B, C)
probs = torch.softmax(logits, dim=0)    # BUG 1: normalizes over the BATCH
preds = logits.argmax(dim=0)            # BUG 2: argmax over batch -> shape (C,), not (B,)
conf  = probs.max(dim=1).values         # "confidences" from batch-normalized junk

# seq2seq variant:
attn = scores.softmax(dim=1)            # (B, L_q, L_k): normalized over QUERIES not keys

Symptom

Verified: with dim=0, per-row probabilities sum to ~2.16 (anything but 1). Confidences depend on which other samples are in the batch; accuracy computations broadcast oddly (shape (C,) vs labels (B,)) or crash downstream; batch size 1 makes every probability exactly 1.0 — a great smoke test.

Diagnostic reasoning

• Softmax's dim must be the class/key dimension — the one that competition is over. For (B, C) logits: dim=1 (or -1, which is robust to adding sequence dims). For attention (B, L_q, L_k): normalize over keys, dim=-1.
• Three instant checks: (1) probs.sum(dim=-1) must be all-ones; (2) predictions for one sample must not change when you change its batchmates; (3) run batch size 1 — degenerate outputs expose batch-dimension reductions immediately.
• Bug 2's shape error usually surfaces as a broadcast rather than a crash (Problem 7's disease): (C,) == (B,) may broadcast if C == B — which is why test batches should never have size equal to the number of classes.

Fix

probs = torch.softmax(logits, dim=-1)
preds = logits.argmax(dim=-1)
attn  = scores.softmax(dim=-1)
assert torch.allclose(probs.sum(-1), torch.ones(probs.shape[0]))

Deeper lesson

• Prefer dim=-1 for class/key reductions — it survives shape evolution (adding a sequence dimension turns (B,C) into (B,L,C) and dim=1 silently becomes wrong).
• Same family: mean(dim=...) over the wrong axis in masked pooling, norm(dim=...) in cosine similarity, max vs amax return-type confusion. The habit that prevents all of them: comment expected shapes and reduce by named intent, or use einops.reduce which encodes the axis semantics in a string.

Problem 31 — The reshape that scrambled every image (view vs reshape vs permute)

Category: Tensor memory layout Difficulty: Medium Frequency tier: Tier 2 — role-standard PyTorch depth check

Buggy code

x = torch.randn(B, 3, 32, 32)              # NCHW
x_hwc = x.view(B, 32, 32, 3)               # BUG 1: view REINTERPRETS memory, does not permute

feat = feat.transpose(1, 2)                # (B, L, D) -> (B, D, L)
flat = feat.view(B, -1)                    # BUG 2: RuntimeError — non-contiguous

# "fixed" by a teammate:
flat = feat.reshape(B, -1)                 # BUG 3: silences the error; is the ORDER right?

Symptoms

• Bug 1: no error; images render as three stacked grayscale bands with hue stripes — every pixel's channels reassigned. A model trained on this converges (CNNs are robust) but a few points low; nobody suspects the input.
• Bug 2 verified: view size is not compatible with input tensor's size and stride — the loud, helpful version.
• Bug 3: no error, but whether elements are ordered (D,L) or (L,D) in the flattened vector depends on the transpose — if a downstream Linear was trained expecting the other order, quality silently degrades.

Diagnostic reasoning

• Mental model interviewers probe: a tensor is a flat buffer + shape/stride metadata. view changes only metadata over the same element order; permute/transpose change stride so logical order ≠ memory order; reshape = "view if possible, else copy into a fresh contiguous buffer honoring the logical order."
• Therefore: to reorder dimensions, permute is the only correct tool; view after a permute is either an error (good) or — after a .contiguous() cargo-culted in — a silent success with the order you hopefully wanted. The question to ask at every reshape(B, -1): "which axis varies fastest, and does the consumer agree?"
• Verification trick: build a tiny tensor with torch.arange, run the op chain, and read the result — element identities make ordering bugs visible in one print.

Fix

x_hwc = x.permute(0, 2, 3, 1)              # actually move the channel axis

flat = feat.transpose(1, 2).contiguous().view(B, -1)   # explicit + intentional
# or document the intended order and test it:
t = torch.arange(6).reshape(1, 2, 3)
assert t.transpose(1, 2).reshape(1, -1).tolist() == [[0, 3, 1, 4, 2, 5]]

Deeper lesson

• The same buffer-vs-logical distinction underlies: torch.from_numpy sharing memory with the source array (mutate one, corrupt the other), Tensor.expand creating zero-stride views (writing to them is UB — use repeat to materialize), and channels-last memory format for conv performance.
• Flattening convention bugs are the classic CNN→FC interface failure when porting between frameworks (PyTorch NCHW vs TF NHWC): weights permuted one way, activations the other.

Problem 32 — The training loop that spends 80% of its time waiting (device placement & hidden syncs)

Category: GPU performance / device semantics Difficulty: Medium Frequency tier: Tier 2 — role-standard; perf-debugging questions are increasingly common

Buggy code

model = Net().cuda()
metric_history = []

for x, y in loader:                        # loader: num_workers=0, pin_memory=False
    x, y = x.cuda(), y.cuda()
    loss = criterion(model(x), y)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

    metric_history.append(loss.item())     # BUG 1: .item() SYNCS every step
    acc = (model(x).argmax(1) == y).float().mean().item()   # BUG 2: extra fwd pass + sync
    print(f"loss={loss.item():.4f} acc={acc:.4f}")          # BUG 3: sync + IO every step

new_data = torch.tensor(np_batch)          # BUG 4 (elsewhere): CPU tensor into CUDA model
out = model(new_data)                      # RuntimeError: expected ... cuda ... got cpu

Symptoms

• nvidia-smi shows 25–40% GPU utilization; training is far slower than the same model in a colleague's script; profiler shows large cudaStreamSynchronize blocks.
• Bug 4 is the loud version everyone learns first: Expected all tensors to be on the same device.

Diagnostic reasoning

• CUDA ops are asynchronous: python queues kernels and runs ahead. Any op that needs the value on the CPU — .item(), .cpu(), print(tensor), float(tensor), if tensor > 0: — forces a full pipeline sync. Once per step is tolerable; three per step (plus an extra forward pass for a training-batch "accuracy") serializes GPU and CPU.
• Diagnosis procedure to narrate: (1) nvidia-smi utilization — low means the GPU is starving or syncing; (2) torch.profiler one epoch — look at what dominates: DataLoader.__next__ (input pipeline, see Problem 42) vs sync ops vs kernels; (3) fix the biggest block, re-measure. Perf debugging is measurement-driven, exactly like correctness debugging.
• Cheap wins list: accumulate loss as a tensor and .item() once per epoch (or every N steps), compute train metrics from the logits you already have (no second forward), log every 50 steps, num_workers>0, pin_memory=True with non_blocking=True transfers.

Fix

loader = DataLoader(ds, batch_size=256, num_workers=8,
                    pin_memory=True, persistent_workers=True)

running = torch.zeros(2, device="cuda")            # [loss_sum, correct]
for step, (x, y) in enumerate(loader):
    x = x.cuda(non_blocking=True); y = y.cuda(non_blocking=True)
    logits = model(x)
    loss = criterion(logits, y)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

    running += torch.stack([loss.detach() * y.numel(),
                            (logits.argmax(1) == y).sum()])   # stays on GPU, no sync
    if step % 50 == 0:
        print(f"loss={running[0].item()/N:.4f}")   # sync every 50 steps, not 3x per step

Deeper lesson

• "The GPU is a remote server you send work to" is the mental model; every value read is a round-trip. The same intuition explains why if loss.isnan(): every step is expensive and why device-side accumulators are the pattern for metrics.
• Loud device bugs have quiet cousins: a model with a buried CPU-resident buffer (tensor created in forward without device=x.device), .to(device) returning a new tensor that someone forgot to assign (x.to(device) vs x = x.to(device)) — the latter is a top-5 beginner bug worth naming.

Problem 33 — The RNN whose backward pass reaches back to batch #1 (hidden-state graph growth)

Category: Recurrent models / truncated BPTT Difficulty: Medium-hard Frequency tier: Tier 3 — differentiator; less common now, but it tests graph-lifetime understanding that transfers to any stateful model

Buggy code

model = nn.LSTM(input_size=64, hidden_size=256, batch_first=True)
h = None
for batch in loader:                       # a long stream, chunked into batches
    out, h = model(batch, h)               # BUG: h carries the graph across batches
    loss = criterion(out, targets)
    loss.backward()                        # backprops through ALL previous batches
    optimizer.step(); optimizer.zero_grad()

Symptoms

• Each step is slower than the previous one; memory grows linearly until either OOM or RuntimeError: Trying to backward through the graph a second time (once a previous segment's buffers were freed by the first backward).
• Verified: the carried hidden state has grad_fn set — it is a live graph node, not a plain value.

Diagnostic reasoning

• The hidden state returned by the LSTM is part of batch N's autograd graph. Feeding it into batch N+1 splices the graphs together, so backward() on batch N+1's loss traverses back through batch N, N−1, … — quadratically growing work and memory, and a double-backward error once earlier buffers are freed.
• The intended semantics is truncated BPTT: carry the value of the hidden state (the model stays stateful) but cut the gradient at the batch boundary. That's precisely h.detach().
• Fingerprints to check on any stateful loop: per-step wall time increasing, memory increasing, and state.grad_fn is not None where you expected a plain value. The identical disease appears in non-RNN settings: EMA teacher outputs carried with graphs, cached embeddings reused across steps, replay buffers storing un-detached tensors (RL's version of this bug).

Fix

h = None
for batch in loader:
    if h is not None:
        h = tuple(s.detach() for s in h)   # LSTM state is (h, c)
    out, h = model(batch, h)
    loss = criterion(out, targets)
    loss.backward()
    optimizer.step(); optimizer.zero_grad()