Policy Optimization — All Variants & Tricks in RL

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0

Contents

1. Foundations: MDP, Value, Policy
2. REINFORCE and Baselines
3. Actor-Critic Family
4. Trust Region Methods: TRPO
5. PPO: The Workhorse
6. Deterministic and Off-Policy: DPG, DDPG, TD3
7. Maximum Entropy RL: SAC
8. Q-Learning Family (for context)
9. Model-Based Policy Optimization
10. Decision Transformer and Sequence Modeling for RL
11. Offline RL
12. Imitation Learning
13. RLHF: PPO Recipe
14. DPO and the Direct-Preference Family
15. GRPO and the Group-Relative Family
16. Pure-RL Reasoning: R1-style and Successors
17. RL for Diffusion Models
18. RL for Robotics: Massive Parallelism
19. Advanced and Niche Methods
20. Multi-Agent RL (Briefly)
21. Exploration
22. Practical Frameworks (2026)
23. Production Stack: 2026 Defaults

• Appendix A: Twenty-Five Equations to Memorize
• Appendix B: Year-by-Year Milestones

1. Foundations: MDP, Value, Policy

1.1 Markov Decision Process

\((\mathcal{S}, \mathcal{A}, P, r, \gamma, \mu_0)\) with state \(s \in \mathcal{S}\), action \(a \in \mathcal{A}\), transition \(P(s'|s, a)\), reward \(r(s, a)\), discount \(\gamma \in [0, 1)\), initial-state distribution \(\mu_0\).

1.2 Policy types

• Stochastic: \(\pi(a|s) = P[a_t = a | s_t = s]\).
• Deterministic: \(a = \mu(s)\).
• Parameterized: \(\pi_\theta(a|s)\), often softmax over logits or Gaussian over \(\mu_\theta(s), \sigma_\theta(s)\).

1.3 Returns and value functions

\[G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}, \quad V^\pi(s) = \mathbb{E}_\pi[G_t | s_t = s], \quad Q^\pi(s, a) = \mathbb{E}_\pi[G_t | s_t = s, a_t = a].\]

Advantage: \(A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)\).

1.4 Bellman equations

Expectation:

\[V^\pi(s) = \mathbb{E}_{a\sim\pi}\big[r(s, a) + \gamma\,\mathbb{E}_{s'}[V^\pi(s')]\big], \quad Q^\pi(s, a) = \mathbb{E}[r + \gamma\,\mathbb{E}_{a'\sim\pi} Q^\pi(s', a')].\]

Optimality:

\[V^*(s) = \max_a \mathbb{E}[r + \gamma V^*(s')], \quad Q^*(s, a) = \mathbb{E}[r + \gamma\max_{a'} Q^*(s', a')].\]

1.5 Policy gradient theorem (Sutton et al.)

\[\nabla_\theta J(\theta) = \mathbb{E}_{s\sim d^\pi, a\sim\pi_\theta}\big[\nabla_\theta\log\pi_\theta(a|s)\,Q^\pi(s, a)\big] = \mathbb{E}[\nabla_\theta\log\pi_\theta(a|s)\,A^\pi(s, a)].\]

The advantage form is preferred: subtracting a baseline does not bias the gradient but reduces variance.

1.5.1 Derivation sketch

Express \(J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} R(\tau)\). Use \(\nabla_\theta\log p_\theta(\tau) = \sum_t \nabla_\theta\log\pi_\theta(a_t|s_t)\) since transitions don't depend on \(\theta\). Combine with reward-to-go and the Markov property to get the per-step form.

Subtract a state-dependent baseline; baseline gradient has zero expectation by score-function lemma.

1.6 Generalized Advantage Estimation (GAE)

TD residual \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\):

\[\hat A_t^{\mathrm{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l\,\delta_{t+l}.\]

\(\lambda = 0\): TD(0), high bias / low variance.

\(\lambda = 1\): full Monte Carlo, low bias / high variance.

Standard:

\[\gamma = 0.99, \quad \lambda = 0.95.\]

2. REINFORCE and Baselines

2.1 Vanilla REINFORCE (Williams 1992)

Sample trajectories under \(\pi_\theta\), compute total return \(R\), update:

\[\theta \leftarrow \theta + \alpha\,\nabla_\theta\log\pi_\theta(a_t|s_t)\,R.\]

On-policy, episodic, high variance.

2.2 REINFORCE with baseline

Replace \(R\) with \(R - b(s_t)\), baseline \(b\) a learned value function. Same expectation; lower variance.

2.3 Reward-to-go vs total return

Use \(G_t = \sum_{k\ge t}\gamma^{k-t} r_k\) instead of total \(R\) at each timestep. Lower variance because earlier rewards aren't credited to later actions.

2.4 Implementation pseudocode

for episode in episodes:
    states, actions, rewards = run_episode(pi)
    Gs = compute_returns_to_go(rewards, gamma)
    Vs = value_net(states)
    advs = Gs - Vs
    pi_loss = -(log_pi(actions, states) * advs).mean()
    v_loss = (Gs - Vs).pow(2).mean()
    optimize(pi_loss + c_v * v_loss)

3. Actor-Critic Family

3.1 Vanilla Actor-Critic

Actor: parameterized \(\pi_\theta\). Critic: \(V_\phi\) or \(Q_\phi\) trained with TD target.

\[\nabla_\theta J = \mathbb{E}[\nabla_\theta\log\pi_\theta(a|s)\,\hat A(s, a)], \quad \mathcal{L}_\phi = \mathbb{E}[(\hat V_\phi(s) - (r + \gamma V_\phi(s')))^2].\]

3.2 A2C (synchronous) vs A3C (asynchronous)

A3C: many actors run in parallel, each with own copy; gradients pushed asynchronously to a central learner.

A2C: synchronize across actors first, then one update; cleaner; usually better.

3.3 V-trace (IMPALA)

Off-policy correction for distributed actor-learner setups. Truncated importance ratios \(\bar\rho_t = \min(\rho_t, \bar\rho)\); trace:

\[\hat V(s_t) = V(s_t) + \sum_{k=t}^{t+n-1}\gamma^{k-t}\Big(\prod_{i=t}^{k-1} c_i\Big)\bar\rho_k\,\delta_k V.\]

4. Trust Region Methods: TRPO

4.1 Surrogate objective

For stationary distribution \(d^{\pi_{\mathrm{old}}}\) and importance ratio \(r_\theta(s, a) = \pi_\theta(a|s)/\pi_{\mathrm{old}}(a|s)\):

\[L^{\mathrm{old}}(\theta) = \mathbb{E}_{s\sim d^{\pi_{\mathrm{old}}},\,a\sim\pi_{\mathrm{old}}}[r_\theta(s, a)\,\hat A^{\pi_{\mathrm{old}}}(s, a)].\]

4.2 TRPO (Schulman et al. 2015)

Maximize \(L^{\mathrm{old}}\) subject to a trust-region constraint:

\[\max_\theta L^{\mathrm{old}}(\theta) \quad\text{s.t.}\quad \mathbb{E}_{s\sim d^{\pi_{\mathrm{old}}}}\big[\mathrm{KL}\big(\pi_{\mathrm{old}}(\cdot|s)\,\|\,\pi_\theta(\cdot|s)\big)\big] \le \delta.\]

Solve via natural policy gradient + conjugate gradient + line search:

\[\theta \leftarrow \theta + \sqrt{\frac{2\delta}{g^\top F^{-1} g}}\,F^{-1} g, \quad g = \nabla_\theta L, \quad F = \mathbb{E}[\nabla\log\pi\,\nabla\log\pi^\top].\]

\(F\) approximated by Fisher information matrix; CG iteratively computes \(F^{-1} g\). Theoretically clean, practically heavy.

4.3 ACKTR

TRPO with K-FAC (Kronecker-factored approximate curvature) instead of CG. Cheaper natural gradient estimate.

4.4 Natural Policy Gradient

Direction \(F^{-1} g\) instead of \(g\). Invariant to reparameterization; faster convergence than vanilla PG in practice.

5. PPO: The Workhorse

5.1 PPO-Penalty (KL regularized)

\[\mathcal{L}^{\mathrm{PEN}}(\theta) = \mathbb{E}[r_\theta\hat A] - \beta\,\mathbb{E}[\mathrm{KL}(\pi_{\mathrm{old}}\,\|\,\pi_\theta)],\]

\(\beta\) adjusted to maintain a target KL.

5.2 PPO-Clip (the standard)

Key

PPO clipped surrogate:

\[\mathcal{L}^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t\big[\min(r_t(\theta)\,\hat A_t, \; \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\,\hat A_t)\big].\]

\(\epsilon \in [0.1, 0.3]\) typical. The clip prevents the ratio from growing without bound when \(\hat A > 0\), or shrinking unboundedly when \(\hat A < 0\).

Total loss (per rollout):

\[\mathcal{L} = \mathcal{L}^{\mathrm{CLIP}} - c_v\mathcal{L}^{\mathrm{VF}} + c_e\,\mathcal{H}[\pi_\theta].\]

5.3 The 17 implementation details that matter

PPO's reported performance can vary 10× based on tiny implementation choices (Engstrom et al. 2020). The most consequential:

• Advantage normalization (per minibatch).
• Orthogonal initialization with appropriate gain.
• Adam \(\epsilon = 10^{-5}\), not the default \(10^{-8}\).
• Value function clipping (matches clip range).
• Reward scaling / clipping.
• Observation normalization (running mean/std).
• Multiple epochs (typically 4–10) per rollout.
• Minibatch updates (not full batch).
• Gradient clipping (global norm \(\sim 0.5\)).
• Annealed learning rate.

Watch out

PPO results in papers are often not reproducible because of these details.

When implementing, follow CleanRL's reference — it documents every detail.

★ 2026 SOTA update — PPO for long-CoT reasoning

• VAPO: Value-based Augmented PPO for long-CoT; revives a value model with length-adaptive GAE, decoupled clip and value pretraining; beats DAPO/R1-Zero-Qwen-32B on AIME24 (60.4) in ~5k stable steps.
• Lite PPO: from 'Tricks or Traps?' deep dive; shows group-level advantage normalization + token-level loss aggregation alone beats stacked GRPO/DAPO tricks for critic-free vanilla-PPO reasoning.

6. Deterministic and Off-Policy: DPG, DDPG, TD3

6.1 Deterministic Policy Gradient theorem

For deterministic policy \(\mu_\theta\):

\[\nabla_\theta J = \mathbb{E}_{s\sim d^\mu}\big[\nabla_\theta\mu_\theta(s)\,\nabla_a Q^\mu(s, a)\big|_{a=\mu_\theta(s)}\big].\]

6.2 DDPG (Lillicrap et al. 2016)

Off-policy DPG with target networks for stability:

\[\mathcal{L}_Q = \mathbb{E}\big[(r + \gamma Q_{\bar\phi}(s', \mu_{\bar\theta}(s')) - Q_\phi(s, a))^2\big].\]

Targets updated via Polyak averaging: \(\bar\phi \leftarrow \tau\phi + (1-\tau)\bar\phi\), \(\tau \sim 0.005\). Exploration via OU or Gaussian noise added to actions.

6.3 TD3 (Fujimoto et al. 2018)

Three fixes to DDPG:

1. Clipped double Q: train two critics; use \(\min(Q_1, Q_2)\) in target.
2. Target policy smoothing: \(\mu(s') + \mathrm{clip}(\mathcal{N}(0, \sigma), -c, c)\).
3. Delayed policy updates: update actor every \(d = 2\) critic updates.

The combination dramatically reduces overestimation bias. Standard for continuous-control off-policy.

7. Maximum Entropy RL: SAC

7.1 Maximum entropy formulation

\[J(\pi) = \sum_t \mathbb{E}[r(s_t, a_t) + \alpha\,\mathcal{H}(\pi(\cdot|s_t))].\]

Encourages exploration; gives multimodal optimal policies; often more stable than reward-only.

7.2 Soft Bellman

\[Q^*(s, a) = r + \gamma\,\mathbb{E}_{s'}\big[\mathbb{E}_{a'\sim\pi}[Q^*(s', a') - \alpha\log\pi(a'|s')]\big].\]

Optimal soft policy:

\[\pi^*(a|s) = \frac{\exp(Q^*(s, a)/\alpha)}{\int \exp(Q^*(s, a')/\alpha)\,\mathrm{d}a'}.\]

7.3 SAC (Haarnoja et al. 2018)

• Two Q-networks (clipped double Q like TD3).
• Target networks via Polyak.
• Stochastic Gaussian policy with reparameterization.
• Auto-tuned \(\alpha\) via entropy constraint \(\mathbb{E}[\mathcal{H}(\pi)] \ge \mathcal{H}_{\mathrm{target}}\):

\[\mathcal{L}(\alpha) = \mathbb{E}[-\alpha(\log\pi(a|s) + \mathcal{H}_{\mathrm{target}})].\]

Policy update minimizes

\[\mathrm{KL}\Big(\pi_\theta(\cdot|s)\,\Big\|\,\frac{\exp(Q(s, \cdot)/\alpha)}{Z(s)}\Big).\]

7.4 SAC vs TD3 vs DDPG quick guide

• SAC: stochastic, auto-tuned exploration, robust default for continuous control.
• TD3: deterministic, slightly faster, needs careful exploration noise tuning.
• DDPG: legacy; replace with TD3 / SAC.

8. Q-Learning Family (for context)

8.1 Q-learning

\[Q(s, a) \leftarrow Q(s, a) + \alpha[r + \gamma\max_{a'} Q(s', a') - Q(s, a)].\]

Off-policy, tabular convergence guaranteed. Function approximation can diverge.

8.2 DQN (Mnih et al. 2015)

Q-learning with neural net + replay buffer + target network. Stabilized Atari learning.

8.3 Improvements (Rainbow integration)

Double DQN: decouple action selection from evaluation. Dueling DQN: factorize \(Q = V + A - \bar A\). Prioritized Experience Replay. Multi-step returns (n-step TD). Noisy Nets: parametric exploration. Distributional RL: predict full return distribution.

8.4 Distributional RL

C51: predict 51-bin categorical return distribution. QR-DQN: predict quantile values. IQN: implicit quantile network sampling continuous quantiles.

9. Model-Based Policy Optimization

9.1 Dyna-Q

Mix real experience with simulated rollouts from a learned model. Cheap, increases sample efficiency.

9.2 PILCO, PETS (probabilistic ensembles)

Gaussian Process or ensemble of NNs models the dynamics. Plan / optimize over imagined rollouts.

9.3 MuZero, EfficientZero

Learn three networks: representation, dynamics, prediction (value + policy). Plan with MCTS at every action.

EfficientZero adds self-supervised auxiliary targets for sample efficiency at \(\sim 100\times\) less data.

9.4 Dreamer V1/V2/V3

RSSM (Recurrent State-Space Model) learns latent dynamics. Imagine \(H\)-step rollouts; train actor/critic on imagined trajectories.

V3 specifics:

• Symlog targets: \(\mathrm{symlog}(x) = \mathrm{sgn}(x)\log(|x| + 1)\) for stability across reward scales.
• Two-hot reward / value distributional heads.
• Free-bits in KL between \(q\) and prior.
• Same hyperparameters across >150 tasks (Atari, DMC, Crafter, Minecraft).

9.5 TD-MPC, TD-MPC2

Latent-dynamics model + sampling-based MPC at planning. Strong on continuous-control benchmarks; robust across embodiments.

9.6 IRIS, DIAMOND

IRIS: tokenizer + Transformer world model. DIAMOND: diffusion world model for Atari. Both impressive at low data.

10. Decision Transformer and Sequence Modeling for RL

10.1 Decision Transformer (Chen et al. 2021)

Frame RL as sequence modeling. Conditioning on returns-to-go:

\[\pi_\theta(a_t | R_t, s_t, a_{<t}, R_{<t}, s_{<t}),\]

trained by cross-entropy on actions. At test, prompt with target return \(R_0\).

10.2 Trajectory Transformer (Janner et al. 2021)

Discretize state, action, reward; train autoregressively; sample full trajectories and select actions via beam search planning.

10.3 Gato (DeepMind)

Single Transformer trained on text, vision, atari, robotics, etc. Showed sequence modeling generalizes across modalities.

10.4 Diffuser, Decision Diffuser

Generate trajectories with diffusion; condition on returns or constraints. Diversity and constraint handling.

11. Offline RL

11.1 The fundamental problem

Pure off-policy training on a fixed dataset \(\mathcal{D}\) overestimates \(Q\) on out-of-distribution actions → exploitation of model errors → catastrophic policy.

11.2 BCQ (Batch-Constrained Q-learning)

Constrain policy to actions seen in \(\mathcal{D}\) via a generative model.

11.3 CQL (Conservative Q-Learning)

Push down \(Q\) on OOD actions:

\[\mathcal{L} = \mathcal{L}_{\mathrm{TD}} + \alpha\big(\mathbb{E}_{s\sim\mathcal{D}}\mathbb{E}_{a\sim\mu(\cdot|s)} Q(s, a) - \mathbb{E}_{(s,a)\sim\mathcal{D}} Q(s, a)\big),\]

\(\mu\) a wide distribution (uniform / current policy) to penalize.

11.4 IQL (Implicit Q-Learning)

Avoid querying \(Q\) at OOD actions. Learn \(V\) via expectile regression of \(Q\):

\[\mathcal{L}_V = \mathbb{E}[\rho_\tau(Q_{\bar\phi}(s, a) - V_\psi(s))], \quad \rho_\tau(u) = |\tau - \mathbb{1}_{u<0}|\cdot u^2.\]

Then \(Q\) regresses to \(r + \gamma V\). Policy: advantage-weighted regression \(\pi \propto e^{\beta(Q-V)}\).

11.5 AWR / AWAC

Advantage-Weighted Regression / Actor-Critic: weight BC by \(\exp(\beta A)\); soft constraint to dataset.

11.6 Diffusion-QL, IDQL

Diffusion as the policy class for offline RL. Captures multi-modal action distributions; combines well with conservative \(Q\).

11.7 Decision Diffuser

Sequence-modeling offline RL via diffusion. Samples trajectories; condition on returns / constraints.

11.8 Offline-to-online

Pre-train offline (CQL/IQL); fine-tune online with PPO or SAC. Combines safety of offline pretraining with adaptation of online RL.

12. Imitation Learning

12.1 Behavior Cloning (BC)

\[\mathcal{L}_{\mathrm{BC}} = \mathbb{E}_{(s,a)\sim\mathcal{D}_{\mathrm{expert}}}[-\log\pi_\theta(a|s)].\]

Distribution shift compounds errors quadratically with horizon.

12.2 DAgger (Ross et al. 2011)

Run \(\pi_\theta\), query expert on visited states, append to dataset:

\[\mathcal{D}_{k+1} = \mathcal{D}_k \cup \{(s, \pi^*(s)) : s \sim \pi_{\theta_k}\}.\]

Reduces compounding error to linear.

12.3 GAIL (Generative Adversarial Imitation Learning)

Discriminator distinguishes expert from policy; policy reward \(= -\log(1 - D(s, a))\). Trains via PPO. No need to know expert reward.

12.4 AIRL (Adversarial IRL)

GAIL with structured discriminator that recovers a reward function: \(D(s, a) = \exp(f(s, a))/(\exp(f) + \pi(a|s))\).

12.5 SQIL (Soft Q Imitation Learning)

Set reward \(+1\) on expert transitions, \(0\) elsewhere. Train with off-policy soft Q-learning. Surprisingly strong baseline.

12.6 Implicit Behavioral Cloning

Learn an energy-based model \(E_\theta(s, a)\); act via \(a = \arg\min_a E_\theta(s, a)\). Captures multi-modal demonstrations.

12.7 ACT (Action Chunking Transformer)

Predict \(H\)-step action chunks via CVAE:

\[\mathcal{L} = \mathbb{E}[\|\hat a_{t:t+H} - a_{t:t+H}\|_1] + \beta\,\mathrm{KL}(q(z|a, o)\,\|\,p(z)).\]

Temporal ensembling at inference for smooth actions. Standard for ALOHA / bimanual manipulation.

12.8 Diffusion Policy

Generate action sequence with conditional diffusion on observation history:

\[\mathcal{L} = \mathbb{E}_{k,\epsilon,a_0,o}\big\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_k}\,a_0 + \sqrt{1 - \bar\alpha_k}\,\epsilon, \; k, \; o)\big\|^2.\]

Multi-modal action distributions. Receding-horizon execution.

13. RLHF: PPO Recipe

13.1 The classic InstructGPT recipe

1. Supervised fine-tuning (SFT) on demonstrations.
2. Train Bradley-Terry reward model on pairwise preferences.
3. PPO with KL penalty against the SFT model.

13.2 Bradley-Terry reward model

Pairs \((x, y_w, y_l)\) with \(y_w\) preferred. Reward model \(r_\phi\):

\[\mathcal{L}_{\mathrm{RM}} = -\mathbb{E}[\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))].\]

Best practices: ranking loss for \(K > 2\) comparisons, reward model ensembles, length normalization.

13.3 KL-regularized RL objective

\[\max_\pi \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi(\cdot|x)}[r_\phi(x, y)] - \beta\,\mathrm{KL}(\pi(\cdot|x)\,\|\,\pi_{\mathrm{ref}}(\cdot|x)).\]

Per-token reward used by PPO:

\[\tilde r_t = -\beta(\log\pi(a_t|s_t) - \log\pi_{\mathrm{ref}}(a_t|s_t)) + r_\phi(x, y)\cdot\mathbb{1}[t = T].\]

13.4 Closed-form optimal policy

\[\pi^*(y|x) = \frac{1}{Z(x)}\pi_{\mathrm{ref}}(y|x)\exp\Big(\frac{r(x, y)}{\beta}\Big).\]

This is the foundation for DPO and friends.

13.5 Rejection-sampling fine-tuning (RFT, RAFT, BoN)

Sample \(N\) responses; keep top-\(K\) by reward; SFT on those. Cheap, no PPO; works surprisingly well as a baseline.

Iterative variants (RAFT) repeat.

13.6 Iterative DPO / online DPO

Generate responses with current \(\pi_\theta\); collect preferences (RM or human); update via DPO. Iterate. Closes the gap with PPO at lower complexity.

14. DPO and the Direct-Preference Family

14.1 DPO derivation

From the closed-form optimal policy, solve for \(r\):

\[r(x, y) = \beta\log\frac{\pi^*(y|x)}{\pi_{\mathrm{ref}}(y|x)} + \beta\log Z(x).\]

Substitute into the BT log-likelihood; the \(\log Z(x)\) cancels in differences.

Key

DPO loss:

\[\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}\Big).\]

No reward model, no value function, no PPO.

14.2 DPO failure modes

• Can reduce \(\pi_\theta(y_w)\) and \(\pi_\theta(y_l)\) both — the loss only cares about the difference.
• Length bias: longer responses → larger \(\log\pi\) magnitude → over-fit to length.
• Reference-policy bias: inherits whatever \(\pi_{\mathrm{ref}}\) knows; bad ref ⇒ bad DPO.
• Pair quality matters more than quantity.

14.3 IPO (Identity Preference Optimization, Azar et al. 2023)

Squared margin loss; robust to deterministic preferences:

\[\mathcal{L}_{\mathrm{IPO}} = \mathbb{E}\Big(h_\theta(y_w, y_l; x) - \frac{1}{2\beta}\Big)^2, \quad h_\theta = \log\frac{\pi_\theta(y_w|x)\,\pi_{\mathrm{ref}}(y_l|x)}{\pi_{\mathrm{ref}}(y_w|x)\,\pi_\theta(y_l|x)}.\]

14.4 KTO (Kahneman-Tversky Optimization)

Single-response thumbs-up/down (no pairs). Prospect-theory utility:

\[v_{\mathrm{KTO}}(y; x) = \begin{cases}\sigma(\beta(z(y) - \mathrm{KL}_0)) & y \succ \\ \sigma(\beta(\mathrm{KL}_0 - z(y))) & y \prec\end{cases}, \quad z(y) = \log\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x).\]

Loss: \(\mathbb{E}[1 - v_{\mathrm{KTO}}]\) with loss-aversion weighting. Practical when collecting pair labels is expensive.

14.5 ORPO (Odds Ratio Preference Optimization)

Combine SFT + odds-ratio penalty in one stage:

\[\mathcal{L}_{\mathrm{ORPO}} = \mathcal{L}_{\mathrm{SFT}}(y_w) - \lambda\log\sigma\Big(\log\frac{\mathrm{odds}(\pi_\theta(y_w|x))}{\mathrm{odds}(\pi_\theta(y_l|x))}\Big).\]

No reference policy needed. Removes the SFT → DPO two-stage pipeline.

14.6 SimPO (Simple Preference Optimization)

Length-normalized log-prob; drops reference policy:

\[\mathcal{L}_{\mathrm{SimPO}} = -\log\sigma\Big(\frac{\beta}{|y_w|}\log\pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l|x) - \gamma\Big).\]

Margin \(\gamma\) prevents over-fitting. Memory cost halved (no \(\pi_{\mathrm{ref}}\)).

14.7 cDPO (conservative), rDPO (robust)

cDPO: down-weight high-confidence pairs. rDPO: assume preference labels noisy with prob \(\epsilon\); corrected loss handles label noise.

14.8 APO, Step-DPO, sDPO

APO (Anchored Preference Optimization): anchor \(\pi_\theta\) to a fixed reference at each step.

Step-DPO: stepwise preferences in chain-of-thought (which step is the wrong one). sDPO: sequential / online DPO with iterative dataset updates.

14.9 SLiC-HF (Sequence Likelihood Calibration)

Hinge loss on log-prob differences:

\[L_{\mathrm{SLiC}} = \max(0, \delta - (\log\pi_\theta(y_w|x) - \log\pi_\theta(y_l|x))).\]

Predates DPO; same family conceptually.

14.10 When to pick what

• Have reward model + budget for PPO infra: PPO-RLHF.
• Have preference pairs only, simplest path: DPO.
• Want smaller memory (no ref): SimPO or ORPO.
• Have only thumbs-up/down: KTO.
• Fighting deterministic preferences: IPO.
• Need reasoning + verifiable rewards: GRPO (next section).

15. GRPO and the Group-Relative Family

15.1 Motivation

Per-token PPO on long sequences has high variance; the value function is hard to fit and consumes memory.

GRPO removes the value function entirely and uses a group baseline.

15.2 GRPO (DeepSeek)

For prompt \(x\), sample \(G\) responses \(\{y_i\}\); score with reward (RM or programmatic verifier) \(r_i\). Group-relative advantage:

\[\hat A_i = \frac{r_i - \mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\}) + \epsilon}.\]

Per-token clipped objective:

Key

\[\mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\big(\rho_{i,t}\hat A_i, \; \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat A_i\big)\right] - \beta\,\mathrm{KL}[\pi_\theta\,\|\,\pi_{\mathrm{ref}}],\]

\[\rho_{i,t} = \pi_\theta(y_{i,t}|x, y_{i,<t})/\pi_{\mathrm{old}}(y_{i,t}|x, y_{i,<t}).\]

15.3 DAPO (Decoupled clip + Dynamic sampling)

• Decoupled clip: asymmetric \(\epsilon^+ > \epsilon^-\); allows larger upward updates than downward.
• Dynamic sampling: for prompts where all \(r_i\) identical, re-sample (no signal otherwise).
• Token-level loss aggregation, not response-level.
• Overlong reward shaping.

15.4 Dr. GRPO (Dr. = "done right")

Drop length normalization (the \(1/|y_i|\)) to remove length and difficulty bias.

Argues GRPO unintentionally rewards longer responses.

15.5 REINFORCE++

REINFORCE with global advantage normalization (across batch) + per-token KL. Simpler than GRPO; competitive in some settings.

15.6 RLOO (REINFORCE Leave-One-Out)

Baseline = mean reward of other samples in group:

\[\hat A_i = r_i - \frac{1}{G - 1}\sum_{j\ne i} r_j.\]

Unbiased and lower variance than vanilla REINFORCE. Popular for RLHF-on-LLMs.

15.7 ReMax

Variance reduction via baseline = greedy-decode reward:

\[\hat A = r(y_{\mathrm{sampled}}) - r(y_{\mathrm{greedy}}).\]

No value network; one extra greedy forward per prompt. Cheap.

15.8 VinePPO

Extension of PPO with multiple rollout branches per state for variance reduction. Requires resettable simulator.

15.9 REBEL (Regression to Relative Reward via Bregman Loss)

Closed-form-style update derived from relative-entropy minimization; no ratio clipping. Approaches PPO performance with simpler objective.

★ 2026 SOTA update — Sequence-level GRPO successors

• GSPO: Qwen3's Group Sequence Policy Optimization; uses sequence-level (not token-level) importance ratios, clipping and rewarding, fixing GRPO's variance/collapse and stabilizing MoE RL.
• CISPO: MiniMax-M1's Clipped IS-weight Policy Optimization; clips importance-sampling weights (detached) rather than token updates, keeping gradient signal on all tokens including low-prob ones.
• GMPO: Geometric-Mean Policy Optimization; maximizes geometric (not arithmetic) mean of token rewards, robust to outlier IS ratios; +4.1% over GRPO on math (ICLR 2026).

16. Pure-RL Reasoning: R1-style and Successors

16.1 DeepSeek-R1 / R1-Zero

R1-Zero: pure RL on base model with GRPO + verifiable rewards (math correct/incorrect, code passes test, format adherence). No SFT. Reasoning emerges:

• Long visible CoT spontaneously appears.
• "Aha moments": model self-corrects mid-reasoning.
• Average response length doubles over training.

R1: cold-start SFT → RL → rejection sampling SFT → RL again. Adds language consistency reward to suppress mixed-language output.

16.2 Replications and extensions

Open replications: TinyZero, Open-R1, SimpleRL, Logic-RL, ReST-MCTS*. Add tool-use, retrieval, code execution.

16.3 Reward design for reasoning

• Format reward: well-formed <think>...</think><answer>...</answer>.
• Accuracy reward: match GT (math, multiple choice, exact-match).
• Code reward: tests pass.
• Length penalty (mild) to prevent unbounded scaling.
• Anti-reward-hacking: filter trivial-pass cases.

16.4 Process Reward Models (PRMs)

Per-step labels \(y^t \in \{0, 1\}\):

\[\mathcal{L}_{\mathrm{PRM}} = -\sum_t\big[y_t\log p_\phi(y_t = 1 | s_{\le t}) + (1 - y_t)\log(1 - p_\phi)\big].\]

Auto-labeling: from each prefix, sample \(K\) continuations; label step as good if \(> \tau\) fraction succeed (Math-Shepherd, OmegaPRM, MIPS).

16.5 Inference-time scaling

• Best-of-N: \(\mathbb{E}[\max_i r_i] \approx \mu + \sigma\sqrt{2\ln N}\).
• Self-consistency: majority vote.
• MCTS / tree search: rStar, AlphaProof, Mulberry; PRM as heuristic.
• Beam search with verifier.
• Reflexion / self-critique loops.

16.6 Multimodal R1-style

VLM-R1, Vision-R1, MM-EUREKA, R1-V, LMM-R1, Video-R1: same GRPO machinery on VLMs.

Programmatic rewards: IoU on detection, mask-IoU on segmentation, exact-match on visual QA, format reward.

★ 2026 SOTA update — RLVR reasoning: recipes & stability

• Kimi k1.5: multimodal long-CoT RL scaling with long-context (128k) rollouts, online mirror-descent policy optimization, length penalty and long2short distillation; no MCTS/value/PRM needed.
• ProRL: Prolonged RL (2k+ steps) with KL control, reference-policy resetting and diverse verifiable tasks; genuinely expands reasoning boundaries (pass@k) beyond the base model.
• Magistral: Mistral's from-scratch RLVR pipeline; Magistral Medium trained pure-RL (no distillation) for ~50% AIME24 gain; GRPO tweaks (no KL, loss normalization, adaptive length reward).
• Entropy Mechanism: explains entropy collapse via covariance of logprob and advantage; Clip-Cov and KL-Cov restrict high-covariance tokens to sustain exploration and downstream gains.

17. RL for Diffusion Models

17.1 Why RL on diffusion

Diffusion trained via MLE produces realistic but not necessarily preferred samples. RL fine-tunes for human preference (aesthetic, prompt fidelity, safety) without architecture changes.

17.2 DDPO (Denoising Diffusion Policy Optimization)

Cast diffusion as multi-step MDP (each denoising step is an action). PPO with terminal reward = aesthetic / preference score. Per-step policy gradient through the sampler.

17.3 DPOK

Per-batch PPO with KL to base diffusion model.

17.4 Diffusion-DPO

Lift DPO. Preferences over images \((x_w, x_l | c)\), surrogate via diffusion losses:

\[\mathcal{L}_{\mathrm{D\text{-}DPO}} = -\mathbb{E}\log\sigma\Big(-\beta\big(L_\theta(x_w, c) - L_{\theta_{\mathrm{ref}}}(x_w, c) - L_\theta(x_l, c) + L_{\theta_{\mathrm{ref}}}(x_l, c)\big)\Big).\]

Standard for SD3/FLUX aesthetic alignment.

17.5 DRaFT, AlignProp, ReFL

Reward backpropagation through the sampler. Diffusion ODE/SDE differentiable with checkpointing; compute \(\partial r(\hat x)/\partial\theta\) end-to-end. Memory-heavy; works for short (\(\sim 25\)-step) samplers. ReFL is the canonical citation.

17.6 SPIN, Diffusion-SPO

Self-play / iterative DPO for diffusion. Generate, prefer, train, repeat.

18. RL for Robotics: Massive Parallelism

18.1 The recipe in 2026

1. Build a GPU-parallel sim (\(10^4+\) envs in Isaac Lab / MJX / Genesis / ManiSkill 3).
2. Domain randomization on visual + dynamics nuisances.
3. PPO with massive batch sizes.
4. Asymmetric actor-critic: critic uses privileged info \(s\), actor only egocentric \(o\).
5. Distill privileged policy into vision-only student.

18.2 Asymmetric A-C distillation

\[\mathcal{L}_{\mathrm{distill}} = \mathbb{E}[\|\pi_S(o) - \pi_T(s, \xi)\|^2] + \alpha\,\mathbb{E}[\mathrm{KL}(\pi_T\,\|\,\pi_S)].\]

Teacher trained with PPO + privileged state; student deployed in real world.

18.3 Eureka (LLM-designed rewards)

Loop:

1. LLM proposes reward function code from task description.
2. Train RL on \(K\) candidates in parallel sim.
3. Evaluate task success on held-out evaluator.
4. Refine via evolutionary update on best candidates.

DrEureka: also designs domain-randomization ranges.

Successful for shadow-hand pen spinning, ANYmal locomotion, humanoid walking.

18.4 Residual RL on top of pretrained VLA

Pretrained VLA (\(\pi_0\), OpenVLA) for behavior; RL fine-tunes residual policy. Reduces sample complexity vs RL from scratch dramatically.

18.5 Curriculum learning

• Forward: start easy, scale difficulty.
• Reverse: start near goal, expand backward.
• Adaptive: adjust per-sample based on success rate.
• Goal-conditioned + HER: hindsight relabeling of failed trajectories.

18.6 HER (Hindsight Experience Replay)

Relabel failed trajectory's goal to whatever was achieved.

Free sparse-reward gradient signal.

Standard for goal-conditioned RL.

19. Advanced and Niche Methods

19.1 V-MPO, MPO

MPO (Maximum a posteriori Policy Optimization): alternating EM-style: E-step computes target distribution from advantage; M-step does supervised regression of policy to target. Stable, low-variance.

V-MPO: on-policy variant; matches PPO at scale on Atari, DMLab, Procgen.

19.2 REPS (Relative Entropy Policy Search)

Constrained policy update via relative entropy. Foundation for MPO and trust-region methods.

19.3 PCL (Path Consistency Learning)

Bridges policy gradient and Q-learning via softmax consistency. Off-policy.

19.4 SVG (Stochastic Value Gradients)

Reparameterization gradient through learned dynamics + value. Connects model-based with policy gradient.

19.5 Phasic Policy Gradient (PPG)

Separates policy and value networks; alternates auxiliary phase to distill value into policy backbone. Better feature sharing without interference.

19.6 Munchausen RL

Add log-policy as a reward bonus: \(r' = r + \alpha\log\pi\). Surprisingly effective regularizer; widely adopted in DQN-style algorithms.

20. Multi-Agent RL (Briefly)

20.1 Self-play and PSRO

Self-play: iteratively train against own past versions. PSRO (Policy-Space Response Oracle): maintain population; train best response to current meta-distribution.

20.2 QMIX, VDN, MAPPO, IPPO

VDN: factor joint \(Q\) as sum of per-agent \(Q\). QMIX: monotonic mixing network for centralized training, decentralized execution. MAPPO: PPO with centralized value function. IPPO: independent PPO (no centralization).

20.3 Counterfactual Multi-Agent (COMA)

Counterfactual baseline: marginalize agent's own action.

21. Exploration

21.1 Action-space exploration

\(\epsilon\)-greedy, Boltzmann (softmax), parametric noise (NoisyNets), entropy bonus.

21.2 Intrinsic motivation

ICM (Curiosity): reward = forward-model error in feature space.

RND (Random Network Distillation): reward = prediction error of a random target network.

NovelD: difference of RND scores between consecutive states.

BYOL-Explore: curiosity in BYOL feature space.

Plan2Explore, RIDE, EX2: various model-based / count-based / contrast-based variants.

21.3 Information gain

VIME: maximize KL between posterior and prior over dynamics. Disagreement: ensemble disagreement as exploration signal.

21.4 Goal-directed exploration

Go-Explore: archive states; replay to interesting frontier; explore from there.

HRL: high-level controller proposes subgoals; low-level pursues.

22. Practical Frameworks (2026)

★ 2026 SOTA update — Async off-policy RL systems

• AReaL: fully asynchronous RL system decoupling generation from training; staleness-controlled off-policy PPO variant gives up to 2.77x speedup at matched/better accuracy on math/code.

23. Production Stack: 2026 Defaults

Appendix A: Twenty-Five Equations to Memorize

1. Policy gradient theorem in advantage form.
2. GAE recurrence.
3. REINFORCE update with baseline.
4. TRPO trust-region constraint.
5. PPO clipped surrogate.
6. DPG theorem.
7. DDPG critic loss with target networks.
8. TD3 clipped double-Q + target smoothing.
9. SAC max-entropy objective and soft Bellman.
10. SAC's auto-tuned \(\alpha\) loss.
11. DQN Bellman target with target network.
12. Distributional Bellman update.
13. Decision Transformer conditioning on returns-to-go.
14. Bradley-Terry RM loss.
15. KL-regularized RLHF objective and closed-form \(\pi^*\).
16. DPO loss derivation cancelling \(\log Z(x)\).
17. IPO squared-margin loss.
18. KTO prospect-utility loss.
19. ORPO odds-ratio combined SFT+pref.
20. SimPO length-normalized loss.
21. GRPO group-relative advantage.
22. GRPO per-token clipped objective.
23. RLOO leave-one-out advantage.
24. Diffusion-DPO surrogate.
25. Asymmetric A-C distillation \(\mathcal{L}_{\mathrm{distill}}\).

Appendix B: Year-by-Year Milestones

• 1992: REINFORCE (Williams).
• 1999: Policy Gradient Theorem (Sutton et al.).
• 2014: DPG (Silver et al.).
• 2015: TRPO (Schulman et al.); DQN (Mnih et al.).
• 2016: A3C, Dueling/Double DQN, GAE, DDPG.
• 2017: PPO (Schulman et al.); IMPALA / V-trace.
• 2018: SAC (Haarnoja); TD3 (Fujimoto); Rainbow DQN; AlphaZero.
• 2019: SAC v2 (auto \(\alpha\)); MuZero.
• 2020: BCQ, CQL; Decision Transformer (released 2021).
• 2021: Decision Transformer / Trajectory Transformer; IQL; PEARL.
• 2022: PPO-RLHF (InstructGPT recipe); Dreamer V3; TD-MPC.
• 2023: DPO; SimPO antecedents; Diffusion Policy; ACT (ALOHA); EfficientZero V2.
• 2024: GRPO (DeepSeek); IPO, KTO, ORPO; Diffusion-DPO; \(\pi_0\) flow-matching policy; Eureka, DrEureka.
• 2025: DeepSeek-R1 (pure RL reasoning); DAPO; Dr. GRPO; RLOO mainstream; VLM-R1, MM-EUREKA, Vision-R1; \(\pi_{0.5}\).
• 2026: GRPO-family standard; multimodal R1-style standard; RL fine-tuning of diffusion routine; world-model + RL converging in robotics.