A — Decision Trees (Master Merge)

A single merged reference consolidating the "decision tree / which-method" guide from every cheat sheet in this knowledge base that has one (26 of 36 files). Each source's tree is preserved verbatim under its topic; paper links are kept intact.

Compiled August 2026. Organized into six thematic parts; a synthesized meta-decision guide (the cross-cutting trade-off axes) is up top. Source titles vary ("Decision Tree — Which X?", "Decision Guide", "How to …?") but all answer "which method should I pick?"

Contents

• Meta-Decision Guide
• Part I — LLMs: Architecture, Tokenization & Efficiency
  • Transformer / xFormer Catalogue
  • KV Cache
  • Mixture-of-Experts (MoE)
  • Quantization
  • Pruning
  • Distillation
  • Parameter-Efficient Fine-Tuning (PEFT)
  • Scaling Laws
• Part II — LLMs: Training, Reasoning, RL & Agents
  • Reasoning Technologies
  • Reward Functions
  • Agentic Intelligence
  • Prompt · Context · Harness · Graph Engineering & Self-Improving AI
  • Test-Time & Training-Free Optimization
• Part III — Data & Evaluation
  • Data Collection & Curation
  • Metrics & Evaluations
• Part IV — Generative Models: Diffusion, Video, 3D & World
  • Video Generation
  • 3D & Multi-View Generation
  • World Models
• Part V — Neural Rendering & 3D Reconstruction
  • NeRF
  • Gaussian Splatting
  • Neural Rendering
  • Structure from Motion
  • Delighting & Relighting
  • Photorealistic Avatars
• Part VI — Core Vision, Robotics & Autonomy
  • Vision-Language-Action (VLA) Models
  • Autonomous Driving

Meta-Decision Guide (the recurring axes)

Before the per-topic trees, the trade-off axes that drive almost all of them. Most "which method?" questions reduce to locating your problem on a few of these:

• Quality vs latency vs memory/cost — the universal triangle; you usually pick two. Every efficiency tree (quantization, pruning, KV cache, MoE) is a walk along it.
• Train vs training-free — reach for inference-time levers first (decoding, steering, model merging, search, test-time adaptation) before committing to a training run.
• Dense vs sparse / conditional compute — MoE, sparse attention, pruning: more capability per FLOP when routing is good, at the cost of memory and complexity.
• Explicit vs implicit representation — 3DGS (explicit, real-time) vs NeRF (implicit, compact); tokens vs latents; vectorized vs dense BEV. Explicit wins on speed/editing, implicit on compactness.
• Parallel vs sequential test-time compute — best-of-\(N\)/search (breadth) vs longer single chains (depth); allocate by problem difficulty.
• How strong is your verifier? — it caps every search, self-consistency, RL, and self-improvement choice. No verifier → majority vote and accept the plateau.
• How much distribution shift at test time? — none → run the frozen model; covariate shift → test-time adaptation (update BN/prompt); per-task/instance shift → test-time training / per-task LoRA.
• Data availability & privacy — public vs proprietary vs synthetic; and where knowledge should live: in weights (fine-tune), in context (RAG/prompt), or in a loadable skill.
• Scale of the problem vs budget — Chinchilla-style compute-optimal balance for pretraining; for inference, spend on the smallest model + strongest inference procedure that clears your bar.

Part I — LLMs: Architecture, Tokenization & Efficiency

Transformer / xFormer Catalogue

Merged from "Architectural Decisions: Quick Decision Tree" — XFormer_Catalogue_SOTA_Updated.md.

22.1 When to pick what?

• Need understanding only (classification, retrieval): encoder-only (ModernBERT, GTE, SigLIP).
• Need generation only (text): decoder-only (LLaMA / Qwen / DeepSeek family).
• Need conditional generation, fixed input/output structure: encoder-decoder (T5 / BART) — but mostly displaced by decoder-only with prompt template.
• Need vision understanding: ViT or hybrid (Swin, ConvNeXt) for backbone; LLaVA-style adapter for VLM.
• Need vision generation: DiT / MM-DiT (image), spatiotemporal MM-DiT (video).
• Need very long context: RoPE+YaRN+ring attention; or Mamba / RWKV / RetNet for very long but lower-quality.
• Need extreme efficiency at scale: MoE (Mixtral, DeepSeek MoE) or distillation (DistilBERT, MobileViT).
• Need permutation invariance (sets): Set Transformer / Perceiver.
• Need handling many modalities at once: Perceiver IO; or native multimodal (Chameleon-style).

22.2 Common architectural mistakes

• Using BERT in 2026: switch to ModernBERT.
• Using vanilla DDPM training: use flow matching instead.
• Using post-norm: it's only stable for shallow / well-tuned setups.
• Using LayerNorm in modern stacks: RMSNorm is faster and equivalent.
• Using sinusoidal absolute position: RoPE wins virtually everywhere.
• Using MHA without GQA at inference: KV cache will dominate memory.
• Using U-Net for new diffusion projects: MM-DiT is the new default.
• Using bolted-on adapter VLM as your long-term bet: native multimodal is winning.

KV Cache

Merged from "Decision Tree — "How to Cap KV?"" — KV_Cache_SOTA_Updated.md.

1. Designing a new model from scratch? \(\to\) MLA (DeepSeek pattern). Best long-term cache efficiency.
2. Retrofitting an existing model? \(\to\) GQA via continued pretraining (Llama 3 pattern).
3. Need streaming / unbounded context, OK with bounded effective context? \(\to\) Sliding window + sinks (Mistral / StreamingLLM).
4. Long context, full attention needed but memory tight? \(\to\) INT4 KV (KIVI) or FP8 KV.
5. Cloud serving with shared prefixes? \(\to\) Paged KV + automatic prefix caching (vLLM / SGLang).
6. Many requests, mixed lengths? \(\to\) Continuous batching (vLLM, SGLang, TGI).
7. Long prompts, short outputs (RAG, document QA)? \(\to\) SnapKV prefill-time compression.
8. VLM with many image tokens? \(\to\) FastV drop vision tokens after early layers.
9. Throughput at scale? \(\to\) Disaggregated prefill/decode + Mooncake-style KV pool.
10. Edge / on-device? \(\to\) GQA + INT4 KV + sliding window.

Mixture-of-Experts (MoE)

Merged from "Decision Tree — "MoE or Dense?"" — MoE_SOTA_Updated.md.

1. Memory-constrained device (mobile, edge)? → Dense. MoE memory overhead too high.
2. Single-batch latency-critical? → Dense, unless you can batch enough requests.
3. Cloud serving with high throughput? → MoE. Amortize memory; per-token compute lower.
4. Pretraining frontier model from scratch? → MoE. Compute-quality Pareto wins above \(\sim 30\mathrm{B}\).
5. Have a strong dense LLM, want bigger? → Sparse upcycle dense to MoE.
6. Want easy quantization + serving? → Dense. MoE quantization is harder.
7. Need experts specialized for domains? → Branch-Train-MiX or fine-grained MoE.
8. Vision / multimodal model? → Both viable; DeepSeek-VL2-style or CuMo / Aria if going MoE.

Quantization

Merged from "Decision Tree — "How to Quantize?"" — Quantization_SOTA_Updated.md.

1. Cloud serving with H100/B200 hardware? → FP8 (W8A8 + FP8 KV); TensorRT-LLM or vLLM.
2. Cloud serving on A100 / older? → \(\operatorname{INT8}\) (SmoothQuant) or \(\operatorname{W4A16}\) (AWQ).
3. Memory-tight / want max throughput? → \(\operatorname{W4A16}\) (AWQ + Marlin) + FP8 KV.
4. Single consumer GPU (24–80 GB)? → GGUF Q4 K M (llama.cpp) or AWQ via vLLM.
5. Fine-tuning on single GPU? → QLoRA (NF4 + LoRA via bitsandbytes / Unsloth).
6. Apple Silicon device? → MLX INT4 / INT8.
7. Mobile / NPU (Qualcomm, ANE, MediaTek)? → INT8 PTQ via OpenVINO / Core ML / QNN.
8. Diffusion model serving? → INT8 W & A; SVDQuant for FLUX 4-bit.
9. MoE serving on workstation? → ktransformers + GGUF Q4 + CPU offload.
10. Training a frontier model from scratch? → FP8 (Transformer Engine) or experimental FP4.

Pruning

Merged from "Decision Tree — "Which Pruning?"" — Pruning_SOTA_Updated.md.

1. LLM weight pruning, post-training? → SparseGPT or Wanda + 2:4 + FP8 + LoRA recovery.
2. LLM layer-level extreme compression? → ShortGPT or LLM-Streamline.
3. LLM hidden-dim compression? → SliceGPT.
4. LLM long-context decode (KV cache)? → H2O or Pyramid KV or SnapKV (prefill).
5. LLM streaming? → StreamingLLM (sinks + window).
6. ViT inference acceleration? → ToMe (drop-in, no training).
7. VLM inference acceleration? → FastV (drop vision tokens after layer K).
8. Diffusion image gen? → ToMe-SD + Block-cache + distill.
9. 3DGS deployment compression? → LightGaussian or CompGS.
10. Edge / mobile? → Combined: layer prune + 2:4 / unstructured prune + INT4 quant + distill.
11. Hardware-friendly speedup (Hopper / Blackwell)? → 2:4 sparsity + FP8 via TensorRT-LLM.
12. Adaptive per-input compute? → Mixture-of-Depths or Quest (KV) or DyDiT (diffusion).

Distillation

Merged from "Decision Tree — "Which Distillation Method?"" — Distillation_SOTA_Updated.md.

1. Closed-API teacher, no logits? → Sequence-KD on teacher samples (open-instruction pattern).

2. Same architecture, smaller / faster? → Hidden-state + attention + logit KD (TinyBERT, DistilBERT).

3. Cross-architecture (Transformer → SSM, ViT → CNN)? → Relational + sequence KD (RKD); avoid feature-matching.

4. Diffusion model, want few-step inference? → LCM (medium quality), DMD2 / Hyper-SD (top quality).

5. Want to bake in long-CoT reasoning? → R1-Distill pattern: SFT on filtered long-CoT traces.

6. Want to compress without losing diversity? → Higher temperature; mixup; reverse KL avoided.

7. Multi-domain, multi-teacher? → Per-input gating or weighted ensemble target.

8. Need both small & aligned? → Constitutional distillation (critique + revise + train).

9. Vision SSL student? → DINOv2-style self-distillation with EMA target.

10. Edge / mobile deployment? → Mobile-arch student (MobileViT, MobileSAM) + KD.

Parameter-Efficient Fine-Tuning (PEFT)

Merged from "Decision Tree — "Which PEFT?"" — PEFT_SOTA_Updated.md.

1. LLM SFT, single GPU? → QLoRA + Unsloth (consumer) or LoRA + FSDP.

2. LLM DPO / SimPO? → LoRA + TRL; rank 8–64; low LR.

3. LLM GRPO / R1-style RL? → LoRA + cold-start + OpenRLHF / verl.

4. Need full FT quality but single GPU? → DoRA or QLoRA at rank 256.

5. Diffusion personalization (subject)? → LoRA + DreamBooth pattern.

6. Diffusion style / concept? → LoRA / LyCORIS LoCon / LoHa / LoKr.

7. Diffusion identity-preserving? → OFT / BOFT.

8. Tiny adapter (storage-tight)? → VeRA / IA3 / BitFit.

9. Multi-tenant serving? → vLLM / S-LoRA / LoRAX with hot-swap.

10. NLU / ViT classification? → LoRA / VPT / Adapters.

11. Massive distribution shift? → Full fine-tune (PEFT may ceiling).

12. RL alignment with shared reference? → LoRA + base-as-reference (no memory doubling).

Scaling Laws

Merged from "Decision Tree — "How to Scale?"" — Scaling_Laws_SOTA_Updated.md.

1. Frontier-quality model from scratch? \(\to\) MoE + Chinchilla-overshoot + µP + extreme compute ($~10M+).

2. Best deploy economics? \(\to\) Dense small + extreme over-train (Llama 3 8B pattern).

3. Want reasoning? \(\to\) Pretrain + cold-start SFT + GRPO with verifiable rewards.

4. Cheap reasoning at small scale? \(\to\) Distill from R1-class (R1-Distill-Qwen-7B).

5. Long context? \(\to\) Linear attn (Mamba) or RoPE+YaRN + ring attention.

6. Image generation? \(\to\) DiT + flow matching + distillation (SD3 / FLUX pattern).

7. Video generation? \(\to\) Causal 3D VAE + spatiotemporal MM-DiT; lots of compute.

8. Vision encoder? \(\to\) DINOv3-style scale + gram-matching loss.

9. VLM? \(\to\) Native multimodal + scale + RL post-training.

10. Hyperparameter tuning at scale? \(\to\) µP + small-scale proxy sweep.

11. Inference-time compute? \(\to\) PRM + search + RL-trained long CoT.

12. Specialist model (medical / legal / code)? \(\to\) Distill + LoRA fine-tune from frontier base.

Part II — LLMs: Training, Reasoning, RL & Agents

Reasoning Technologies

Merged from "Decision Tree — "Which Reasoning Method?"" — Reasoning_Technologies_SOTA_Updated.md.

1. Is the answer programmatically verifiable? → Add verifier; consider RL fine-tuning with GRPO.
2. Is the task math / code / logic? → CoT + tool use (code interpreter / sympy); self-consistency.
3. Is the answer free-form text? → CoT + verifier with LLM-as-judge or constitutional check.
4. Multi-step with branching? → Tree-of-Thought or MCTS with PRM.
5. Multi-document / multi-hop? → ReAct + retrieval + sub-question decomposition.
6. Long-horizon agentic? → Plan-Act-Reflect with bounded budget + tool schema.
7. Visual? → Visual CoT; MCTS for hard problems; consider Vision-R1-style RL.
8. Real-time / latency-critical? → Distilled R1 model + CoT only; skip search.
9. Frontier accuracy needed? → Frontier reasoning model (o3 / R1 / Claude Opus) + MCTS + tools.
10. Safety-critical? → Multi-agent debate + external verification; never trust CoT alone.

Reward Functions

Merged from "Decision Tree for "What Reward Should I Use?"" — Reward_Functions_SOTA_Updated.md.

1. Is the goal naturally verifiable (math, code, simulator success)? \(\to\) Programmatic reward + GRPO / PPO. Skip RM.

2. Do you have human preferences, no programmatic check? \(\to\) Bradley-Terry RM + PPO-RLHF or DPO. Add KL anchor.

3. Do you have demonstrations but no preferences and no verifier? \(\to\) IRL / GAIL / AIRL or Behavior Cloning + RL fine-tune.

4. Sparse extrinsic reward, exploration is the problem? \(\to\) HER / curriculum / intrinsic motivation (RND, NovelD).

5. Multiple competing objectives? \(\to\) Constrained MDP + Lagrangian or lexicographic (if priorities are strict).

6. Robot task with hand-designed components is tedious? \(\to\) Eureka / DrEureka (LLM-designed reward search).

7. Diffusion model needs preference alignment? \(\to\) HPS/ImageReward + Diffusion-DPO.

8. Multimodal reasoning task? \(\to\) Composite verifiable: IoU / mask-IoU / EM + format, GRPO-style.

9. Safety-critical alignment? \(\to\) Constitutional AI + Rule-Based Rewards + KL anchor.

Agentic Intelligence

Merged from "Decision Tree — "Which Agent Pattern?"" — Agentic_Intelligence_SOTA_Updated.md.

1. Single tool call solves it? \(\to\) Function calling, no loop. Fastest, cheapest.
2. Sequential multi-step but predictable? \(\to\) Plan-and-Execute with explicit plan.
3. Variable structure, exploratory? \(\to\) ReAct loop.
4. Long horizon (\(>10\) steps)? \(\to\) PAR + reflection + bounded budget.
5. Specialized expertise needed? \(\to\) Multi-agent (manager-worker or society of mind).
6. Web / GUI tasks? \(\to\) Computer-use VLM + Set-of-Mark + sandboxed VM.
7. Code-heavy? \(\to\) Code agent (Aider / Cursor / Cline / Devin pattern).
8. Research / synthesis? \(\to\) Deep Research pattern (multi-source + synthesis + citations).
9. Persistent across sessions? \(\to\) LangGraph + memory + checkpointing.
10. Real-time conversation? \(\to\) Realtime voice API + minimal tool palette.

Prompt · Context · Harness · Graph Engineering & Self-Improving AI

Merged from "Decision Guide — "Which Technique?"" — Prompt_Context_Harness_Graph_Engineering_SOTA_Updated.md.

1. One-shot factual/format task? → Clear zero-shot instruction + structured outputs.
2. Multi-step reasoning on a non-reasoning model? → CoT + Self-Consistency.
3. Using a reasoning model? → Direct instructions; skip manual CoT; add a verifier for best-of-\(N\).
4. Have eval data and want a better prompt? → Compile with DSPy / ★ GEPA.
5. Answer needs private/fresh knowledge? → RAG (hybrid + rerank; ★ Contextual Retrieval).
6. Global/multi-hop questions over a corpus? → GraphRAG / HippoRAG.
7. Long-horizon agent losing the thread? → Compaction + note-taking + sub-agent isolation; temporal-KG memory (Zep).
8. Context too big / expensive? → LLMLingua compression + prompt caching.
9. Building an agent? → Simplest pattern that works; ReAct + MCP tools; add verification.
10. Need explicit/loopy control flow? → LangGraph; to learn the flow → AFlow/GPTSwarm.
11. Want the agent to improve at runtime? → Reflexion + a real verifier (never rely on intrinsic self-correction).
12. Want the model to improve offline? → STaR/ReST-EM on verifiable tasks; self-play (Absolute Zero) where a checker exists.
13. Want the scaffold to improve itself? → ADAS / ★ Darwin Gödel Machine, benchmark-gated + sandboxed.
14. Want reusable, shareable procedural capability with no fine-tuning? → Author an Agent Skill (crisp description, progressive disclosure, bundled scripts); ship it in a plugin. Let the agent grow its own library (Voyager) for weight-free self-improvement.

Test-Time & Training-Free Optimization

Merged from "Decision Guide — "Which Test-Time Method?"" — Test_Time_and_Training_Free_Optimization_SOTA_Updated.md.

1. Want the same outputs, faster? → Speculative decoding (EAGLE/self-spec/lookahead).
2. Want higher quality for free from a frozen LM? → Decoding contrast (DoLa/CAD) or, for images, guidance (CFG/PAG).
3. Want to change behavior with no data/training? → Activation steering (CAA/RepE).
4. Have multiple finetunes? → Merge (Task Arithmetic → TIES+DARE).
5. Hard problem + automatic checker? → Best-of-\(N\) / PRM search; allocate by difficulty (Snell).
6. Hard problem + no checker? → Self-consistency, or ★ TTRL to fold the vote into weights.
7. Reasoning model, want to dial effort? → ★ budget forcing (s1).
8. Distribution shift, unlabeled stream? → TTA: TENT → ★ SAR/EATA; update BN affine first.
9. Per-instance/task shift + an SSL task? → TTT (MAE-based); abstraction → ★ ARC per-task LoRA.
10. Long-context / streaming? → ★ TTT layers / Titans / ATLAS.
11. Align/steer outputs at inference, no retrain? → ★ TPO; VLM zero-shot → TPT/TDA.

Part III — Data & Evaluation

Data Collection & Curation

Merged from "Decision Tree — "How Do I Get Data?"" — Data_Collection_Curation_SOTA_Updated.md.

1. Pretraining LLM, frontier budget?
   → Common Crawl via FineWeb / DCLM pipeline + tier-1 sources + synthetic.
2. Pretraining LLM, modest budget?
   → FineWeb-Edu at small scale; FineWeb at larger.
3. SFT instruction-tuning?
   → Tulu / OpenHermes or Self-Instruct on top of frontier teacher.
4. RLHF preferences?
   → Iterative pairs from current model; label via Surge / in-house.
5. Reasoning RL?
   → NuminaMath + verifiable-reward problems (math, code).
6. Long-CoT distillation?
   → Sample from R1 / o3; filter by correct final answer.
7. Image generation training?
   → DataComp / DFN + re-caption with strong VLM.
8. Video generation training?
   → Panda-70M / HD-VG / InternVid + re-caption.
9. Robot foundation model?
   → Open-X-Embodiment + own teleop fleet.
10. Safety / alignment?
    → HH-RLHF / Constitutional + red-team gen + adversarial pairs.
11. Production deployment improvement?
    → Data-engine pattern: telemetry → label → retrain.

Metrics & Evaluations

Merged from "Decision Tree — "Which Metric / Benchmark?"" — Metrics_Evaluations_SOTA_Updated.md.

1. LLM general quality? → MMLU-Pro + Arena Elo + MT-Bench / AlpacaEval 2.

2. LLM frontier reasoning? → AIME / Putnam / FrontierMath / HLE / ARC-AGI.

3. LLM code? → LiveCodeBench + SWE-bench-Verified.

4. LLM safety? → HarmBench + JailbreakBench + XSTest + red-team.

5. Multimodal VLM? → MMMU-Pro + MathVista + Vision Arena.

6. Long-context? → RULER + LongBench + ∞-Bench Eval.

7. Image generation? → HPSv3 + VQAScore + Image Arena + FID for legacy.

8. Video generation? → VBench-2 + VideoScore + Video Arena + FVD.

9. Detection / segmentation? → COCO mAP / Cityscapes mIoU / PQ.

10. Agent task? → GAIA / WebArena / OSWorld / SWE-bench-Verified.

11. Robotics? → SimplerEnv / LIBERO / real-world success rate.

12. Production deployment? → A/B + interleave + guardrail drift metrics + continuous monitoring.

Part IV — Generative Models: Diffusion, Video, 3D & World

Video Generation

Merged from "Decision Tree — "Which Video Gen?"" — Video_Generation_SOTA_Updated.md.

1. Need premium creative quality, willing to pay? \(\to\) Sora 2 / Veo 3 / Kling 2 (closed APIs).
2. Self-hosted open frontier? \(\to\) Hunyuan Video or Wan 2.1/2.2.
3. Real-time interactive? \(\to\) LTX-Video or distilled Mochi.
4. Need native audio? \(\to\) Veo 3 (closed) or Wan 2.2 / MovieGen (open).
5. Image-to-video? \(\to\) SVD / Hunyuan I2V / Kling I2V.
6. Pose-driven character? \(\to\) AnimateAnyone / MimicMotion / Champ.
7. Audio-driven portrait? \(\to\) EMO / Live Portrait.
8. Camera control? \(\to\) Veo / Sora / Luma (closed) or CameraCtrl (open).
9. Long-form coherent (\(> 30\text{s}\))? \(\to\) Sora 2 / Veo 3 or chunked AR with anchor frames.
10. Action-conditioned (world model for robotics / AV)? \(\to\) Cosmos Predict or GAIA-2.
11. Custom style / character? \(\to\) LoRA on Hunyuan or Wan via ComfyUI.
12. Real-time playable? \(\to\) Genie 2 / Oasis / GameNGen (specialized).

3D & Multi-View Generation

Merged from "Decision Tree — "Which 3D Generation?"" — 3D_MultiView_Generation_SOTA_Updated.md.

1. Single image \(\to\) 3D asset, open-source? \(\to\) Trellis or Hunyuan3D-2.
2. Fastest open-source (< 1s)? \(\to\) TripoSR / SF3D / SPAR3D.
3. Text \(\to\) 3D? \(\to\) FLUX/SDXL \(\to\) Trellis / Hunyuan3D-2 (two-stage).
4. Need clean mesh topology (game asset)? \(\to\) MeshAnything V2 / MeshLRM / EdgeRunner.
5. Multi-view diffusion (4–6 views)? \(\to\) MVDream / Wonder3D / Zero123++ / SV3D.
6. Photogrammetry replacement? \(\to\) VGGT / MASt3R-SfM + 3DGS.
7. Scene scanning (phone)? \(\to\) Polycam / Luma AI + 3DGS.
8. Avatar (face)? \(\to\) Codec Avatars / Gaussian Avatars (premium) or Portrait3D (single image).
9. Custom art style, no 3D dataset? \(\to\) ProlificDreamer (VSD) / Magic3D (SDS-based).
10. Premium / commercial product? \(\to\) Rodin Gen-1.5 / Tripo / Meshy / CSM.
11. 4D / dynamic 3D? \(\to\) 4D-GS optimization / Animate3D / DreamScene4D.
12. Scene-level generation? \(\to\) World Labs (closed) / CityDreamer / Set-the-Scene (research).

World Models

Merged from "Decision Tree — "Which World Model?"" — World_Models_SOTA_Updated.md.

1. Classical RL benchmark, sample efficiency? \(\to\) Dreamer V3 / TD-MPC2 / IRIS.
2. Need diffusion-quality but Atari-scale data? \(\to\) DIAMOND.
3. Driving / AV closed-loop simulation? \(\to\) GAIA-2 / Cosmos + StreetGaussians.
4. Humanoid / general robotics? \(\to\) Cosmos + Isaac Lab + GR00T or 1X World Model.
5. General video as world simulator? \(\to\) Sora 2 / Veo 3 / Cosmos Predict (open).
6. Want action-conditioned open weights? \(\to\) Cosmos Predict (NVIDIA) or DriveDreamer-2.
7. Playable / game generation? \(\to\) Genie 2 (closed) or Oasis (open) or GameNGen.
8. Single-image \(\to\) explorable 3D world? \(\to\) World Labs (closed) or Genie 2.
9. Internet-video pretraining for embodied? \(\to\) V-JEPA 2 / Cosmos Predict.
10. Need physical commonsense filter / evaluator? \(\to\) Cosmos Reason VLM.

Part V — Neural Rendering & 3D Reconstruction

NeRF

Merged from "decision tree — which NeRF variant?" — NeRF_SOTA_Updated.md.

• Goal: photoreal NVS, room-scale. Zip-NeRF or Mip-NeRF 360.
• Goal: fastest NeRF training. Instant-NGP or Nerfacto.
• Goal: dense / sharp mesh. NeuralAngelo, BakedSDF, or NeuS / VolSDF.
• Goal: glossy / specular. Ref-NeRF or NeRO.
• Goal: relightable scene. NeRO, NeRD, NeRFactor, TensoIR.
• Goal: dynamic from multi-camera. K-Planes, D-NeRF, HyperNeRF.
• Goal: dynamic from monocular. NSFF / Robust Dynamic NeRF + DUSt3R/Marigold/CoTracker priors.
• Goal: avatars / humans. HumanNeRF / Vid2Avatar / NeRFace / RigNeRF.
• Goal: sparse-view feed-forward. PixelNeRF / MVSNeRF / IBRNet / pixelSplat (GS).
• Goal: text-to-3D. ProlificDreamer / Magic3D / Latent-NeRF / Trellis (GS).
• Goal: city-scale. Block-NeRF / Mega-NeRF / BungeeNeRF / NeuRAD.
• Goal: SLAM. NICE-SLAM / Co-SLAM (NeRF) or SplaTAM (GS).
• Goal: editing. Instruct-NeRF2NeRF / DreamEditor or LERF + SAM.
• Goal: on-device / mobile. Mobile-NeRF / MERF / BakedSDF.
• Goal: pose-free. BARF / GARF / NeRF-- / Dust3R-initialized Nerfacto.

Gaussian Splatting

Merged from "decision tree — which GS variant?" — Gaussian_Splatting_SOTA_Updated.md.

• Goal: photoreal NVS, room-scale, plenty of views. → Vanilla 3DGS or Mip-Splatting.
• Goal: surface mesh / geometry. → 2DGS, optionally + SuGaR / GOF.
• Goal: tiny model size. → LightGaussian + Scaffold-GS + SOG.
• Goal: city-scale / aerial. → VastGaussian / CityGaussian / Hierarchical 3DGS.
• Goal: dynamic from multi-camera rig. → Spacetime Gaussians / Dynamic 3DGS.
• Goal: dynamic from monocular phone video. → MoSca / Shape-of-Motion (with VGG-T / DUSt3R / Depth-Anything priors).
• Goal: SLAM / online mapping. → SplaTAM / Gaussian-SLAM / MonoGS.
• Goal: relightable / inverse-rendering. → Relightable 3DG / GS-IR / GShader.
• Goal: text or image to 3D. → Trellis / Hunyuan3D-2 / DreamGaussian / LGM.
• Goal: digital human / talking head. → GaussianAvatars / AnimatableGaussians / Codec Avatar 3D.
• Goal: AV closed-loop sim. → StreetGaussians / OmniRe / NeuRAD.
• Goal: physics simulation in the same scene. → PhysGaussian.
• Goal: editable scene. → 2DGS + SuGaR (mesh) or GaussianEditor (text-driven).
• Goal: pose-free / sparse-view. → InstantSplat / NoPoSplat / pixelSplat / MVSplat.
• Goal: ray-traced refraction / shadows. → 3D-GRT (NVIDIA Gaussian Ray Tracing).

Neural Rendering

Merged from "Decision Tree — "Which Neural Rendering?"" — Neural_Rendering_SOTA_Updated.md.

1. Real-time render needed? → 3D Gaussian Splatting (Mip-Splatting variant).
2. Best photometric quality, render speed less critical? → Mip-NeRF 360 / Zip-NeRF.
3. Few-second training, decent quality? → Instant-NGP.
4. Single image to 3D asset? → Trellis / Hunyuan3D-2 (native 3D diffusion).
5. Multi-image to 3D, no SfM available? → VGGT / MASt3R-SfM (feed-forward).
6. Dynamic scene (people, fluids)? → 4D-GS / Deformable 3DGS.
7. Need mesh extraction (graphics pipeline)? → SuGaR / Gaussian Frosting or 2D-GS.
8. Relighting needed? → Relightable 3D Gaussians / GS-IR (early).
9. Avatar (face)? → Gaussian Avatars / Codec Avatars.
10. City / kilometer scale? → CityGaussian / Hierarchical 3DGS.
11. AV closed-loop simulation? → Cosmos / EmerNeRF / OmniRe.
12. Edit a captured scene with text? → GaussianEditor / Instruct-NeRF2NeRF.

Structure from Motion

Merged from "Decision Tree — "Which SfM?"" — Structure_from_Motion_SOTA_Updated.md.

1. Highest-precision photogrammetry? \(\to\) Reality Capture or Metashape.
2. Open-source max precision? \(\to\) COLMAP + HLoc + LightGlue + MAGSAC++.
3. Fast SfM (seconds)? \(\to\) VGGT or MASt3R-SfM.
4. 3DGS pipeline init? \(\to\) VGGT (replacing COLMAP).
5. Phone capture? \(\to\) Polycam / Luma AI / Scaniverse / RealityScan.
6. Real-time SLAM? \(\to\) ORB-SLAM3 (no IMU) or VINS-Fusion (with IMU).
7. Photoreal map needed? \(\to\) MonoGS / SplaTAM (3DGS-SLAM).
8. Aerial / drone survey? \(\to\) Pix4D / Metashape + RTK.
9. AR persistent localization? \(\to\) Niantic Lightship / ARKit Cloud Anchors.
10. Long-term loc (day-night)? \(\to\) AnyLoc + LightGlue for illum-robust.
11. Visual-inertial? \(\to\) VINS-Fusion / OKVIS-2 / Kimera.
12. VFX camera tracking? \(\to\) PFTrack / SynthEyes / Reality Capture.

Delighting & Relighting

Merged from "Decision Tree — "Which Relighting Method?"" — Delighting_Relighting_SOTA_Updated.md.

1. Single-image portrait, fast result? \(\to\) SwitchLight (commercial) or IC-Light (open).
2. Composite portrait into a new background? \(\to\) Relightful Harmonization + IC-Light FBC.
3. Single-image object, e-commerce? \(\to\) IC-Light text-conditioned or PBR re-render via IntrinsicAnything.
4. Single-image SVBRDF estimation? \(\to\) Deschaintre / \(\mathbf{RGB}{\to}\mathbf{X}\) / IntrinsicAnything.
5. Multi-view scene with arbitrary lighting? \(\to\) Relightable 3D Gaussians / NeRO / GS-IR.
6. Premium digital human (film)? \(\to\) Light stage scan + PBR engine.
7. Photogrammetry asset for game? \(\to\) Cross-polarized capture + bake to PBR maps (Quixel pattern).
8. AR insertion of virtual object? \(\to\) ARKit / ARCore env probe + IBL in RealityKit / Unity.
9. VFX virtual production? \(\to\) LED volume (Stagecraft) + Unreal.
10. Live video relighting? \(\to\) Distilled IC-Light or temporal-consistent diffusion (research).

Photorealistic Avatars

Merged from "Decision Tree — "Which Avatar Tech?"" — Photorealistic_Avatars_SOTA_Updated.md.

1. Premium digital human, film? \(\to\) Light stage capture + offline render (ICT / Meta Sociopticon-class).
2. Telepresence on Vision Pro? \(\to\) Apple Persona.
3. Telepresence on Quest 3? \(\to\) Codec Avatars 3.0.
4. Real-time monocular avatar? \(\to\) FlashAvatar or GaussianAvatars.
5. Audio-driven portrait video (TikTok-style)? \(\to\) EMO or Live Portrait.
6. Audio-driven full body? \(\to\) Audio2Photoreal.
7. Pose video drives character? \(\to\) AnimateAnyone / MimicMotion / Champ.
8. Single image \(\to\) animatable 3D head? \(\to\) Portrait3D or LiveHead.
9. Identity preservation, zero-shot? \(\to\) InstantID / PhotoMaker / PuLID.
10. Premium identity, willing to fine-tune? \(\to\) DreamBooth-LoRA on subject.
11. Stylized / cartoon avatar? \(\to\) Trellis / Hunyuan3D-2 character mode.
12. Mobile / web deployment? \(\to\) LightGaussian + INT8 on 3DGS avatar.
13. Synthetic / generative avatar? \(\to\) Native 3D diffusion (Trellis) or 2D-only (StyleGAN-class).

Part VI — Core Vision, Robotics & Autonomy

Vision-Language-Action (VLA) Models

Merged from "Decision Tree — "Which VLA?"" — VLA_Models_SOTA_Updated.md.

1. Research baseline / academic project? → OpenVLA (open + reproducible).
2. Want diffusion policy specifically? → Octo (small) or RDT-1B/2B (frontier).
3. Production manipulation, commercial? → \(\pi_0\) / \(\pi_{0.5}\) (Physical Intelligence).
4. Humanoid robot, NVIDIA stack? → GR00T N1 / N2 + Cosmos + Isaac Lab.
5. Industrial humanoid, on-board? → Helix (Figure, closed) or build S2/S1 with π0/GR00T.
6. Bimanual table-top demos? → ACT or Diffusion Policy on ALOHA.
7. Need 3D / spatial reasoning? → SpatialVLM fine-tune or 3D-VLA with point-cloud branch.
8. Cross-embodiment scaling? → Open-X-Embodiment + RT-X pattern.
9. RL fine-tuning? → Q-chunking / Residual RL / Diffusion DPO / GRPO on pretrained VLA.
10. Edge / on-robot deployment? → Distill + INT8 + Jetson Thor.

Autonomous Driving

Merged from "Decision Tree — "Which AV Component?"" — Autonomous_Driving_SOTA_Updated.md.

1. Camera-only 3D detection? → StreamPETR or SparseBEV (+ RayDN).
2. LiDAR-only 3D detection? → PointPillars (production) or DSVT / SAFDNet (SOTA).
3. Sensor fusion? → BEVFusion / TransFusion / CMT / IS-Fusion.
4. Free-space + general obstacles? → Occupancy network (FB-OCC / SparseOcc / GaussianFormer-2).
5. Online HD mapping? → MapTRv2 / StreamMapNet (+ SD-map priors).
6. Multi-object tracking? → ByteTrack / OC-SORT / TransTrack.
7. Motion prediction? → Wayformer / MTR++ / QCNet / Diffusion Planner.
8. End-to-end driving (research)? → UniAD / VAD / Hydra-MDP / DiffusionDrive / GoalFlow.
9. VLM / VLA reasoning? → LINGO-2 / DriveVLM / Senna / EMMA / AutoVLA.
10. World model for closed-loop? → GAIA-2 / Cosmos Predict.
11. Neural reconstruction (sim re-render)? → StreetGaussians / EmerNeRF / OmniRe / HUGSIM.
12. Closed-loop benchmark? → Bench2Drive / NAVSIM(v2) / HUGSIM / Waymax.
13. On-vehicle compute platform? → NVIDIA Thor or Tesla AI5 or Mobileye EyeQ Ultra.

Files without a decision-tree section

These sheets have no standalone decision tree (their selection guidance lives inline or in comparison tables / cheat cards instead): Attention, CV Principal Deep Dive, CV Principal Math, Diffusion Derivations, Diffusion Models, Foundation Models, Policy Optimization, RL Training Strategies & Recipes, Tokenization & Context (v1 & v2).