The X-Former Catalogue — Every Notable Transformer Variant

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

July 2026 · Updated Edition

Contents

1. Anatomy of a Transformer (Quick Reference)
2. Encoder-Only Transformers (BERT Family)
3. Decoder-Only Transformers (GPT / LLaMA Family)
4. Encoder-Decoder Transformers (T5 / BART Family)
5. Vision Transformers
6. Efficient / Long-Context X-Formers
7. Recurrent / SSM Successors to Transformers
8. Mixture-of-Experts Transformers
9. Multimodal: Dual-Encoder (CLIP Family)
10. Multimodal: VLM (LLaVA / BLIP / Flamingo Family)
11. Multimodal: Native Early-Fusion (Chameleon Family)
12. Detection / Segmentation Transformers
13. Diffusion Transformers
14. Video Transformers
15. Audio Transformers
16. Code Transformers
17. Robotics / VLA Transformers
18. 3D Vision Transformers
19. Specialized Transformers
20. Foundation 3D / Geometric Transformers
21. The 2026 Production Stack: Cross-Modal Summary
22. Architectural Decisions: Quick Decision Tree

• Appendix A: Pictures Worth Memorizing
• Appendix B: Year-by-year shortlist

1. Anatomy of a Transformer (Quick Reference)

1.1 The original (Vaswani et al. 2017)

Encoder-decoder for English-to-German translation. Stacked self-attention + feedforward, sinusoidal positional encoding, post-norm residuals.

\[\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V, \quad y = \mathrm{LN}(x + f(x)).\]

Three architectural archetypes grew from this: encoder-only (bidirectional, classification / embeddings), decoder-only (autoregressive, generation), encoder-decoder (sequence-to-sequence).

1.2 The modern Transformer block (2026 default)

• Pre-norm residual: \(y = x + f(\mathrm{LN}(x))\).
• RMSNorm instead of LayerNorm.
• RoPE positional encoding (1D for text, 2D for vision, 3D for video).
• GQA / MQA / MLA for inference KV cache.
• SwiGLU MLP (gated activation).
• FlashAttention 2/3 kernels.
• BF16 weights, FP8 forward where supported.

1.3 The genealogy in one tree

Where the families branched

Original Transformer (2017) branches into:

• Encoder-only: BERT → RoBERTa → DeBERTa → ModernBERT.
• Decoder-only: GPT-2 → GPT-3 → LLaMA → Mistral / Mixtral → DeepSeek / Qwen / Gemma.
• Encoder-decoder: T5 → FLAN-T5 / mT5 / UL2; BART → Pegasus.
• Vision: ViT → DeiT → Swin → ConvNeXt; DINO → MAE → JEPA.
• Multimodal: CLIP → Flamingo → BLIP-2 → LLaVA → Chameleon / GPT-4o.
• Generative diffusion: DiT → MM-DiT (SD3, FLUX) → MM-DiT (Sora, Veo, MovieGen).

2. Encoder-Only Transformers (BERT Family)

2.1 BERT (2018)

Bidirectional encoder, trained with masked-language-model (MLM) + next-sentence-prediction (NSP). 12 (Base) or 24 (Large) layers, \(d = 768\) or \(1024\).

Loss:

\[\mathcal{L}_{\mathrm{MLM}} = -\sum_{i\in\mathcal{M}} \log P(x_i \mid x_{\backslash\mathcal{M}})\]

mask \(\sim 15\%\) of tokens; replace 80% with [MASK], 10% random, 10% unchanged.

2.2 RoBERTa (2019)

BERT with a clean recipe: drop NSP, larger batches, longer training, dynamic masking. Strictly better than BERT at every benchmark.

2.3 ALBERT (2019)

Parameter-efficient BERT:

• Factorized embedding \(V\times d \to V\times e + e\times d,\ e\ll d\).
• Cross-layer parameter sharing (one Transformer block applied \(L\) times).
• Sentence-Order Prediction (SOP) instead of NSP.

2.4 ELECTRA (2020)

Replace MLM with replaced-token detection: a small generator proposes plausible replacements, a discriminator predicts which tokens were replaced. Sample-efficient: every token contributes loss, not just the 15% masked.

2.5 DeBERTa / DeBERTa-V3 (2021–22)

Disentangled attention: separate content and position embeddings, attention computed as

\[A_{ij} = q_i^c\,{k_j^c}^\top + q_i^c\,{k_{i-j}^r}^\top + k_j^c\,{q_{j-i}^r}^\top\]

Plus enhanced mask decoder. DeBERTa-V3 swaps MLM for ELECTRA-style. State of the art for many NLU benchmarks.

2.6 ModernBERT (2024)

A 2024 redesign of BERT-Base with all the modern tricks: RoPE, GeGLU, alternating local + global attention, FlashAttention, \(\sim 8\mathrm{K}\) context. Replaces BERT for 2025–2026 retrieval / classification deployments.

2.7 Specialized encoder-only

• XLM, XLM-R: multilingual MLM; XLM-R-Large is the standard multilingual encoder.
• SciBERT, BioBERT, ClinicalBERT, FinBERT: domain-specific.
• SpanBERT: mask contiguous spans rather than random tokens.
• Longformer / BigBird (encoder): sliding-window + global tokens for \(\sim 32\mathrm{K}\) context.
• Sentence-BERT, GTE, BGE, E5: encoder + pooling for sentence embeddings.

3. Decoder-Only Transformers (GPT / LLaMA Family)

3.1 GPT-2 (2019)

First decisive autoregressive Transformer at scale \((\sim 1.5\mathrm{B})\). Pre-norm with LayerNorm, learned absolute position, GeLU.

3.2 GPT-3 (2020)

175B parameters. Demonstrated few-shot in-context learning emerges with scale. No architectural surprise vs GPT-2; scale was the lesson.

3.3 LLaMA (2023) and successors

LLaMA 1 introduced the open recipe that everyone copied:

• Pre-norm with RMSNorm.
• RoPE positional encoding.
• SwiGLU MLP.
• AdamW, cosine schedule.

LLaMA 2 (2023): grouped-query attention (GQA), 4K context. LLaMA 3 / 3.1 (2024): 128K context (RoPE-scaled), 128k vocab. LLaMA 4 (2025): native multimodal in some sizes, MoE variants.

3.4 Mistral and Mixtral (2023–24)

Mistral 7B: GQA + sliding-window attention + "rolling buffer" KV cache. Punched well above weight class.

Mixtral \(8\times 7\mathrm{B}\ /\ 8\times 22\mathrm{B}\): sparse MoE; 8 experts per layer, top-2 routing.

3.5 Qwen, Gemma, Yi, DeepSeek, Falcon, MPT

• Qwen 1/1.5/2/2.5/3: Chinese-strong open models, multilingual data, scale up to 72B/110B + MoE variants. Qwen 3 is the 2025 frontier-open default.
• Gemma / Gemma 2 / Gemma 3: Google's open with logit-soft-cap, knowledge distillation from Gemini.
• DeepSeek-V2 / V3: MLA (Multi-Latent Attention) + DeepSeekMoE; V3 is 671B-MoE / 37B-active.
• Yi: Chinese-led, 6B/9B/34B; strong English/Chinese balance.
• Falcon: TII's open releases (7B / 40B / 180B).
• MPT, Pythia, OLMo: fully-open including data and training code; useful for research.

3.6 Reasoning models

• o1 / o3 / o4 (OpenAI): RL-trained for long visible / hidden CoT.
• DeepSeek-R1 / R1-Zero: open pure-RL reasoning model; R1-Zero learned reasoning emerging from RL alone, no SFT.
• QwQ / Qwen2.5-Reasoner / Qwen3-Reasoner: Alibaba's reasoning line.
• Marco-o1, Skywork-o1, Mulberry, LLaVA-CoT: o1-style replications.

Key

The 2026 frontier-open decoder-only stack: RMSNorm + RoPE (with NTK/YaRN) + GQA (or MLA) + SwiGLU + FlashAttention 3, often + MoE. All variants share this skeleton; differences are in (a) attention scheme, (b) MoE routing, (c) data, (d) post-training (RLHF / DPO / GRPO).

★ 2026 SOTA update

• Llama 4 (Scout / Maverick): Meta's first natively-multimodal, MoE Llama (early-fusion). Scout = 17B active / 16 experts, up to 10M-token context, fits one H100; Maverick = 17B active / 128 experts. Released Apr 5, 2025.
• Qwen3: Alibaba frontier-open family, dense + MoE from 0.6B to 235B-A22B, unified thinking / non-thinking modes, 119 languages, Apache-2.0. The 2025 frontier-open default.
• Gemma 3: Google open 1B-27B, now multimodal; decoder + GQA + QK-norm + interleaved local/global attention for 128K context, frozen SigLIP vision path (256 image tokens), distilled from Gemini.
• gpt-oss-120b & gpt-oss-20b: OpenAI's first open-weight (Apache-2.0) reasoning models; MoE, post-trained like o4-mini with SFT + high-compute RL, CoT + tool use. gpt-oss-120b approaches o4-mini accuracy.
• DeepSeek-R1: DeepSeek open reasoning model (Jan 2025); R1-Zero shows chain-of-thought reasoning emerging from pure RL (GRPO) with no SFT, distilled into dense Qwen/Llama variants. Already named in the catalogue; adding the primary reference.

4. Encoder-Decoder Transformers (T5 / BART Family)

4.1 T5 (2019)

Text-to-text unification: every NLP task framed as sequence-to-sequence. Pretrained with span-corruption MLM (sentinel tokens):

input: ⟨cloze⟩ \(X\) mask span by sentinel; target: ⟨sentinel⟩ \(X\) span.

Sizes: Small (60M) \(\to\) XXL (11B). Single architecture for translation, summarization, QA, classification, etc.

4.2 mT5, FLAN-T5, T5X

mT5: multilingual; FLAN-T5: instruction-tuned on 1800+ tasks; T5X: JAX/TPU codebase.

4.3 BART (2019)

Encoder-decoder with denoising autoencoder pretraining: noise the input (mask, delete, permute, infill), reconstruct the original. Strong for summarization, dialog, controllable generation.

4.4 Pegasus (2020)

BART variant with gap-sentence generation: mask important sentences (selected by ROUGE) and predict them. Designed for summarization.

4.5 UL2 (2022)

Mixture-of-denoisers pretraining: combines short / long span MLM with prefix-LM and standard LM objectives via task tokens. Single model good at both understanding and generation.

4.6 Switch Transformer (2021)

Encoder-decoder sparse MoE; one expert per token (top-1 routing) with load-balancing loss. Demonstrated MoE could match dense quality at \(\sim 1/7\) FLOPs.

5. Vision Transformers

5.1 ViT (2021)

Image \(\to N\) patches of size \(p\times p \to\) linear embed \(\to\) standard Transformer encoder. [CLS] token for classification.

\[z_i = W_p\,\mathrm{vec}(X_{\mathrm{patch}_i}) + PE_i, \quad W_p \in \mathbb{R}^{d\times 3p^2}.\]

Sizes: B/16, L/16, H/14. Needs huge data (JFT-300M) to match conv-net accuracy at smaller scale.

5.2 DeiT (2021)

Data-efficient ViT trained on ImageNet-1k with strong augmentation + a distillation token that learns from a CNN teacher. Removed ViT's data dependency.

5.3 Swin / Swin V2 (2021–22)

Hierarchical with shifted-window attention: partition image into \(M\times M\) windows, attention only within each, shift by \(M/2\) in alternating layers. Patch-merging halves resolution per stage. \(O(NM^2)\) vs ViT's \(O(N^2)\).

V2: log-spaced relative position bias, post-norm, scaled cosine attention \(\to\) stable to ViT-3B at \(1536^2\).

5.4 MViT, MViTv2, Hiera

MViT: hierarchical pyramid via pooled attention (stride-2 pooling on \(K, V\)). Hiera: pure ViT with simple pooling between stages; argues mask-pretraining alone yields locality without window tricks.

5.5 ConvNeXt and ConvNeXt V2

"CNN strikes back". Take ResNet-50, sequentially apply every modernization Swin assumed (large depthwise kernels, inverted bottleneck, GELU, fewer norms, LayerNorm). Matches Swin at same FLOPs. V2 adds FCMAE pretraining + Global Response Normalization (GRN).

5.6 CoAtNet, MaxViT

Hybrid conv + attention. CoAtNet: conv stages then Transformer stages. MaxViT: per layer, do block attention (local) + grid attention (dilated global).

5.7 Mobile / efficient

MobileViT, EfficientFormer, FastViT, RepViT: ViT ↔︎ MobileNet hybrids; structural reparameterization for fast inference.

5.8 ViT-22B and scaling

Largest publicly described ViT (Google). Architectural changes for stable training at 22B: parallel attention + MLP, QK-LN (LayerNorm on Q, K before attention), no biases. Strong evidence ViTs scale cleanly when stability is engineered.

5.9 DINO, DINOv2, DINOv3 (self-supervised)

ViT trained with self-distillation:

\[\mathcal{L} = -\mathrm{softmax}\!\left(\frac{g_\xi(v) - c}{\tau_t}\right)^\top \log\mathrm{softmax}\big(g_\theta(v')/\tau_s\big),\]

with EMA teacher \(\xi\). DINOv2: + iBOT mask prediction + KoLeo regularizer.

DINOv3: 7B params + gram-matching loss to retain dense quality at scale.

5.10 MAE, SimMIM, BEiT, EVA, FCMAE

Masked Image Modeling family. MAE masks 75% patches and reconstructs in pixel space. SimMIM: simpler with raw pixels; BEiT: predict discrete VQ tokens; EVA: predict frozen CLIP features; FCMAE: ConvNeXt-friendly with sparse convs.

5.11 V-JEPA, V-JEPA 2

Predict in latent space (EMA target) instead of pixels. V-JEPA 2 scales to billion frames. Yann LeCun's bet.

5.12 Other notable visions

PVT, Twins, CrossViT, CaiT, T2T-ViT, LeViT, BeiT v3.

★ 2026 SOTA update

• DINOv3: Meta self-supervised ViT scaled to 7B params on 1.7B curated images; introduces Gram anchoring (constrain feature Gram matrix to a stable checkpoint) to stop dense-feature degradation at scale.

6. Efficient / Long-Context X-Formers

6.1 Longformer, BigBird (2020)

Sliding window + dilated + global tokens. \(O(n)\) attention. Extended BERT to 16K context for document tasks.

6.2 Reformer (2020)

LSH attention (bucket similar Q,K) + reversible residuals. \(O(n \log n)\), very deep models with constant activation memory.

6.3 Linformer (2020)

Project K, V along sequence dim to fixed length \(k\). \(O(nk)\).

6.4 Performer (2020)

Random-feature approximation of softmax kernel. Linear time + memory. Unbiased estimator; quality gap at long sequences.

6.5 Nyströmformer, Synthesizer, Routing Transformer

Various sub-quadratic approximations. None survived into 2026 production deployment in major models.

6.6 TransformerXL, Compressive Transformer

Recurrence over segments: cache last segment's hidden states, attend to them in current segment. Compressive Transformer adds a "compressed memory" tier of pooled past states.

6.7 XLNet

Permutation language modeling: train on factorizations of token order; combines BERT bidirectionality with autoregressive objective.

6.8 LongNet

Dilated attention with exponentially growing dilation rates. Theoretical 1B context; less proven in practice.

6.9 TNT (Transformer-iN-Transformer)

Hierarchical: outer Transformer over patches, inner Transformer over sub-patches within each patch. Stronger fine-grained vision.

★ 2026 SOTA update

• DeepSeek-V3.2 (V3.2-Exp): Adds DeepSeek Sparse Attention (DSA) on top of MLA: a lightning indexer + fine-grained token selection cut attention from \(O(L^2)\) to \(O(Lk)\) for long-context efficiency at near-parity quality. V3.2-Exp released Sep 29, 2025.

7. Recurrent / SSM Successors to Transformers

7.1 Mamba and Mamba-2

Selective SSM with input-dependent state-transition matrices, hardware-aware parallel scan:

\[h_t = \bar{A}(x_t)h_{t-1} + \bar{B}(x_t)x_t, \quad y_t = C(x_t)h_t.\]

Linear time, constant per-token state. Mamba-2 reformulates as structured state-space duality (SSD) connecting SSMs to attention.

7.2 Vision Mamba, VMamba

Mamba for vision: scan in 2D (multiple directions). Competitive with ViT at high resolution and long video.

7.3 RWKV (1→5→6→7)

"RNN with Transformer-level performance." Linear-attention recurrence with token-shift mixing. Trains in parallel like Transformer, runs as RNN at inference. Versions 5–7 (RWKV-Eagle, RWKV-Finch, Goose) close the gap with Mamba.

7.4 RetNet

Multi-scale retention with parallel + recurrent + chunkwise dual forms:

\[S_t = \gamma S_{t-1} + K_t^\top v_t, \quad O_t = Q_t S_t.\]

\(\gamma\) fixed decay. Linear inference, parallel training.

7.5 Hyena, Striped Hyena

Long convolutions parameterized implicitly via small networks, combined with element-wise gating. Sub-quadratic in sequence length; competitive at smaller scales.

7.6 HGRN / xLSTM / Griffin / Recurrent Gemma

Renaissance of gated recurrence. xLSTM: sLSTM (scalar gating) and mLSTM (matrix memory). Griffin: gated linear recurrence + local attention. Recurrent Gemma builds on Griffin.

★ 2026 SOTA update

• Kimi Linear: Hybrid linear-attention architecture (48B total / 3B active) built on Kimi Delta Attention (KDA, a gated DeltaNet with DPLR transitions) layered with MLA; first to beat full attention across short/long-context and RL regimes.

8. Mixture-of-Experts Transformers

8.1 The MoE block

Replace MLP with \(N\) expert MLPs and a router:

\[y = \sum_{i\in\mathrm{Top}\text{-}k(g(x))} g_i(x)\cdot \mathrm{Expert}_i(x),\]

where \(g(x) = \mathrm{softmax}(W_g x)\) is the per-token routing distribution. Active params per token = \(k\,d_{\mathrm{expert}}\); total params \(\sim N\) times that.

8.1.1 Auxiliary losses

• Load-balancing: \(\mathcal{L}_{lb} = N\sum_i f_i p_i\), \(f_i\) = fraction routed to expert \(i\), \(p_i\) = mean router weight.
• Z-loss / router z-loss: penalize router logit norm to keep stable.

8.2 GShard, Switch (2020–21)

GShard: top-2 routing across thousands of experts on TPU. Switch Transformer: top-1 (one expert) for simplicity and speed.

8.3 Mixtral 8×7B / 8×22B

Sparse MoE on Mistral backbone: 8 experts per layer, top-2 routing. Active params \(\sim 12\mathrm{B}\) for \(8\times 7\mathrm{B}\); total 46B. Strong open MoE baseline.

8.4 DeepSeekMoE / DeepSeek-V2 / V3

Fine-grained experts (many small) + shared experts (always-on). V2: 236B total / 21B active. V3: 671B / 37B active. Combined with MLA \(\to\) frontier-open performance.

8.5 DBRX, Grok-1, Snowflake Arctic, Qwen MoE

Various open MoE releases. DBRX (Databricks): 132B / 36B active. Grok-1: 314B / 86B active. Arctic: 480B / 17B active. Qwen-2 MoE: 57B / 14B active.

8.6 Soft MoE, Expert Choice

Alternative routing: Expert Choice (each expert picks tokens, not vice versa) provides automatic load balancing. Soft MoE: each expert receives a learnable weighted average of all tokens.

★ 2026 SOTA update

• GLM-4.5: Z.ai/Zhipu Agentic-Reasoning-Coding (ARC) MoE, 355B total / 32B active (+ GLM-4.5-Air 106B), hybrid thinking + direct modes, trained on 23T tokens with expert-model iteration.
• Kimi K2: Moonshot AI trillion-scale open MoE: 1.04T total / 32B active, trained with the MuonClip optimizer (Muon + QK-Clip) on 15.5T tokens; SOTA open agentic / non-thinking model.

9. Multimodal: Dual-Encoder (CLIP Family)

9.1 CLIP (2021)

Two encoders (image ViT, text Transformer) trained with symmetric InfoNCE:

\[\mathcal{L} = -\frac{1}{2B}\sum_i\left[\log\frac{e^{u_i^\top t_i/\tau}}{\sum_j e^{u_i^\top t_j/\tau}} + \log\frac{e^{u_i^\top t_i/\tau}}{\sum_j e^{u_j^\top t_i/\tau}}\right].\]

Zero-shot classification by computing cosine of image with text embeddings of class names.

9.2 ALIGN (2021)

Same idea, much larger noisy web data (1.8B pairs). Argued data quality matters less than scale.

9.3 OpenCLIP, MetaCLIP, DFN, EVA-CLIP

Open replications and improvements; MetaCLIP balances metadata, DFN learns a small filter network for data curation.

9.4 SigLIP / SigLIP 2

Per-pair sigmoid loss instead of softmax over batch:

\[\mathcal{L} = -\frac{1}{B}\sum_{i,j}\log\sigma\big(z_{ij}(t\,u_i^\top v_j + b)\big).\]

No batch-wide normalization \(\to\) scales to small or huge batch and is sample-efficient. SigLIP 2 adds captioning + self-distillation auxiliaries.

9.5 LiT, FLIP, BeiT-3

LiT: lock the image encoder, train only the text encoder. FLIP: mask 50% of image tokens during contrastive training \(\to\) massive speedup. BeiT-3: a single Transformer with modality-specific experts, contrastive + denoising.

10. Multimodal: VLM (LLaVA / BLIP / Flamingo Family)

10.1 Flamingo (2022)

Frozen LLM + trainable Perceiver Resampler + gated cross-attention layers inserted into the LLM. Tanh gate initialized to 0 so LM is undisturbed at start.

10.2 BLIP / BLIP-2

BLIP-2: frozen image encoder \(\to\) Q-Former (\(\sim 32\) learnable queries cross-attend over image features) \(\to\) frozen LLM. Two-stage pretraining (vision-language alignment, then generation).

10.3 LLaVA / LLaVA-1.5 / LLaVA-NeXT / LLaVA-OneVision

Minimalist: CLIP features \(\to\) 2-layer MLP projector \(\to\) LLM prompt prefix. LLaVA-NeXT adds AnyRes tiling for high-res. LLaVA-OneVision unifies image, multi-image, and video.

10.4 InstructBLIP, MiniGPT-4, mPLUG-Owl, IDEFICS, KOSMOS

Variations on the BLIP-2 / LLaVA pattern with different projectors, training data, and instruction-tuning recipes.

10.5 Qwen-VL / Qwen2-VL / Qwen2.5-VL / Qwen3-VL

Native dynamic resolution via 2D-RoPE on patch positions; pixel unshuffle for token compression; hourly video. Qwen2.5-VL is a 2025 open frontier VLM.

10.6 InternVL / 2 / 2.5 / 3

Tiling for high-res; very large vision encoder (InternViT-6B). InternVL3 with \(\sim 78\mathrm{B}\) LLM.

10.7 CogVLM, MiniCPM-V, Molmo, PixMo, MGM, GLM-4V

Open VLMs with various architectural twists. Molmo notable for fully-open captioned data (PixMo).

10.8 Frontier closed VLMs

GPT-4V / GPT-4o / GPT-5; Gemini 1.5 / 2.0 / 2.5 (native multimodal, very long context); Claude 3.5 Sonnet / 4 / Opus 4.6 (vision); Grok-Vision.

11. Multimodal: Native Early-Fusion (Chameleon Family)

11.1 Chameleon (Meta, 2024)

Tokenize images into VQ tokens; single autoregressive Transformer trained on interleaved text + image tokens. Same vocabulary, same loss. No bolted-on adapter.

11.2 Show-o, Transfusion, Janus, Janus-Pro, Emu3

• Show-o: text AR + image discrete diffusion (parallel masking), one backbone.
• Transfusion: text AR + image continuous diffusion, one backbone with separate loss heads.
• Janus / Janus-Pro: decoupled encoders for understanding vs generation, shared LLM backbone.
• Emu3: pure next-token prediction over text + image + video tokens; matches diffusion on text-to-image.

11.3 OFA, Unified-IO, Unified-IO 2

Encoder-decoder unified-vocabulary models predating Chameleon. Unified-IO 2 adds audio.

11.4 GIT, PaLI, PaLI-X, PaLI-3

Google's generative image-text Transformers with strong captioning + VQA. PaLI-X: 55B; PaLI-3: 5B with SigLIP.

11.5 KOSMOS-1/2/3

Microsoft's interleaved multimodal models with grounded perception (KOSMOS-2 outputs bounding boxes as tokens).

11.6 Native multimodal, 2026

GPT-4o, Gemini 2.5, Claude Opus 4.6 (proprietary) and Chameleon, Show-o, Transfusion, Emu3, Janus-Pro (open) demonstrate the early-fusion trend across both ecosystems. The principal-level prediction: by 2027 most CV stacks default to a native multimodal backbone.

12. Detection / Segmentation Transformers

12.1 DETR (2020)

Encoder-decoder Transformer: image features (encoder) + N object queries (decoder); Hungarian matching in loss eliminates anchors and NMS:

\[\mathcal{C}_{ij} = -\hat{p}_i(c_j) + \lambda_1\|b_j - \hat{b}_i\|_1 + \lambda_g\,(1 - \mathrm{GIoU}).\]

12.2 Deformable DETR

Each query attends to a small set of \(K\) predicted offsets per feature level; sparse, scale-aware. \(10\times\) faster convergence than DETR.

12.3 Conditional / DAB / DN / DINO-DETR

• Conditional DETR: positional cross-attention conditioned on content.
• DAB-DETR: queries are 4D anchor boxes (dynamic, refined per layer).
• DN-DETR: feed noised GT boxes during training to stabilize matching.
• DINO-DETR: combines DAB + DN + contrastive denoising; SOTA on COCO.

12.4 Co-DETR, Group DETR

Auxiliary one-to-many heads during training give richer positive signal; deactivate at inference. Strong recipe for COCO leaderboard.

12.5 RT-DETR / RT-DETRv2 / RT-DETRv3 / D-FINE / DEIM

Real-time DETR variants. By 2025 they outperform YOLOv9/v10/v11 in many benchmarks at similar throughput.

12.6 Mask2Former, MaskFormer, OneFormer, kMaX-DeepLab

Universal mask paradigm: predict N binary masks + class probs:

\[P(c\mid x) = \sum_i p_i(c)\,m_i(x).\]

Unifies semantic, instance, panoptic. OneFormer: single set of queries handles all three with a task token.

12.7 SegFormer, SegNeXt

Hierarchical encoder + lightweight all-MLP decoder. Strong efficiency-accuracy trade-off.

12.8 Open-vocabulary / promptable

Grounding DINO 1.5/1.6/Pro / DINO-X: open-set detection with text grounding. OWL-ViT / OWLv2: open-vocab via CLIP. T-Rex2: visual + text prompting. SAM, SAM 2, MobileSAM, EfficientSAM, HQ-SAM, Semantic-SAM: promptable segmentation.

★ 2026 SOTA update

• SAM 3 (Segment Anything with Concepts): Meta model for Promptable Concept Segmentation: detect/segment/track ALL instances of a concept from a noun phrase or image exemplar. Shared backbone image detector + memory video tracker with a presence head; 2x prior systems on SA-Co.

13. Diffusion Transformers

13.1 DiT (2022)

Replace U-Net with Transformer over noised latent tokens. Conditioning via adaLN-Zero (timestep + class):

\[\mathrm{adaLN}(x,c) = \gamma(c)\frac{x-\mu}{\sigma} + \beta(c), \quad y = x + \alpha(c)\cdot f(\mathrm{adaLN}(x,c)).\]

\(\alpha\) initialized to 0 so block is no-op at init. Scales cleaner than U-Net.

13.2 PixArt-α / Σ / δ

DiT with cross-attention to T5 text. PixArt-α small + efficient; Σ scales to 4K; δ adds LCM distillation.

13.3 Lumina-T2I

DiT with LLaMA backbone reused for text and image tokens; strong scaling properties.

13.4 MM-DiT (SD3, FLUX)

Two parallel streams (text, image) with separate \(W_Q/W_K/W_V/W_O\) and MLPs; interact through one joint self-attention over concatenated tokens. Cleaner cross-modal coupling than U-Net cross-attention.

13.5 Hunyuan-DiT, Step-Video, Allegro

Various MM-DiT-style implementations from Asian labs (Tencent, Stepfun) with different attention backbones, tokenizers, and training data.

13.6 Sora, Veo 3, Kling 2, MovieGen

Video MM-DiT at frontier scale. Spatiotemporal attention; causal 3D VAE for tokens; flow matching loss; joint image + video training. MovieGen also generates audio.

13.7 Cosmos World Foundation Models

NVIDIA's video diffusion + autoregressive families positioned as world models for robotics and AV.

★ 2026 SOTA update

• Sora 2: OpenAI frontier video+audio generation model (released Sep 30, 2025); markedly better physics, controllability and synchronized dialogue/sound; described as the 'GPT-3.5 moment for video'.

14. Video Transformers

14.1 VideoBERT (2019)

Tokenized video frames + text in one BERT-style model. The progenitor.

14.2 ViViT (2021)

4 factorization variants: full space-time, factorized encoder, factorized self-attention, factorized dot-product. Establishes the design space.

14.3 TimeSformer (2021)

Divided space-time attention: alternate space-only and time-only blocks. Cheap, effective.

14.4 MViT, MViTv2, Video Swin

Hierarchical video attention with pooled K, V (MViT) or 3D shifted windows (Video Swin).

14.5 VideoMAE, VideoMAE V2

Mask 90%+ tube patches; reconstruct in pixel space. The video analog of MAE.

14.6 V-JEPA, V-JEPA 2

Predict in latent space (EMA target). V-JEPA 2 scales to billion frames.

14.7 InternVideo, InternVideo 2

1B+ multimodal video encoder; trained with masked reconstruction + video-text contrastive + multimodal next-token. State of the art for many video benchmarks.

14.8 Long-form video VLMs

MovieChat, MA-LMM, LLaMA-VID, LongVU, LongVA, LongVILA, Goldfish: hour-long video understanding via memory, hierarchical pooling, retrieval, or token compression.

14.9 Sa2VA

SAM 2 + LLaVA fusion for grounded video Q&A: "segment X over time" returns mask + answer.

15. Audio Transformers

15.1 Wav2Vec 2.0, HuBERT, WavLM

Self-supervised speech representations. Quantize masked latents (Wav2Vec) or predict cluster IDs (HuBERT). Backbone for ASR.

15.2 Whisper (2022)

Encoder-decoder Transformer for speech-to-text trained on 680k hours of weakly-supervised multilingual audio. Robust to noise; state-of-the-art open ASR.

15.3 Conformer

Convolution-augmented Transformer for speech: each block has self-attention + a convolution module. Standard ASR backbone.

15.4 AudioLM (2022)

Two-stage hierarchical: semantic tokens (from w2v-BERT) generated first, then acoustic tokens (from SoundStream). Long-context audio continuation.

15.5 MusicGen, AudioGen, AudioLDM, Stable Audio

Generative audio with various conditioning. MusicGen: single-stage Transformer over codebook-interleaved EnCodec tokens. Stable Audio: latent diffusion with a CLAP-conditioned DiT.

15.6 Suno, Udio (closed)

Frontier music generation; architectural details unpublished but believed to combine LLM-style decoder with diffusion components.

15.7 NaturalSpeech, VALL-E, VALL-E X, F5-TTS

Modern TTS via in-context learning over neural audio codecs.

16. Code Transformers

16.1 Codex / GitHub Copilot

GPT-3 fine-tuned on GitHub code. Original code-Copilot stack.

16.2 CodeBERT, GraphCodeBERT, CodeT5, CodeT5+

Encoder or encoder-decoder pretrained on code. Used for code search, summarization, completion.

16.3 Code Llama, Codestral

LLaMA / Mistral fine-tuned on code, with Fill-in-the-Middle (FIM) pretraining:

prefix ⟨FIM⟩ suffix ⟨MID⟩ middle ⟨EOM⟩.

Enables in-the-middle completion (Copilot-style).

16.4 StarCoder, StarCoder 2

Open code models (BigCode), trained on permissively-licensed code only.

16.5 DeepSeek-Coder, Qwen-Coder, Yi-Coder

Strong Asian-led open code models; DeepSeek-Coder-V2 matches GPT-4 on HumanEval at much smaller size.

16.6 Reasoning code models

DeepSeek-R1, o3, GPT-5, Claude 4: code via long CoT + verifiable test-execution rewards.

17. Robotics / VLA Transformers

17.1 RT-1, RT-2, RT-X

Google Robotics line. RT-1: small Transformer over image + instruction tokens; discretized actions. RT-2: VLM (PaLI-X, PaLM-E) with action tokens in vocab; web data co-trained. RT-X: cross-embodiment trained on Open-X-Embodiment.

17.2 OpenVLA, Octo, RDT-1B

Open VLAs. OpenVLA: 7B Llama 2 + DINOv2/SigLIP + discretized actions. Octo: smaller Transformer policy. RDT-1B: 1B-param diffusion-policy Transformer.

17.3 π0, π0.5 (Physical Intelligence)

VLM backbone with a small flow-matching action head:

\[\mathcal{L}_{\mathrm{FM}} = \mathbb{E}\,\|v_\theta(a_t,t,o) - (a_1 - a_0)\|^2.\]

Continuous actions, fine manipulation. \(\pi_{0.5}\) extends to long-horizon hierarchical control.

17.4 GR00T, Helix

Humanoid VLA foundation models. GR00T (NVIDIA): open VLA-action stack with Cosmos sim. Helix (Figure): two-network architecture (slow planner + fast executor).

17.5 ACT, Diffusion Policy, 3D Diffuser Actor

Imitation-learning policies. ACT: Action Chunking Transformer with CVAE head. Diffusion Policy: action sequence as conditional diffusion. 3D Diffuser Actor: condition on point cloud or 3D feature volume.

17.6 Spatial reasoning VLMs

SpatialVLM, RoboPoint, SpatialBot, SpatialVLA: fine-tune VLMs with synthetic 3D-grounded spatial QA data \(\to\) spatial reasoning for robot tasks.

18. 3D Vision Transformers

18.1 Point cloud Transformers

Point Transformer (1, 2, 3): per-point self-attention with k-NN neighborhoods. PCT, PointNeXt: alternatives. Replaced PointNet++ for many 3D segmentation tasks.

18.2 LRM family (Large Reconstruction Models)

LRM: feed-forward triplane prediction from one image. InstantMesh, MeshLRM, GS-LRM, Long-LRM: variants producing meshes / Gaussians, from single or multi-view input.

18.3 DUSt3R, MASt3R, Spann3R

Predict per-pixel 3D pointmaps from N images directly, no SfM. The breakthrough that made feed-forward 3D dominant.

18.4 VGGT (Visual Geometry Grounded Transformer, 2025)

Large transformer mapping unposed N-image set to depth + camera + per-pixel 3D in one forward pass. By 2025 the dominant feed-forward 3D model.

18.5 3DETR, GroupFree3D

Detection in 3D point clouds with DETR-style query decoders.

18.6 BEV / occupancy

BEVFormer, BEVDet, PETR, StreamPETR, Far3D: camera-only 3D detection in BEV. TPVFormer, OccFormer, FB-OCC, SparseOcc, OPUS: occupancy prediction. (See §18.)

19. Specialized Transformers

19.1 Decision Transformer (RL)

Conditioning on returns-to-go: \(\pi_\theta(a_t \mid R_t, s_t, a_{<t}, R_{<t}, s_{<t})\), trained by cross-entropy on actions. At test, prompt with target return.

19.2 Trajectory Transformer (RL)

Same idea but also models states and rewards autoregressively; samples future trajectories and selects actions via planning.

19.3 Music Transformer

Autoregressive Transformer over MIDI tokens with relative position encoding. Established Transformers for symbolic music.

19.4 Set Transformer

Permutation-invariant Transformer for set-input tasks. Uses Pooling-by-Multihead-Attention (PMA): fixed query set attends over input set.

19.5 Perceiver, Perceiver IO

Cross-attention from a small fixed-size latent array to large input array; iterate. Decouples model FLOPs from input size. Handles arbitrary modalities (image, audio, point cloud, video).

19.6 Graphormer, GraphGPS

Transformers for graph learning. Add structural / distance encodings to standard attention. Strong on molecular property prediction.

19.7 Galactica

LLM trained on scientific papers + math + code; LaTeX-aware tokenization. Prototype scientific assistant.

19.8 Med-PaLM, Med-PaLM 2

PaLM fine-tuned for medical QA. Demonstrated chain-of-thought prompting + ensemble + self-consistency yields expert-level performance on USMLE.

19.9 Compression-friendly

• DistilBERT: distilled BERT, 40% smaller, 60% faster, 97% performance.
• TinyBERT, MobileBERT, MiniLM: various task-specific distillations.
• ALBERT: factorized embedding + cross-layer sharing.

19.10 Universal / multi-task

Pathways / PaLM, PaLM-2, Gemini: Google's large-scale multi-task / multimodal models. Pathways the underlying training infrastructure. PaLM-E: embodied multimodal for robotics.

19.11 Domain-specific

AlphaFold-2 / 3 (protein structure), ESM / ESM-3 (protein language models), DNA-BERT, Nucleotide Transformer, Evo (genomics), ChemBERTa, MolFormer (chemistry).

20. Foundation 3D / Geometric Transformers

20.1 Point Transformer family

Already covered; point-cloud self-attention with k-NN.

20.2 NeRF + Transformer hybrids

IBRNet, GNT: render novel views by attending across reference images. LRM family: triplane prediction via Transformer.

20.3 Geometric foundation models (2025)

DUSt3R, MASt3R, MASt3R-SfM, Spann3R, Splatt3R, NoPoSplat, Fast3R, VGGT, MoGe, MoGe-2: feed-forward geometry prediction from images. Successors to classical SfM/MVS.

20.4 Mesh generation Transformers

MeshGPT, MeshXL, MeshAnything, MeshAnything V2, EdgeRunner, BPT: autoregressive face-vertex prediction. Native mesh topology output.

20.5 Gaussian splatting + Transformer

GS-LRM, MeshLRM, Long-LRM: feed-forward 3D Gaussian prediction.

21. The 2026 Production Stack: Cross-Modal Summary

22. Architectural Decisions: Quick Decision Tree

22.1 When to pick what?

• Need understanding only (classification, retrieval): encoder-only (ModernBERT, GTE, SigLIP).
• Need generation only (text): decoder-only (LLaMA / Qwen / DeepSeek family).
• Need conditional generation, fixed input/output structure: encoder-decoder (T5 / BART) — but mostly displaced by decoder-only with prompt template.
• Need vision understanding: ViT or hybrid (Swin, ConvNeXt) for backbone; LLaVA-style adapter for VLM.
• Need vision generation: DiT / MM-DiT (image), spatiotemporal MM-DiT (video).
• Need very long context: RoPE+YaRN+ring attention; or Mamba / RWKV / RetNet for very long but lower-quality.
• Need extreme efficiency at scale: MoE (Mixtral, DeepSeek MoE) or distillation (DistilBERT, MobileViT).
• Need permutation invariance (sets): Set Transformer / Perceiver.
• Need handling many modalities at once: Perceiver IO; or native multimodal (Chameleon-style).

22.2 Common architectural mistakes

• Using BERT in 2026: switch to ModernBERT.
• Using vanilla DDPM training: use flow matching instead.
• Using post-norm: it's only stable for shallow / well-tuned setups.
• Using LayerNorm in modern stacks: RMSNorm is faster and equivalent.
• Using sinusoidal absolute position: RoPE wins virtually everywhere.
• Using MHA without GQA at inference: KV cache will dominate memory.
• Using U-Net for new diffusion projects: MM-DiT is the new default.
• Using bolted-on adapter VLM as your long-term bet: native multimodal is winning.

Appendix A: Pictures Worth Memorizing

For each, you should be able to draw the architecture cleanly on a whiteboard:

1. Original Transformer (encoder-decoder).
2. BERT block (pre-norm vs post-norm).
3. GPT block (causal mask).
4. T5 (encoder + decoder with cross-attn).
5. ViT (patch + CLS + Transformer).
6. Swin (hierarchical + shifted window).
7. DETR (encoder + object-query decoder + Hungarian matching).
8. Mask2Former (universal mask predictor).
9. CLIP (dual encoder + InfoNCE).
10. BLIP-2 (frozen image \(\to\) Q-Former \(\to\) frozen LLM).
11. LLaVA (CLIP \(\to\) MLP \(\to\) LLM).
12. Flamingo (gated cross-attn into frozen LLM).
13. Chameleon (interleaved tokens, single AR Transformer).
14. DiT (adaLN-Zero on noised tokens).
15. MM-DiT (two streams + joint attention).
16. Mixtral / Mixture-of-Experts block (router + top-k experts).
17. Mamba (selective SSM + parallel scan).
18. Perceiver (latent array \(\to\) cross-attn \(\to\) refine).
19. RT-2 / OpenVLA (VLM + action token vocab).
20. \(\pi_0\) (VLM + flow-matching action head).

Appendix B: Year-by-year shortlist

• 2017: Transformer.
• 2018: BERT, GPT-1.
• 2019: GPT-2, RoBERTa, T5 (released 2020), ALBERT, XLNet, BART.
• 2020: GPT-3, ELECTRA, DETR, ViT (released 2021), iGPT.
• 2021: CLIP, Swin, DALL-E, ALIGN, GLIDE, Codex.
• 2022: PaLM, Flamingo, BLIP-2, Stable Diffusion (LDM), Whisper, Diffusion Transformer (DiT), DreamFusion, NeRF → Instant-NGP.
• 2023: GPT-4, LLaMA, LLaMA 2, Mistral, BLIP-2, MiniGPT-4, LLaVA, SDXL, Segment Anything, Control-Net, NeRF → 3DGS.
• 2024: Gemini 1.5, GPT-4o, Claude 3, LLaMA 3, Mixtral \(8\times 22\mathrm{B}\), DeepSeek-V2 (MLA + MoE), Sora, SD3, FLUX, \(\pi_0\), MM-DiT, Chameleon.
• 2025: GPT-5, Claude 4 / Opus 4.6, Gemini 2.5, LLaMA 4, DeepSeek-V3, DeepSeek-R1 (pure-RL reasoning), Qwen 3, FLUX Kontext, Veo 3, Wan 2.1, VGGT, GR00T, \(\pi_{0.5}\), Cosmos.
• 2026: native multimodal default, GRPO standard for reasoning, MM-DiT standard for diffusion, MLA / GQA / MoE composed everywhere.