Data Collection & Curation — All Variants & Tricks

Updated July 2026 with 2025–2026 SOTA additions — new entries marked ★. Algorithm names link to their papers (arXiv / project page).

April 2026 · Version 1.0 Algorithm names link to their original papers (arXiv / project page).

Contents

1. Why Data Is the Moat
2. Data Sources
3. Web Crawling and Common Crawl
4. Quality Filtering
5. Deduplication
6. Multimodal Datasets
7. Captioning at Scale
8. Synthetic Data
9. Human Labeling and Annotation
10. Preference Data for RLHF / DPO
11. Reasoning Data
12. Safety and Alignment Data
13. Tokenization-Aware Curation
14. Curation Pipeline Architecture
15. Data Mixture and Curriculum
16. Long-Context Data
17. Data Ethics and Legal
18. Data Engines (Production Loops)
19. Tools and Infrastructure
20. Evaluation of Data Quality
21. Multimodal Data Specifics
22. Frontier 2025–2026
23. Production Stack 2026

Appendix A: Twenty-Five Things to Know Appendix B: Decision Tree — "How Do I Get Data?" Appendix C: Year-by-Year Data Milestones

1. Why Data Is the Moat

1.1 The thesis

Key

Modern AI quality is dominated by data quality and curation, not model architecture. At fixed compute, the team with better data wins by 2–10×. Architecture innovations matter; data dominates.

1.2 Evidence (2022–2026)

• Phi family: 7B-class models matching 70B via curated synthetic "textbook" data.
• LLaMA 3: data-recipe improvements + scale = much better than LLaMA 2 at same architecture.
• R1 / R1-Distill: 800k curated long-CoT traces transfer reasoning to smaller models.
• FineWeb / FineWeb-Edu: shows simple quality classifier >> raw web at fixed token budget.
• DataComp / DataComp-LM: data benchmarks show data-curation gains often dwarf architecture.

1.3 The pipeline at a glance

Source → filter → dedupe → score / rank → mix → tokenize → pack → shard → stream to training.

1.4 Per-stage data needs

1.5 Data Pareto frontier

For pretraining, the curve is roughly: quality matters at small scale; quantity matters at large scale. FineWeb-Edu beats raw web at small token budget; raw web with dedup wins at multi-trillion budget.

1.6 The Chinchilla revision

• Chinchilla: 20 tokens per parameter, compute-optimal.
• Modern frontier: 200–2000 tokens per parameter (LLaMA 3.1 trained on 15T tokens; LLaMA 4 even more).
• Reason: at inference, smaller well-trained model is cheaper than larger compute-optimal.
• Implication: data demand exceeded all expectations; getting more high-quality tokens is the bottleneck.

2. Data Sources

2.1 Text sources

• Common Crawl: monthly web scrape; petabytes of raw HTML.
• Books: Project Gutenberg, Books3 (controversial), licensed publishers.
• Wikipedia / Wikidata: high-quality structured.
• ArXiv / PMC: scientific papers (PDF, LaTeX).
• StackExchange / GitHub Issues: technical Q&A.
• Reddit / Forums: conversational; quality variable.
• Code (GitHub, GitLab): BigCode's The Stack v1/v2.
• News / Magazines: licensed deals (NYT, WSJ, Axel Springer with OpenAI).
• Court documents, government reports: high-quality public.

2.2 Image sources

• Web crawl (LAION-5B, COYO-700M).
• Filtered subsets (LAION-aesthetic, LAION-2B-en).
• Licensed (Shutterstock-OpenAI, Adobe Stock).
• Captured / studio (proprietary product).
• Synthetic from generative models.

2.3 Video sources

• Web crawl (Panda-70M, HD-VG-130M, InternVid).
• YouTube (filtered by license / API).
• Movies / TV (licensed deals).
• User-uploaded (TikTok, Instagram — proprietary).
• Stock video (Shutterstock, Pond5, Storyblocks).

2.4 Audio sources

• LibriSpeech / GigaSpeech / Common Voice (speech).
• AudioSet / AudioCaps / Clotho (general audio).
• Free Music Archive / Million Song Dataset.
• Licensed music libraries.

2.5 3D sources

• Objaverse / Objaverse-XL (10M+ 3D assets).
• ShapeNet, ABO, GSO.
• Sketchfab, TurboSquid (licensed).
• Photogrammetry from web.

2.6 Robot data

• Open-X-Embodiment (1.4M trajectories).
• DROID, Bridge V2, ALOHA datasets.
• AgiBot World, RH20T.
• Proprietary teleop fleets (1X, Figure, Tesla).

2.7 Synthetic from generative models

• Text from GPT-4 / Claude.
• Images from FLUX / SDXL / Imagen.
• Video from Sora / Veo.
• Code from Copilot / Codestral.
• Captures the "frontier knowledge" for distillation into smaller models.

2.8 Source quality hierarchy

• Tier 1: Wikipedia, books, peer-reviewed papers (high quality, low volume).
• Tier 2: news, StackExchange, GitHub (good quality, medium volume).
• Tier 3: filtered web (variable; biggest volume).
• Tier 4: forums, social media (low quality; useful for diversity).

3. Web Crawling and Common Crawl

3.1 Common Crawl 101

• Non-profit foundation; monthly snapshots.
• ~3–5B pages per month.
• Petabytes of raw HTML.
• WARC (Web ARChive) format.
• WET (text extraction), WAT (metadata).

3.2 Major web-derived corpora

3.3 C4 (Colossal Clean Crawled Corpus)

T5's training data. Filters: keep only English; min line / doc length; remove placeholder text; remove offensive content (controversial blocklist).

3.4 The Pile (EleutherAI)

Community-curated; explicitly mixes 22 high-quality sources alongside CC. Influential design.

3.5 RefinedWeb (Falcon)

Demonstrated CC alone, well-filtered + deduped, beats Pile-style mixes. Heavy URL/exact/fuzzy dedup.

3.6 Dolma (AI2)

Open-source preprocessed corpus + open processing toolkit. Reproducibility-first.

3.7 DCLM-baseline (DataComp-LM, 2024)

• Single-pass classifier filter trained on OpenHermes / Tulu / r/ELI5.
• Outperforms much larger raw-CC baselines.
• Demonstrates: a small classifier filter beats elaborate hand-rules.

3.8 FineWeb / FineWeb-Edu (HuggingFace 2024)

• FineWeb: all CC dumps; rule-based + lightweight filtering; 15T tokens.
• FineWeb-Edu: + classifier trained on Llama-3-70B-judged "educational quality."
• FineWeb-Edu 1.3T tokens beats raw FineWeb 15T at small-scale training.

3.9 Robots.txt and ethics

• CC respects robots.txt at crawl time.
• Some publishers added to CC blocklist post-2023 (NYT, etc.).
• Site-level opt-out via robots.txt extensions emerging.
• Legal landscape uncertain; ongoing lawsuits.

4. Quality Filtering

4.1 Heuristic filters

• Min length (e.g., > 200 chars).
• Max ratio of symbols / numbers.
• Stop word ratio (must contain enough common words).
• Average line length.
• Repetition ratio (Gopher's repetition heuristics).
• Boilerplate detection (cookie banners, footers).

4.2 Gopher heuristics (DeepMind)

First well-documented quality-filter recipe:

• 50–100k chars per doc.
• Mean word length 3–10.
• <30% lines starting with bullet / digit.
• Stop word presence.

Industry default until classifier-based filters (DCLM, FineWeb-Edu).

4.3 Classifier-based filtering

• Train small classifier (fastText / linear) on positive examples (Wikipedia, OpenHermes, ELI5) vs negative (random web).
• Score each document; threshold.
• Cheap; easy to retrain for new criteria.

4.4 Educational quality (FineWeb-Edu)

• Use Llama-3-70B as judge: "rate 0–5 educational value."
• Train small classifier on 500k judged samples.
• Apply to FineWeb at scale.
• Result: ~1.3T tokens of high educational value.

4.5 Toxicity / NSFW filters

• Perspective API (Google).
• Detoxify, ToxicBERT.
• Custom classifiers per safety policy.
• Different thresholds per training stage (more lenient for pretraining; strict for SFT).

4.6 PII filters

• Email, phone, SSN regexes (high precision).
• Address detection.
• Named entity recognition for names.
• Face / license-plate blur for image data.
• Typically replace with placeholders, not delete.

4.7 Code quality filters

• Lint (flake8 / pylint / clang-tidy).
• Cyclomatic complexity.
• Test pass rate (executable subset).
• Star count / fork count as signals.
• License compatibility (StarCoder permissive-only).

4.8 Domain / topic filters

• Specialty corpora: medical (PubMed), legal (court records), code, math.
• Per-domain classifier: "is this medical content?"
• Used to up-weight specific domains during training.

4.9 DSIR (Data Selection by Importance Resampling)

• Importance-sample web data to match a target distribution (e.g., books).
• Train classifier p(target)/p(source).
• Resample with weight ∝ ratio.
• Used in some open recipes.

4.10 Quality vs quantity

Key

Empirical pattern: strict filtering wins at small training budget; lenient at huge. FineWeb-Edu 1.3T > FineWeb 15T at 1B-param scale; reverses at 70B-param at 15T-token budget. Pick filter based on your scale.

5. Deduplication

5.1 Why dedupe?

• Memorization risk on duplicates.
• Wasted compute on repeats.
• Eval contamination if test sets overlap with train.
• Diversity hurts: 10% of CC is near-duplicate by content.

5.2 Exact dedup

• Hash document; if seen, drop.
• Standard pre-processing step.
• Removes ~30% of CC.

5.3 Near-duplicate detection (MinHash + LSH)

• Hash each document into K MinHash signatures.
• Locality-Sensitive Hashing (LSH) buckets similar docs.
• Within-bucket pairwise Jaccard check.
• Standard library: datasketch / Spark MinHashLSH.
• ~70% reduction common after dedup.

5.4 SemDeDup (semantic dedup)

• Embed each document with a sentence encoder (CLIP / text-embedding-3).
• Cluster (k-means or graph) by embedding similarity.
• Keep one representative per cluster.
• Catches paraphrases / translations.
• Used in DataComp.

5.5 Document-level vs span-level

• Document-level: drop entire doc if duplicate.
• Span-level (e.g., 50-token window): drop just the duplicated span.
• Span-level preserves more diverse context but is expensive.

5.6 URL / domain dedup

Drop multiple pages from same URL pattern; cap per-domain count to prevent over-representation of large sites.

5.7 Cross-corpus dedup

• Dedupe against eval datasets (avoid contamination).
• Cross-corpus dedupe (CC vs books vs Wikipedia).
• Released datasets often pre-deduped against major eval suites.

5.8 N-gram-overlap deduplication

• Compute n-grams (n = 8–13) for each document.
• Match against eval set; drop docs with high overlap.
• Standard contamination check.

5.9 Practical dedup tools

• slimpajama-dedup, datasketch (Python).
• dolma dedup pipeline.
• Apache Spark MinHashLSH (scale).
• text-dedup (HuggingFace).

6. Multimodal Datasets

6.1 Image-text pair datasets

6.2 Interleaved multimodal datasets

• OBELICS (HuggingFace 2023): 141M interleaved image-text web docs.
• MMC4: interleaved CC.
• Used for VLM pretraining (Flamingo / IDEFICS / LLaVA-NeXT).

6.3 Video-text datasets

• HowTo100M: 136M instructional video clips.
• WebVid-10M: 10M short clips.
• Panda-70M: 70M clips with captions.
• HD-VG-130M: 130M HD clips.
• InternVid: 7M long-form (up to 24 min).
• LSMDC: 200 movies with descriptions.
• Re-captioned variants emerging (e.g., Open-Sora's recaption).

6.4 Audio-text datasets

• AudioSet: 2M YouTube clips.
• AudioCaps, Clotho: captioned audio.
• LibriSpeech, LibriLight: speech.
• Common Voice: crowdsourced multilingual.
• GigaSpeech, MLS: large-scale speech.

6.5 3D datasets

• Objaverse: 800k 3D assets.
• Objaverse-XL: 10M+ 3D assets.
• ShapeNet: 51k synthetic models.
• ABO (Amazon Berkeley Objects): 8k product 3D.
• GSO (Google Scanned Objects): 1k high-quality scans.

6.6 Robot datasets

• Open-X-Embodiment: 1.4M trajectories, 22 embodiments.
• DROID: 76k trajectories, 13 institutions.
• Bridge V2: 60k tabletop demos.
• ALOHA / Mobile ALOHA: bimanual demos.
• AgiBot World: humanoid demos at scale.
• RH20T: 110k diverse trajectories.

6.7 AV datasets

• nuScenes, Waymo Open, Argoverse 2, KITTI-360, Lyft: standard.
• Commercial: Tesla, Cruise, Wayve internal (orders of magnitude more).

7. Captioning at Scale

7.1 Why re-caption?

Web alt-text is poor:

• "image1.jpg" style filenames.
• SEO spam.
• Missing detail.
• Often unrelated to image.

Re-captioning with strong VLM dramatically improves downstream training.

7.2 Re-captioning pipelines

1. Sample images from raw web data.
2. Generate dense caption with strong VLM (GPT-4V, Qwen-VL, BLIP-2).
3. Filter: ensure faithfulness, diversity.
4. Use re-captioned pairs for downstream training.

Used by Sora, FLUX, SD3, DALL-E 3.

7.3 Multi-aspect captions

Generate multiple captions per image:

• Factual (objects + scene).
• Aesthetic (style + mood).
• Compositional (spatial relations).
• Action / motion (for video).
• Use mix during training; improves prompt-following diversity.

7.4 DALL-E 3's recaptioning

• Train on (re-caption, image) pairs.
• Use frontier VLM (GPT-4-class) for captions.
• User prompt → rewritten by GPT-4 to match caption style.
• Result: excellent prompt-following.

7.5 Sora's recaptioning

Re-captioned all training videos with detailed VLM-generated descriptions. Critical for prompt fidelity.

7.6 LAION-Aesthetic + recaption pattern

Open community workflow: filter LAION by aesthetic score, then re-caption with open VLM (LLaVA / Qwen-VL). Used in many open T2I / T2V training runs.

7.7 Caption-quality eval

• CLIP-Score between image and caption.
• Caption-fidelity human rating.
• Downstream FID after training on the captions.

8. Synthetic Data

8.1 The synthetic-data thesis

Curated synthetic data often beats unfiltered real data.

Phi-1 demonstrated: "textbook-quality" synthetic Python data trains a 1.3B model to outperform many larger models on coding.

8.2 Self-Instruct (Wang et al. 2022)

1. Seed: 175 hand-written instructions.
2. Generate: LLM produces new instructions + responses.
3. Filter: rule-based + diversity check.
4. Iterate: bootstrap from generated set.

Foundation of many instruction-tuned open models.

8.3 Phi family pattern (Microsoft)

• Generate textbook-quality data via GPT-4.
• Filter for diversity + quality.
• Train small model on synthetic only or synthetic + real.
• Phi-1 (1.3B), Phi-1.5, Phi-2 (2.7B), Phi-3 (3.8B / 7B / 14B), Phi-4 (14B), Phi-4-mini.

8.4 R1-Distill (DeepSeek 2025)

Generate ~800k long-CoT traces from R1; SFT smaller models on those traces. Strong reasoning at small scale.

8.5 Distillation from frontier APIs

• Generate Q&A / instructions via GPT-4 / Claude / Gemini.
• SFT open base on the generated set.
• Vicuna, WizardLM, OpenChat, Tulu, OpenHermes pattern.
• Legal: most closed-API ToS forbid using outputs to train competing models. Check.

8.6 Constitutional AI synthetic data

• Model generates responses.
• Critiques itself against constitution.
• Generates revisions.
• Train on (response, critique, revision) tuples.
• Used in Anthropic's Constitutional AI; foundation of RLAIF.

8.7 Rejection-sampling fine-tuning (RFT, RAFT)

• Sample N responses per prompt from current model.
• Keep top-K by reward.
• SFT on filtered responses.
• Iterate.

Cheaper than PPO; often competitive.

8.8 Synthetic image / video for CV

• Render with game engine (Unity, Unreal, NVIDIA Isaac).
• Procedural generation (Kubric, Hypersim, Habitat).
• Diffusion-augmented (use SD / FLUX to vary captures).
• Critical for rare-class augmentation in detection / segmentation.

8.9 Synthetic robot data

• Sim teleoperation in Isaac / Genesis / ManiSkill.
• Procedurally generated tasks + objects.
• Used to bootstrap VLAs without massive real demos.

8.10 Quality control for synthetic data

• Filter by reward / verifier.
• Diversity metrics (n-gram, embedding spread).
• Hold-out human eval on samples.
• Risk: model collapse if pure synthetic without real anchor.

Watch out

Model collapse: training models on their own outputs degrades over generations. Mix synthetic with real data; use frontier teachers, not your own previous-gen model.

9. Human Labeling and Annotation

9.1 The labeling industry

• Scale AI: largest commercial labeling vendor.
• Surge AI: high-quality preference data for RLHF.
• Labelbox: tooling + workforce.
• Sama, iMerit, CloudFactory: emerging-market workforce.
• Mercor, Outlier: expert / AI-trained-on-experts.
• In-house: Anthropic / OpenAI build their own labeler ops.

9.2 Labeling cost models

• Per-image segmentation: $0.05–$0.50.
• Per-pair RLHF preference: $0.50–$3.00.
• Per-expert reasoning trace: $5–$50.
• Per-multi-step agent eval: $20–$200.

9.3 Active learning

1. Train initial model.
2. Identify uncertain examples (high-loss / ensemble disagreement).
3. Label only uncertain.
4. Re-train.
5. Iterate.

~10× label efficiency vs random sampling.

9.4 Weak supervision (Snorkel)

• Multiple noisy "label functions" (rules / heuristics / classifiers).
• Combine via probabilistic graphical model.
• Train final model on combined labels.
• Useful when labels are abundant but noisy.

9.5 LLM-as-labeler

• GPT-4 / Claude as judge.
• Cheaper than human; biased (position, length, self-preference).
• Best for: filtering, ranking, low-stakes classification.
• Worst for: creative quality, edge cases, safety-critical.

9.6 Inter-annotator agreement

• Cohen's κ, Fleiss' κ for categorical.
• Krippendorff's α for general.

• 0.8: high agreement.

• 0.6–0.8: moderate.
• < 0.6: signal too noisy; re-spec the task.

9.7 Labeler training and calibration

• Detailed rubric with examples.
• Calibration set: hidden in stream; track per-labeler accuracy.
• Outlier-labeler removal.
• Periodic re-training.

9.8 Quality assurance (QA)

• Multi-rater for sensitive labels.
• Spot-check by senior annotators.
• ML-based outlier detection on labels.
• Adversarial probes.

10. Preference Data for RLHF / DPO

10.1 Pairwise preferences (Bradley-Terry)

For prompt \(x\), two responses \((y_w, y_l)\) with \(y_w\) preferred. Standard for RM training and DPO.

10.2 Listwise preferences (Plackett-Luce)

Rank K responses ordered worst-to-best. More efficient per-prompt.

10.3 Single-response thumbs (KTO data)

Per-response +/− flag (no pairs). Cheaper to collect.

10.4 Best-of-N rejection sampling

• Sample N responses.
• Keep top by reward / human preference.
• SFT on chosen.
• RAFT, RFT pattern.

10.5 Pair generation strategies

• Two samples from current model (online).
• One from current, one from base / weaker model.
• One from helpful response set, one from harmful (safety).
• Diverse prompts to cover distribution.

10.6 Preference data quality

Critical:

• Pair-quality variance: high-confidence pairs better.
• Length-balance: avoid systematic length bias.
• Diversity of prompt distribution.
• Disagreement filter: drop pairs labelers disagree on.

10.7 Iterative / online preference data

• Generate pairs from current model.
• Label / score.
• Update model (DPO / GRPO).
• Repeat.

Tracks model's evolving distribution; avoids stale data issue.

10.8 Self-Rewarding / SPIN

• Model judges its own pairs.
• Iterative bootstrap.
• Cheap; risk of preference drift.

10.9 Dataset releases (open)

• HH-RLHF (Anthropic): 161k helpful + harmless pairs.
• UltraFeedback: 64k pairs across 256 models.
• Nectar: 183k pairs.
• Skywork-Reward, ArmorRM: open RM training data.

11. Reasoning Data

11.1 Math reasoning

• GSM8K: 8.5k grade-school problems.
• MATH: 12.5k competition problems.
• AIME: USA olympiad qualifier.
• MathInstruct, MetaMathQA: instruction-tuned.
• NuminaMath: 860k math problems with solutions.

11.2 Code reasoning

• HumanEval, MBPP, CodeContests.
• LiveCodeBench: continually updated to avoid contamination.
• SWE-bench, SWE-bench-Verified: real GitHub issues.
• Synthetic: Code Llama synthetic instruction tunes; OpenCoder data.

11.3 Long-CoT distillation

• Generate from R1 / o1 / o3.
• Filter by correct final answer.
• Filter for trace quality (not too short, not rambling, no language mixing).
• R1-Distill series (DeepSeek, 800k traces): canonical example.

11.4 Process Reward Model (PRM) training data

• PRM800K (OpenAI): human-labeled per-step correctness.
• Math-Shepherd: auto-label via continuation success rate.
• OmegaPRM: tree-search-based MC labels.
• Implicit PRM: derived from outcome via DPO closed-form.

11.5 Verifiable-reward data (R1 style)

• Math: with extractable numeric / symbolic answer.
• Code: with passing test cases.
• Logic puzzles: with checkable constraints.
• Programming competitions: ICPC / Codeforces archives.

11.6 Curating reasoning datasets

• Filter trivial-easy and impossible problems.
• Difficulty-stratify (curriculum).
• Diverse problem types.
• Anti-contamination check vs eval suites.

11.7 Multi-step reasoning data quality

• Verifier-curated correct answers.
• Step-quality labels (PRM).
• Diverse solution paths per problem.
• Avoid memorization-friendly templates.

12. Safety and Alignment Data

12.1 Red-team prompts

• Manually crafted attacks (jailbreak attempts).
• Automated red-team via LLM adversary.
• Categorized: misinformation, bio / chem / nuke, cyber, etc.
• HarmBench, JailbreakBench, AdvBench: benchmarks.

12.2 Refusal data

• (Harmful prompt, refusal response) pairs.
• Calibration: refuse harmful, comply with similar-but-benign.
• False-positive prevention: "I can't tell you this" on legitimate query is a bug.

12.3 Constitutional pairs

• Generate response.
• Critique against constitution.
• Revise.
• Train RM on (original, revised) pairs.

Anthropic's CAI flow.

12.4 Truthfulness data

• TruthfulQA: deceptive-pattern probes.
• Retrieval-grounded fact-checks.
• Hallucination-pair training.

12.5 Bias / fairness data

• BBQ, BOLD, StereoSet, CrowS-Pairs: bias benchmarks.
• Counterfactual data augmentation (swap demographics).
• Per-group performance evaluation.

12.6 Safety datasets (open)

• HH-RLHF (helpful-harmless).
• AART (Adversarial Response Training).
• Bai et al. Constitutional AI dataset.
• XSTest: false-refusal benchmark.

12.7 Continuous safety pipeline

• Production telemetry: detect new attack patterns.
• Generate counter-examples.
• Update training data.
• Re-train; A/B before deployment.

13. Tokenization-Aware Curation

13.1 Per-tokenizer effects

The same data tokenizes differently under different vocabs. CJK languages are 2–5× as many tokens under Llama 2 vs Llama 3.

13.2 Number tokenization

• Modern (Llama 3, GPT-4o): split numbers into individual digits.
• Avoids "100" / "1000" becoming single tokens.
• Improves arithmetic dramatically.

13.3 Code tokenization

• Multi-character indents as single token (4-space, 8-space).
• Preserve common operators.
• Per-language specialized tokens (e.g., HTML).

13.4 Multilingual fairness

• Train tokenizer on balanced multilingual corpus.
• Larger vocab helps low-resource languages.
• Llama 3's 128k vocab vs Llama 2's 32k: huge multilingual improvement.

13.5 Special tokens for instruction-tuning

• Chat template tokens (<|im start|>).
• Tool call tokens.
• Reasoning tokens (<think>).
• Multimodal placeholders (<image>).

14. Curation Pipeline Architecture

14.1 Standard pipeline stages

1. Raw data ingestion (S3, GCS).
2. Format conversion (HTML → text, image → standardized).
3. Deduplication.
4. Quality filtering.
5. Domain / topic classification.
6. PII / safety filtering.
7. Tokenization + length-stratification.
8. Sharding into training-ready format.

14.2 Distributed processing

• Apache Spark: standard for terabyte-scale.
• Ray Data: Python-native, Pythonic UDFs.
• Dask: pandas-style for medium-scale.
• Beam: portable, used in some teams.

14.3 Data lake vs object store

• Object store (S3): raw petabytes.
• Data lake (Iceberg, Delta Lake, Hudi): queryable + versioned.
• Parquet / Arrow for tabular metadata.

14.4 Streaming formats for training

• WebDataset: tar shards over object storage; sequential I/O.
• Mosaic Streaming: shuffled streaming with deterministic resume.
• TFRecord: TensorFlow legacy.
• Parquet: structured; smaller for tabular.
• LMDB: random-access; smaller datasets.

14.5 Versioning and lineage

• Per-dataset version IDs.
• Per-document content-addressable hash.
• Pipeline-version stamped on output.
• Reproducibility: "which data was this checkpoint trained on?"

14.6 Dolma toolkit (AI2)

Open-source data curation toolkit; reproducible OLMo data pipeline. Reference implementation.

14.7 Other open toolkits

• datatrove (HuggingFace): processing FineWeb-style.
• slimpajama-dedup: dedup specifically.
• red-pajama-data: end-to-end open recipe.
• text-dedup: HuggingFace dedup utilities.
• NeMo Curator (NVIDIA): production-scale curation.

14.8 Production data engineering teams

Modern frontier labs have dedicated data teams (5–50 engineers) building the curation pipeline. Often considered as critical as model architecture team.

15. Data Mixture and Curriculum

15.1 Mixture weights

What fraction from each source. Critical hyperparameter; massive quality impact.

15.2 Common mixtures

• LLaMA 1: CC 67%, books 4.5%, GitHub 4.5%, Wikipedia 4.5%, ArXiv 2.5%, etc.
• Most modern: similar with adjusted weights based on per-source quality.

15.3 DoReMi (Domain Reweighting)

• Train small proxy model with adversarial weighting.
• Reweighting maximizes worst-domain loss reduction.
• Apply weights to large training run.
• Reduces over-fit to dominant domains.

15.4 Quality-weighted sampling

• Sample high-quality docs more often.
• Quality from classifier (FineWeb-Edu).
• Per-doc importance weight.

15.5 Curriculum strategies

• Easy-to-hard: simpler text first; complex later.
• Length curriculum: short context first; extend.
• Domain curriculum: general first, specialized later.
• Code curriculum: documentation first, then code.
• Most large training runs use simple uniform mixing; curriculum gains are mild but real.

15.6 Long-context data mixing

• Concatenate documents to fill context.
• Or: pack at document boundaries with separator.
• Synthetic long-context: question-over-document; multi-doc synthesis tasks.
• Critical for > 32K context training.

15.7 Annealing on high-quality

Many recipes: pre-train on bulk web; final ~100B tokens on highest-quality (math, code, instruction). Polish.

16. Long-Context Data

16.1 The challenge

Most natural docs are short. Training a 100K-context model needs millions of long examples. Sources are scarce.

16.2 Sources of natural long-context

• Books (full novels).
• Code repositories (whole-repo training).
• Scientific papers.
• Long arXiv tex / dissertations.
• Legal documents.
• Government reports.

16.3 Synthetic long-context

• Concatenation: pack short docs together with separator.
• Question over long doc: generate (long context, query, answer).
• Multi-doc synthesis: combine multiple docs; require reasoning across.
• Distractor injection: insert irrelevant docs between query-relevant.

16.4 Length stratification

Sort docs by length; sample by length bucket to ensure training sees full range. Avoid all-short or all-long batches.

16.5 Quality at long context

• Document-level dedupe more important.
• Boilerplate doubly costly at long lengths.
• Per-position loss weighting (down-weight first K tokens).

16.6 Long-context evaluation data

• NIAH (Needle in a Haystack): retrieval probe.
• RULER: extended NIAH with multi-step.
• LongBench, ∞Bench, LV-Eval: real-world long-doc tasks.
• Care: NIAH-pass ≠ real long-doc capability.

17. Data Ethics and Legal

17.1 Copyright and fair use

• Open question: is training on copyrighted text fair use?
• Multiple lawsuits (NYT vs OpenAI, authors vs Meta, etc.) ongoing.
• Some labs (Adobe Firefly): only licensed + public-domain training.
• Stable Diffusion / LAION: face controversy over copyrighted images.

17.2 Per-jurisdiction regulation

• EU AI Act: provenance + transparency requirements.
• GDPR: right to erasure includes training data?
• California AB-2013: ML training data disclosure.
• Japan, Singapore: more permissive.
• Compliance teams now standard at frontier labs.

17.3 PII handling

• Detect: regex + NER + classifier.
• Redact: replace with placeholders.
• Or: drop entire document.
• Per-data-type policy.

17.4 Right to be forgotten (data unlearning)

• Active research; no good solution.
• Workarounds: filter at inference (system prompt), retrain from snapshot.
• Machine unlearning: methods to surgically remove training examples.

17.5 Licensing

• Code: MIT, Apache, GPL (copyleft); StarCoder uses permissive-only.
• Image: CC0, CC-BY, CC-BY-SA, all-rights-reserved.
• Books: Project Gutenberg (public domain), licensed.
• Per-project license-compatibility audit.

17.6 C2PA provenance

• Industry standard: Adobe, Microsoft, OpenAI, Google.
• Cryptographically signed metadata.
• Track "where did this image / video / text come from?"
• Apply to both training data and model outputs.

17.7 Watermarking outputs

• Stable Signature for images.
• SynthID for Veo.
• Text watermarking (less mature; OpenAI exploring).
• Mostly to identify model outputs in future training data.

17.8 Avoiding data contamination

• Cross-check training corpus against eval suites.
• N-gram overlap detection.
• Time-stamped data: only use pre-eval-date.
• LiveCodeBench / continually-updated benchmarks address this.

17.9 Ethical sourcing

• Labeler wages and conditions: fair-pay commitments.
• Mental health support for content moderation.
• Trauma-informed processes for safety labeling.

18. Data Engines (Production Loops)

18.1 Tesla's data engine (AV reference)

The original "data engine" pattern:

1. Deploy current model in production.
2. Detect failures (uncertainty / disagreement).
3. Surface to labelers.
4. Label.
5. Retrain.
6. Deploy.
7. Repeat.

Continuously improves over millions of operating hours.

18.2 Pattern adopted across AI

Now standard in:

• LLM RLHF (preference data flywheel).
• AV perception (Wayve, Mobileye, Cruise).
• Robotics (collect more demos for failure modes).
• Computer Use (Anthropic, OpenAI agents).

18.3 Long-tail mining

• Production telemetry identifies rare / failing scenarios.
• Rare classes / scenarios up-weighted in next training.
• Synthetic augmentation of rare cases.
• Active learning targets long tail.

18.4 Auto-labeling

• Use bigger / slower model to label data for smaller / faster.
• Distillation pattern.
• Reduces human cost.
• Tesla, Google, Wayve all use heavily.

18.5 Human-in-the-loop QA

• AI suggests labels; human approves.
• ~10× throughput vs pure manual.
• Quality > pure auto.

18.6 Continuous improvement loop

1. Production model deployed.
2. Telemetry on errors.
3. Auto-label most; human-label hardest.
4. Continuous training (or periodic retrain).
5. Canary / A-B before swap.

18.7 Labeler ops at scale

• Outsourced for volume.
• In-house for sensitive / expert.
• Per-task workflow systems.
• Quality dashboards per labeler.

18.8 Anthropic / OpenAI labeler teams

• Large in-house labeler ops (hundreds-thousands).
• Specialized expertise (reasoning, safety, coding).
• Compensated comparable to engineers for expert labeling.

19. Tools and Infrastructure

19.1 Curation toolkits

19.2 Distributed compute

• Apache Spark: standard.
• Ray Data: Python-friendly.
• Dask: pandas-style.
• Spark on EMR / Dataproc / Databricks.
• Custom on Kubernetes.

19.3 Storage

• Object storage (S3, GCS, Azure Blob).
• HDFS legacy.
• Object lifecycle policies (hot → cold).
• Compression (zstd standard).

19.4 Streaming / loading

• WebDataset (tar shards).
• Mosaic Streaming.
• Litdata.
• Datasets (HuggingFace).

19.5 Labeling tools

• Label Studio: open-source.
• CVAT: image / video annotation.
• Roboflow: vision-focused.
• Prodigy: spaCy team's NLP.
• Internal tools (Anthropic, OpenAI proprietary).

19.6 Eval tools

• HuggingFace Datasets + Evaluate.
• lm-evaluation-harness (EleutherAI).
• Mosaic Composer eval.
• OpenAI Evals.
• Big-Bench.

19.7 Observability

• Per-shard data composition tracking.
• Per-domain loss curves.
• Token-level sample weighting visualization.
• Wandb / TensorBoard / MLflow.

20. Evaluation of Data Quality

20.1 Direct metrics

• Document-level classifier scores.
• Length distribution.
• Per-language / per-domain composition.
• Dedup ratio.
• N-gram diversity.

20.2 Held-out perplexity

Train small proxy model; measure perplexity on held-out high-quality set (Wikipedia, eval suites). Use as ranking signal for data variants.

20.3 Downstream eval correlation

• Train small proxy model on each data variant.
• Eval on standard benchmarks.
• Use to rank data variants.
• Faster than full-scale training; not always perfectly correlated.

20.4 Scaling laws for data

• Per-source scaling laws (Hoffmann et al. Chinchilla revisit).
• Per-quality scaling laws (FineWeb-Edu).
• Per-domain mixture optimization (DoReMi).

20.5 Mix ablations

• Train identical models with different mixtures.
• Compare on broad eval suite.
• Per-task per-mix performance heatmap.

20.6 Counterfactual auditing

• Hold out specific domain; measure performance loss.
• Identifies which data is most valuable.
• Used in DoReMi-style optimization.

20.7 Long-running data eval cadence

• Per-checkpoint eval on broad suite.
• Per-data-version eval on proxy model.
• Periodic full-scale ablation.

21. Multimodal Data Specifics

21.1 Image-text quality filters

• CLIP-Score between image and caption (drop low).
• Aesthetic predictor (LAION-aesthetic).
• NSFW filter.
• Watermark detection.
• Face / minor detection (privacy + safety).

21.2 Image deduplication

• Perceptual hash (pHash) for near-dups.
• CLIP embedding clustering for semantic dedup.
• Watermark / template detection (stock photo overlays).

21.3 Video quality filters

• Motion magnitude (drop static).
• Aesthetic per-frame.
• OCR for text overlays (drop banners).
• Shot-cut detection (segment by scenes).
• Audio-presence filter.

21.4 Action-labeled data for VLAs

• Robot teleop with synchronized actions.
• Open-X-Embodiment standardized format.
• Per-embodiment normalization.
• Demo quality filter (drop low-success).

21.5 3D data normalization

• Canonicalize pose / scale.
• Per-dataset mesh format conversion.
• Material / texture standardization.

21.6 Synthetic image data

• Diffusion-augmented: vary captures with prompts.
• Render-to-image pipelines (Hypersim, Kubric).
• Mix synthetic + real for diversity.

22. Frontier 2025–2026

22.1 Synthetic data dominance

• Phi pattern continues to scale (Phi-4-mini at 3.8B competitive with much larger).
• R1-Distill demonstrates reasoning transfer.
• Open data labs (Tulu, OpenChat) heavily synthetic.

22.2 Continual learning data flywheel

• Production telemetry → labeling → retrain → deploy.
• Anthropic, OpenAI, Cursor all run this.
• Increasingly automated.

22.3 Verifiable-reward data

• Math / code with checkable answers.
• Drives R1-style RL gains.
• Curating large verifiable problem sets is the new bottleneck.

22.4 Multimodal data scarcity

• Text data: trillions of tokens; saturated for some tasks.
• Image data: billions of pairs; reasonable.
• Video data: scarce relative to demand.
• Robot data: scarcest; massive labeling cost.
• Drives synthetic / sim push.

22.5 Privacy-preserving data

• Differential privacy training.
• Federated learning.
• Synthetic data as proxy.
• Active research; production rare.

22.6 Open data benchmarks

• DataComp-LM, DataComp-1B: standardized data competition.
• Open frontier labs (AI2 Tulu / OLMo) lead reproducibility.
• Pushes the field toward measurable data improvements.

22.7 Open research questions

• Optimal data mixture given compute.
• Synthetic data scaling laws.
• Per-task optimal data composition.
• Continual learning without forgetting.
• Data-efficiency improvements.
• Privacy-preserving training at scale.

23. Production Stack 2026

Appendix A: Twenty-Five Things to Know

1. Data quality dominates architecture for frontier models.
2. Common Crawl is the foundation of nearly every LLM corpus.
3. C4 / The Pile / RefinedWeb / Dolma / DCLM / FineWeb lineage.
4. FineWeb-Edu: classifier-filtered > raw at small scale.
5. DCLM-baseline: small classifier filter beats hand rules.
6. Gopher heuristics for rule-based filtering.
7. MinHash + LSH for near-duplicate dedup.
8. SemDeDup for semantic dedup via embeddings.
9. Re-captioning is critical for image / video training (Sora, FLUX, DALL-E 3).
10. Phi-1: textbook synthetic data → small model wins.
11. R1-Distill: 800k long-CoT traces transfer reasoning.
12. Self-Instruct foundation for synthetic instructions.
13. Constitutional AI generates safety data via critique-revise.
14. Best-of-N rejection sampling (RAFT, RFT) cheap alternative to PPO.
15. HH-RLHF, UltraFeedback, Nectar are the canonical open preference datasets.
16. PRM800K, Math-Shepherd, OmegaPRM for process reward data.
17. NuminaMath: 860k math reasoning problems.
18. LAION-5B, COYO-700M, DataComp-1B, DFN-2B for image-text.
19. OBELICS for interleaved multimodal.
20. Open-X-Embodiment 1.4M trajectories for robot foundation.
21. Tesla's data engine: deploy → failure detect → label → retrain.
22. DoReMi for domain-mixture optimization.
23. dolma / datatrove / NeMo Curator for production curation.
24. WebDataset / Mosaic Streaming for training data loading.
25. Per-license, per-jurisdiction compliance is now mandatory.

Appendix B: Decision Tree — "How Do I Get Data?"

1. Pretraining LLM, frontier budget?
   → Common Crawl via FineWeb / DCLM pipeline + tier-1 sources + synthetic.
2. Pretraining LLM, modest budget?
   → FineWeb-Edu at small scale; FineWeb at larger.
3. SFT instruction-tuning?
   → Tulu / OpenHermes or Self-Instruct on top of frontier teacher.
4. RLHF preferences?
   → Iterative pairs from current model; label via Surge / in-house.
5. Reasoning RL?
   → NuminaMath + verifiable-reward problems (math, code).
6. Long-CoT distillation?
   → Sample from R1 / o3; filter by correct final answer.
7. Image generation training?
   → DataComp / DFN + re-caption with strong VLM.
8. Video generation training?
   → Panda-70M / HD-VG / InternVid + re-caption.
9. Robot foundation model?
   → Open-X-Embodiment + own teleop fleet.
10. Safety / alignment?
    → HH-RLHF / Constitutional + red-team gen + adversarial pairs.
11. Production deployment improvement?
    → Data-engine pattern: telemetry → label → retrain.

Appendix C: Year-by-Year Data Milestones

• 2018–19: Common Crawl + WikiText for BERT-era; Books3 controversially included in The Pile.
• 2020: The Pile (EleutherAI); C4 (T5); LAION-400M.
• 2021: LAION-5B; multilingual extensions (mC4); CLIP demonstrates scale + curation.
• 2022: Self-Instruct; InstructGPT demonstrates RLHF data flywheel; HH-RLHF released.
• 2023: Phi-1 synthetic-data winning; LLaMA recipe; RefinedWeb (Falcon); Vicuna pattern; Stanford Alpaca; OBELICS; Open-X-Embodiment announced.
• 2024: FineWeb / FineWeb-Edu (HuggingFace); DCLM-baseline (DataComp-LM); Dolma / OLMo data toolkit; Phi-3; UltraFeedback; LLaMA 3 15T tokens; PRM800K / Math-Shepherd PRM data; Tulu 3 + OpenHermes 2.5; NuminaMath; DROID (76k robot demos); Re-captioning standard for image / video gen (DALL-E 3, Sora).
• 2025: R1-Distill traces (800k); Phi-4; Llama 4 30T+; AgiBot World; Wan / Hunyuan video data recipes; verifiable-reward problem sets become a curation focus; Anthropic / OpenAI labeler ops scale to thousands; data-engine flywheel standard at frontier labs.
• 2026: synthetic data generation → filter → verify pipelines mature; multi-modal data scarcity increasingly addressed via sim + diffusion augmentation; per-jurisdiction compliance baked into pipeline; data-quality benchmarking standardized via DataComp-LM / DataComp-1B; reasoning data + verifiable rewards driving frontier RL.