WHAT IS LLM TRAINING DATA?

What is LLM Training Data?

LLM training data is plain text converted into token sequences (subword units). GPT tokens average ~4 chars each; a 1M-word book = ~250K tokens. LLMs learn statistical patterns from billions of tokens to predict the next token.

GPT-4 (2023) trained on ~10T tokens. GPT-3.5 (~300B params) trained on ~500B tokens. Training cost scales with data size: 10x more tokens = 10x longer training, but ~2-3% accuracy improvement (logarithmic returns).

Primary Data Sources (2026)

Common Crawl

Public web snapshot (cleaned). Released quarterly. ~50-100 billion pages per snapshot. Largest source for LLM training. Raw quality is low (spam, boilerplate); requires aggressive filtering.

GitHub Public Repositories

~500M public repos, ~100B source code files. High signal-to-noise: code is structured, semantically correct, useful for code generation models.

Books & Academic Papers

Gutenberg Project (~70K public domain books), arXiv (2.3M papers). High quality but smaller corpus (< 1TB uncompressed). Used for reasoning and factual knowledge.

Reddit, Twitter, Discourse

~20TB/year of social text. Lower quality than books, but captures conversational patterns, slang, current events. Reddit data (2005-2023) is ~500GB compressed.

Data Composition & Filtering (2026)

Typical composition: Common Crawl 50%, code 10%, books 15%, social 20%, other 5%.

Filtering pipeline: Raw → deduplication (remove exact duplicates) → quality scoring (perplexity, language detection) → toxicity filtering (Jigsaw Perspective API, 0.5-0.7 threshold) → PII removal (regex + ML) → domain-specific filtering.

Result: 60-80% of raw crawl data discarded. Quality filtering is the difference between ChatGPT (good) and unfiltered LLaMA (contains toxicity, bias).

Bias in Training Data

Internet text reflects historical biases. Solutions:

• Debiasing (hard): Reweight underrepresented groups, synthetic data generation, adversarial debiasing
• Documentation (meta): Datasheet for Datasets approach. Document sources, composition, known biases
• RLHF (practice): Fine-tune with human feedback to reduce harmful outputs

OpenAI's approach: filter toxic content, then RLHF to align with human values. Result: ChatGPT safer than GPT-3 despite same base training.

Scale Economics (2026 Pricing)

Training cost = data sourcing + compute + storage. Data sourcing is 10-20% of total training budget for large models. A reliable crawl to collect 1TB of web data costs ~$10-50K in infrastructure + proxies.

Enterprises like Anthropic invest heavily in data quality. Smaller teams rely on public datasets (Common Crawl, HuggingFace), which is why proprietary data is competitive advantage.