Quick verdict
Common Crawl is the backbone of modern LLM pretraining but requires engineering investment to use responsibly. COCA and BNC serve academic linguistic research with structured, representative designs that web corpora cannot replicate. OPUS is the definitive free resource for multilingual and translation work. Wikipedia dumps offer the best combination of quality and accessibility for encyclopedic domain training. Matc
Common Crawl - Best for Large-Scale Model Pretraining
Common Crawl is the largest publicly accessible web corpus in existence, consisting of petabytes of raw crawl data collected since 2008 across billions of web pages. It forms the basis of training datasets used by GPT, LLaMA, and most other large language models, either directly or through filtered derivatives like C4 and the Pile. The data is available free via Amazon S3, but raw Common Crawl requires substantial preprocessing to remove boilerplate, deduplicate, filter for language quality, and handle encoding issues. For teams building foundation models or large-scale pretraining experiments, Common Crawl is the starting point - but budget significant engineering effort for data cleaning before training.
Check price on Amazon →The right corpus dataset determines the quality of your NLP model, linguistic research, or language learning tool. These five corpus resources cover every scale from academic research to production AI training.
A corpus is the foundation of any serious NLP project, linguistic research study, or AI language model. The dataset you train on or analyze from shapes every downstream output – a poorly constructed or unrepresentative corpus produces biased, brittle models regardless of architecture quality. The five resources below cover the spectrum from raw web-scale text to carefully curated academic corpora, with honest trade-offs between scale, licensing, and data quality.
| Corpus | Size | Language Coverage | Access | Best For |
|—|—|—|—|—|
| Common Crawl | Petabyte-scale | Multilingual | Free (S3) | Large-scale LLM pretraining |
| COCA (Corpus of Contemporary American English) | 1B+ words | English | Subscription | American English research |
| British National Corpus (BNC) | 100M words | English | Free/licensed | Balanced academic reference |
| OPUS | Billions of tokens | 500+ languages | Free | Multilingual and translation models |
| Wikipedia Text Dumps | ~20GB compressed | 300+ languages | Free | Clean general-knowledge pretraining |
How we picked
We compare every pick against the field on real specifications, certifications, and aggregated owner reviews. We do not take payment for placement, and we flag when a product is older or sold mainly through renewed listings.
Top picks compared
| Pick | Best for | Score | |
|---|---|---|---|
| Common Crawl - Best for Large-Scale Model Pretraining | Check price | ||
| COCA (Corpus of Contemporary American English) - Best for American English Lingu | Check price | ||
| British National Corpus (BNC) - Best for Balanced Reference Corpus | Check price | ||
| OPUS - Best for Multilingual and Translation Models | Check price | ||
| Wikipedia Text Dumps - Best for Clean General-Knowledge Pretraining | Check price |
Our picks up close
Common Crawl - Best for Large-Scale Model Pretraining
Common Crawl is the largest publicly accessible web corpus in existence, consisting of petabytes of raw crawl data collected since 2008 across billions of web pages. It forms the basis of training datasets used by GPT, LLaMA, and most other large language models, either directly or through filtered derivatives like C4 and the Pile. The data is available free via Amazon S3, but raw Common Crawl requires substantial preprocessing to remove boilerplate, deduplicate, filter for language quality, and handle encoding issues. For teams building foundation models or large-scale pretraining experiments, Common Crawl is the starting point - but budget significant engineering effort for data cleaning before training.
COCA (Corpus of Contemporary American English) - Best for American English Lingu
COCA is the largest freely accessible corpus of American English, with over one billion words of text from spoken, fiction, popular magazines, newspapers, and academic sources collected from 1990 to the present. It is maintained by Brigham Young University and provides a web-based query interface that allows frequency analysis, collocate searches, and genre comparison without requiring local download or processing. COCA is the reference standard in American English linguistics research and is widely used in lexicography, language teaching, and corpus-based grammar studies. The full corpus with download access requires a paid subscription, but the web interface is free for most research uses.
British National Corpus (BNC) - Best for Balanced Reference Corpus
The British National Corpus is a 100-million-word snapshot of written and spoken British English from the late 20th century, assembled with deliberate balance across text types, genres, and registers. Its balanced design makes it the preferred reference corpus for studies requiring proportional genre representation rather than internet-skewed distributions. The BNC XML edition is freely downloadable for academic and non-commercial use. While it has not been updated since the 1990s sampling period, its controlled construction means it remains a valuable reference for synchronic studies of British English and as a benchmark against which contemporary corpora are compared.
OPUS - Best for Multilingual and Translation Models
OPUS is a collection of translated text corpora from the internet, compiled by researchers at the University of Helsinki. It covers over 500 languages and includes parallel corpora (texts aligned across language pairs) from sources including European Parliament proceedings, OpenSubtitles movie dialogue, Wikipedia, and legal documents. For training multilingual translation models, cross-lingual embeddings, or low-resource language systems, OPUS is the primary free resource available at meaningful scale. The data quality varies significantly by source and language pair - high-resource European language pairs from parliamentary sources are clean and reliable, while low-resource pairs from web-scraped sources require more careful filtering.
Wikipedia Text Dumps - Best for Clean General-Knowledge Pretraining
Wikipedia text dumps are one of the cleanest and most consistently structured large-scale text resources available for free. Wikimedia Foundation releases monthly dumps in XML format for all 300+ language editions, covering tens of gigabytes of encyclopedic text with internal link structure, article metadata, and revision history. The encyclopedic register, consistent article structure, and volunteer editorial quality make Wikipedia one of the cleanest portions of most LLM training datasets. The WikiExtractor tool simplifies extraction to plain text. For domain-specific models in factual, encyclopedic, or educational contexts, Wikipedia dumps are often the highest-quality data-per-byte available without licensing costs.
Before you buy
License compatibility with your use case
Free access does not mean unrestricted use. Academic-only licenses, commercial restrictions, and evolving legal interpretations around training data mean you must verify licensing for any corpus before using it in a commercial product.
Domain match to your application
A general web corpus is not ideal for training a specialized legal or medical NLP system. Seek corpora that match the domain, register, and user language patterns of your actual application. In-domain data quality beats raw scale in most fine-tuning scenarios.
Deduplication and quality filtering
Web-scale corpora contain enormous quantities of duplicated, low-quality, and machine-generated text. The preprocessing pipeline - deduplication, perplexity filtering, language identification, and content filtering - is as important as the raw data source in determining model quality.
Language balance for multilingual systems
Most web corpora are dominated by English and a handful of high-resource European languages. If your application requires balanced multilingual performance, intentionally oversample lower-resource languages during training rather than relying on natural distribution.
The wrap-up
Common Crawl is the backbone of modern LLM pretraining but requires engineering investment to use responsibly. COCA and BNC serve academic linguistic research with structured, representative designs that web corpora cannot replicate. OPUS is the definitive free resource for multilingual and translation work. Wikipedia dumps offer the best combination of quality and accessibility for encyclopedic domain training. Matc
Quick answers
'A corpus (plural: corpora) is a large, structured collection of text or speech used for linguistic analysis, language model training, or computational research. Corpora can be general-purpose (covering broad language use) or domain-specific (legal, medical, conversational). In NLP, the quality and diversity of the training corpus is one of the primary determinants of model performance and bias characteristics.'
A balanced corpus is a fixed-size collection assembled to represent different text types, genres, or time periods in deliberate proportions - the British National Corpus is a classic example. A monitor corpus is a continuously updated collection that grows over time, designed to track how language evolves. Monitor corpora are more useful for studying contemporary language change; balanced corpora are better for controlled comparative studies.
Common Crawl data is publicly available and has been used to train many commercial language models, but it comes with important caveats. The crawl contains copyrighted material, and the legal landscape around training data and copyright is actively evolving in multiple jurisdictions. Always consult legal counsel before using web-scraped corpus data for commercial model training, particularly in the EU where the AI Act and database directive apply.
Tom Reeves has reviewed consumer electronics for over a decade, with a focus on televisions, monitors, laptops, and smart home devices. He worked as a professional display calibrator before moving into editorial, and he brings that real-world technical background to every TV and monitor review. At TheTestedHub, Tom covers display calibration, computer monitors, laptops and 2-in-1s, smart home platforms, home theater setups, and HDR performance.