Open-Source Datasets for LLM Training

Open-Source Datasets for LLM Training

Training effective Large Language Models (LLMs)requires substantial high-quality data. Fortunately, several open-source datasets are available, each with its own strengths and weaknesses. The choice depends heavily on your specific needs regarding LLM task, size constraints, and computational resources.

Below, we present a curated selection of these datasets, categorized for ease of use. Remember to always check the individual dataset's licensing terms before use. Preprocessing the data is crucial for optimal results, as discussed in the following sections.

Text-Based Datasets

• Common Crawl: A massive corpus of raw web data (terabytes), used to train models like GPT-3 and LLaMA. Learn more.
• RefinedWeb: A cleaned and deduplicated subset of Common Crawl, offering higher quality but still immense scale (5 trillion tokens, 600 billion publicly available). Used in the training of Falcon-40B. Read more about RefinedWeb's cleaning process.
• The Pile: An 800 GB corpus combining data from 22 diverse sources, improving model generalization. Used in training GPT-Neo, LLaMA, and OPT. Access The Pile.
• C4 (Colossal Clean Crawled Corpus): A 750 GB English corpus derived from Common Crawl, focusing on clean natural language data. Used in pre-training MPT-7B and T5. See the original C4 paper.
• BookCorpus: Text from 11,000 unpublished books, totaling 985 million words. Used in training LLMs such as RoBERTA, XLNET, and T5. View the BookCorpus paper.
• Wikipedia: Cleaned text data from Wikipedia in various languages. The English version alone contains 19.88 GB of data. Used to train models like Roberta, XLNet, and LLaMA. Access Wikipedia data dumps.
• Red Pajama: An open-source effort replicating the LLaMA dataset (1.2 trillion tokens)using data from Common Crawl, C4, GitHub, books, and more. Used in training MPT-7B and OpenLLaMA. Explore the Red Pajama project.

Code-Based Datasets

• Starcoder Data: A 783 GB dataset of code in 86 programming languages from GitHub and Jupyter Notebooks (250 billion tokens). Used to train Salesforce CodeGen, Starcoder, and StableCode. Learn about Starcoder.

Multilingual Datasets

• ROOTS: A 1.6 TB multilingual dataset in 59 languages, heavily deduplicated and filtered. Used to train the BLOOM language model. Read the ROOTS paper by the BigScience team.

This list is not exhaustive, and the landscape of available open-source datasets is constantly evolving. Further exploration of repositories like Hugging Face is recommended to locate datasets suitable for specific tasks and languages.