What Data Fuels Large Language Models?

What Data Fuels Large Language Models?

Large language models (LLMs)are trained on massive datasets, predominantly consisting of text scraped from publicly available internet resources. This data comes in truly enormous volumes; we're talking billions, even trillions, of words. A prime example of such a dataset is the Common Crawl, a massive collection of web pages numbering in the billions. Humza Naveed et al. (2023) provide a detailed overview of this data-intensive process. But the data isn't limited to just web pages.

Beyond web-scraped text, LLMs often incorporate other data types to enhance their capabilities. Code from public repositories like GitHub is frequently included to improve the ability of the models to understand and generate programming code. The inclusion of diverse data sources is essential for making LLMs more versatile. For instance, incorporating structured data can enhance their reasoning abilities and improve performance on specific tasks. The scale of the data itself is a critical factor, directly influencing the model's performance and the breadth of its knowledge. Kaplan et al. (2020) highlight the importance of scale in their research on scaling laws for neural language models.

However, this data-intensive approach has significant implications. Web-scraped data is inherently messy; accuracy varies widely, and bias and misinformation are common occurrences. The sheer quantity of data makes quality control extremely challenging. Moreover, a significant concern is the presence of personal data within these datasets, raising important privacy and data protection issues – a crucial aspect highlighted in the Global Privacy Assembly Resolution on Generative Artificial Intelligence Systems (2023).