2026-10-01 — views
Scaling Laws for Wild AI-Generated Web Text Reveal Token Value Shifts
Read this because AI engineers must adjust pretraining strategies as AI-generated content saturates web corpora, impacting model performance on human data.
Research shows adding AI tokens to pretraining data initially helps but quickly harms human text loss, prompting new scaling laws.
The Rise of Wild AI Text
Web text constitutes the majority of pretraining data for language models, and its composition is shifting rapidly. Recent analysis of web data from mid-2026 reveals that 27.5% of tokens in June were labeled as AI-generated by Pangram, a figure that rose to 31.1% by August. Unlike synthetic datasets or controlled model-collapse experiments, this “wild” AI text originates from numerous models, is written for human readers, and enters pretraining corpora without labels. This uncontrolled influx raises critical questions about how such data affects the training of large language models.
Experimental Findings on Token Value
To understand the impact of this phenomenon, researchers pretrained 800 language models, systematically varying the ratio of added AI tokens to human tokens. They then fitted scaling laws to held-out losses on both human and AI-generated text. The results indicate that for data-starved models, introducing AI tokens initially lowers the loss on human text. However, this benefit saturates quickly as more AI tokens are added and eventually reverses into significant harm.
In contrast, for models trained on high budgets of human text, the addition of AI tokens raises loss almost immediately. Conversely, adding the same number of fresh human tokens continues to lower the loss. These dynamics demonstrate that the value of an AI token is not static; it can change sign depending on the training context. Existing scaling laws, such as those by Hoffman et al. (2022), fail to predict this complex behavior.
A New Scaling Law and Recommendations
The study proposes a new scaling law that includes separate benefit and harm terms. This model allows the value of an AI token to change sign while still reducing to the Chinchilla scaling law in the absence of AI text. When fitted on smaller models, this new law predicts the effect of AI text on held-out human-text loss for models up to 3.6 times larger, achieving 41% lower error than the best existing law across all AI ratios.
The authors provide specific recommendations for practitioners. When the target is human text, they advise filtering AI text from the training corpus. Alternatively, developers should repeat human text before expanding the training dataset with AI-generated web text. Additionally, validation loss should be reported separately for human and AI text. The study notes that AI text remains valuable when the target is AI text. To support further research, the team released WildAI, an 83B-token corpus containing AI, topic, and format labels, along with the 800 trained models and code.
Practitioner note
Developers should implement strict filtering for AI-generated content when optimizing for human text performance, or alternatively, oversample existing human data before incorporating synthetic web text. Always monitor validation loss on human and AI text separately to detect early signs of performance degradation caused by AI token saturation.