Scaling Laws for Wild AI-Generated Web Text Reveal Token Value Shifts
Read this because AI engineers must adjust pretraining strategies as AI-generated content saturates web corpora, impacting model performance on human data.
Research shows adding AI tokens to pretraining data initially helps but quickly harms human text loss, prompting new scaling laws.