NVIDIA and AI4Bharat Release 2.8 Trillion Indic Tokens and Open Pipeline
Read this because This release addresses the critical data scarcity in Indian languages by providing a massive, auditable synthetic corpus and the reusable tools used to create it.
NVIDIA and AI4Bharat release a 2.8 trillion token Indic pretraining corpus and an open GPU-accelerated data generation pipeline for 22 languages.