2026-10-16 — views
NVIDIA and AI4Bharat Release 2.8 Trillion Indic Tokens and Open Pipeline
Read this because This release addresses the critical data scarcity in Indian languages by providing a massive, auditable synthetic corpus and the reusable tools used to create it.
NVIDIA and AI4Bharat release a 2.8 trillion token Indic pretraining corpus and an open GPU-accelerated data generation pipeline for 22 languages.
Massive Indic Corpus Release
NVIDIA and AI4Bharat have introduced Nemotron-pretraining-Indic-v1, a pretraining corpus containing approximately 2.8 trillion tokens across 22 Indian languages. This release, developed in collaboration with IIT Madras, targets the first languages for 97% of Indians, covering nearly 1.2 billion citizens according to the 2011 Census. The corpus is designed to address the significant data deficit in Indian language technology by providing a substantial new resource for model builders. It includes a broad mix of curated English material, such as mathematics, code, and question-answer content, translated into Indian languages to help models explain technical concepts and solve problems.
Open Data Generation Pipeline
Alongside the data, the team is releasing an open, GPU-accelerated data generation pipeline. This pipeline captures the full generation recipe, including prompts, large language model workflows, and quality checks, allowing developers to adapt the process for their own source materials. The workflow uses NeMo Data Designer to run translation, quality assessment, and targeted repair. Every record in the corpus traces back to a curated source document, ensuring that synthetic text remains grounded in real information and allowing for full provenance auditing.
Workflow Details and Token Counts
The corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted using the Gemma-4 tokenizer. The data covers 34 language-script configurations. For twelve languages, including Bengali, Hindi, and Tamil, the workflow involves translating English documents, assessing quality using a rubric inspired by Multidimensional Quality Metrics, and repairing low-scoring outputs. Only outputs scoring at least 75 out of 100 are retained. A subset of these accepted translations is then converted into Latin script to create romanized data. This two-step approach was found to produce fewer hallucinations and less residual English than direct generation.
For the remaining ten languages, such as Sanskrit and Santali, the team uses Indic Translate for document-level translation. Gemma-4-31B-it then assesses these translations. For longer documents in Santali, overlapping windows are translated and reconstructed. This workflow produces native-script data only, without feedback-based rewriting. The corpus also complements naturally authored Indic text, which remains essential for local knowledge and cultural context. Continued pretraining for Nemotron 3.5 Super VL previously added 10 billion Indic tokens from a subset of this corpus, demonstrating the utility of this data for model improvement.
Practitioner note
Developers should utilize the provided NeMo Data Designer workflow to audit and adapt the translation and quality assessment pipelines for their specific language needs. You can directly use the 2.8 trillion token corpus for pretraining or generate additional synthetic data by applying the open prompts and repair logic to new source documents.