Open SourceHugging Face

Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

#indic#pretraining#open data#language models#nvidia

English

Hugging Face and NVIDIA have announced the release of a substantial pretraining corpus comprising 2.8 trillion tokens across 22 Indian languages. This open data generation pipeline aims to enhance the availability of training material for Indic language models, addressing the data scarcity issue in Indian language technology.

中文

Hugging Face 和 NVIDIA 宣布发布一个包含 2.8 万亿个标记的预训练语料库，涵盖 22 种印度语言。这个开放的数据生成管道旨在增加印度语言模型的训练材料的可用性，以解决印度语言技术中的数据稀缺问题。