Hugging Face Releases FineWeb2: 8TB of Compressed Text Data with Almost 3T Words and 1000 Languages Outperforming Other Datasets Hugging Face researchers released FineWeb2, a dataset that sets a new benchmark for multilingual training resources. Spanning 8 terabytes of compressed text data—roughly equivalent to 3 trillion words—FineWeb 2 draws from 96 CommonCrawl snapshots collected between 2013 and April 2024. This dataset is the result of extensive processing and refinement using the Datatrove library, ensuring high-quality text content organized into 1,893 language-script pairs. Released under the permissive ODC-By 1.0 license, FineWeb 2 is accessible for both research and commercial applications, making it a versatile resource for the NLP community. Key Takeaways from FineWeb2 ✅ FineWeb2 comprises 8TB of compressed text data, equivalent to nearly 3 trillion words, sourced from 96 CommonCrawl snapshots spanning 2013 to 2024. ✅ It covers over 1,000 languages, organized into 1,893 language-script pairs, supporting research and applications in low-resource languages. ✅ Processed using the Datatrove library, the dataset is meticulously deduplicated and filtered to ensure high quality and relevance. ✅ It outperforms leading multilingual datasets like CC-100, mC4, CulturaX, and HPLT on diverse tasks and even rivals some single-language specialized datasets. ✅ Available under the ODC-By 1.0 license, FineWeb 2 is suitable for both research and commercial use. Read the full article here: marktechpost.com/2024/12/08/… Dataset: huggingface.co/datasets/Hugg… @huggingface