Datasets: A Community Library for Natural Language Processing

• DOI: 10.18653/v1/2021.emnlp-demo.21
• Publication Date: 2021-12-17T03:56:42Z
• Authors: Quentin Lhoest; Albert Villanova ...; Yacine Jernite; Abhishek Thakur; Patrick von Platen; Suraj Patil; Julien Chaumond; Mariama Drame; Julien Plu; Lewis Tunstall; Joe Davison; Mario Šaško; Gunjan Chhablani; Bhavitvya Malik; Simon Brandeis; Teven Le Scao; Victor Sanh; Canwen Xu; Nicolas Patry; Angelina McMillan...; Philipp Schmid; Sylvain Gugger; Clément Delangue; Théo Matussière; Lysandre Debut; Stas Bekman; Pierric Cistac; Thibault Goehringer; Victor Mustar; François Lagunas; Alexander Rush; Thomas Wolf
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology; Computation and Language (cs.CL); 01 natural sciences; 0105 earth and related environmental sciences

The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.<br/>EMNLP Demo 2021<br/>