training classifiers with natural language explanations

• DOI: 10.48550/arxiv.1805.03818
• Publication Date: 2018-01-01
• Authors: Percy Liang; Martin Bringmann; Paroma Varma; Christopher Ré; Braden Hancock; Stephanie Wang
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology; Computation and Language (cs.CL)

Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100$\times$ faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.<br/>ACL 2018; v4 adds references and link to code<br/>