Parachute: Evaluating Interactive Human-LM Co-writing Systems

• DOI: 10.48550/arxiv.2303.06333
• Publication Date: 2023-01-01
• Authors: Hua Shen; Tongshuang Wu
• Keywords: Artifact (error); Second language writing

A surge of advances in language models (LMs) has led to significant interest using LMs build co-writing systems, which humans and interactively contribute a shared writing artifact. However, there is lack studies assessing systems interactive settings. We propose human-centered evaluation framework, Parachute, for systems. Parachute showcases an integrative view interaction evaluation, where each aspect consists categorized practical metrics. Furthermore, we present with use case demonstrate how evaluate compare Parachute.