ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

• DOI: 10.48550/arxiv.2303.13648
• Publication Date: 2023-01-01
• Authors: Haoran Wu; Wenxuan Wang; Yuxuan Wan; Wenxiang Jiao; Michael R. Lyu
• Keywords: Benchmark (surveying)

ChatGPT is a cutting-edge artificial intelligence language model developed by OpenAI, which has attracted lot of attention due to its surprisingly strong ability in answering follow-up questions. In this report, we aim evaluate on the Grammatical Error Correction(GEC) task, and compare it with commercial GEC product (e.g., Grammarly) state-of-the-art models GECToR). By testing CoNLL2014 benchmark dataset, find that performs not as well those baselines terms automatic evaluation metrics $F_{0.5}$ score), particularly long sentences. We inspect outputs goes beyond one-by-one corrections. Specifically, prefers change surface expression certain phrases or sentence structure while maintaining grammatical correctness. Human quantitatively confirms suggests produces less under-correction mis-correction issues but more over-corrections. These results demonstrate severely under-estimated could be promising tool for GEC.