Revisiting Acceptability Judgements

• DOI: 10.48550/arxiv.2305.14091
• Publication Date: 2023-01-01
• Authors: Hai Hu; Ziyin Zhang; Weifang Huang; Jackie Yan‐Ki Lai; Aini Li; Yina Ma; Jiahui Huang; Peng Zhang; Rui Wang
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computation and Language (cs.CL)

In this work, we revisit linguistic acceptability in the context of large language models. We introduce CoLAC - Corpus Linguistic Acceptability Chinese, first large-scale dataset for a non-Indo-European language. It is verified by native speakers and that comes with two sets labels: linguist label crowd label. Our experiments show even largest InstructGPT model performs only at chance level on CoLAC, while ChatGPT's performance (48.30 MCC) also much below supervised models (59.03 human (65.11 MCC). Through cross-lingual transfer fine-grained analysis, provide detailed analysis predictions demonstrate time knowledge can be transferred across typologically distinct languages, as well traced back to pre-training. publicly available \url{https://github.com/huhailinguist/CoLAC}.