Revisiting Acceptability Judgements
FOS: Computer and information sciences
Computer Science - Computation and Language
Artificial Intelligence (cs.AI)
Computer Science - Artificial Intelligence
Computation and Language (cs.CL)
DOI:
10.48550/arxiv.2305.14091
Publication Date:
2023-01-01
AUTHORS (9)
ABSTRACT
In this work, we revisit linguistic acceptability in the context of large language models. We introduce CoLAC - Corpus Linguistic Acceptability Chinese, first large-scale dataset for a non-Indo-European language. It is verified by native speakers and that comes with two sets labels: linguist label crowd label. Our experiments show even largest InstructGPT model performs only at chance level on CoLAC, while ChatGPT's performance (48.30 MCC) also much below supervised models (59.03 human (65.11 MCC). Through cross-lingual transfer fine-grained analysis, provide detailed analysis predictions demonstrate time knowledge can be transferred across typologically distinct languages, as well traced back to pre-training. publicly available \url{https://github.com/huhailinguist/CoLAC}.
SUPPLEMENTAL MATERIAL
Coming soon ....
REFERENCES ()
CITATIONS ()
EXTERNAL LINKS
PlumX Metrics
RECOMMENDATIONS
FAIR ASSESSMENT
Coming soon ....
JUPYTER LAB
Coming soon ....