Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

• DOI: 10.18653/v1/2022.findings-emnlp.329
• Publication Date: 2023-08-04T20:21:02Z
• Authors: Bernal Jimenez Gu...; Nikolas McNeal; Clayton Washington; You Chen; Lang Li; Huan Sun; Yu Su
• Keywords: Biomedicine

Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are highly appealing for domains biomedicine that feature high and diverse demands of technologies but also data annotation costs. In this paper, we present the first systematic comprehensive study to compare few-shot performance with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction (IE) tasks: named entity recognition relation extraction. We follow true setting avoid overestimating models’ by model selection over a large validation set. optimize GPT-3’s known techniques contextual calibration dynamic example retrieval. However, our results show still significantly underperforms compared simply PLM. addition, yields gains in accuracy when more training becomes available. More in-depth analyses further reveal issues may be detrimental IE tasks general. Given cost experimenting GPT-3, hope provides helpful guidance researchers practitioners towards practical solutions small before better is available IE.