Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights

• DOI: 10.48550/arxiv.2407.12826
• Publication Date: 2024-07-02
• Authors: Elphin Tom Joe; Sai Dileep Koneru; Christine Kirchhoff
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computation and Language (cs.CL)

In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve groups domain experts who manually review synthesize vast amounts literature. The exponential growth scientific literature recent advances in LLMs provide an opportunity complementing these traditional with new age tools. We assess efficacy GPT-4o do on sample from dataset created by Global Adaptation Mapping Initiative (GAMI) where check accuracy climate change adaptation related feature extraction across three levels expertise. Our results indicate that while can achieve high low-expertise like geographic location identification, their performance intermediate high-expertise tasks, as stakeholder identification depth response, is less reliable. findings motivate need designing utilize strengths models also providing refinements improve