A Computational Framework for Behavioral Assessment of LLM Therapists

• DOI: 10.48550/arxiv.2401.00820
• Publication Date: 2024-01-01
• Authors: Yu Ying Chiu; Ashish Sharma; Inna Wanyin Lin; Tim Althoff
• Keywords: Psychoeducation

The emergence of ChatGPT and other large language models (LLMs) has greatly increased interest in utilizing LLMs as therapists to support individuals struggling with mental health challenges. However, due the lack systematic studies, our understanding how LLM behave, i.e., ways which they respond clients, is significantly limited. Understanding their behavior across a wide range clients situations crucial accurately assess capabilities limitations high-risk setting health, where undesirable behaviors can lead severe consequences. In this paper, we propose BOLT, novel computational framework study conversational when employed therapists. We develop an in-context learning method quantitatively measure based on 13 different psychotherapy techniques including reflections, questions, solutions, normalizing, psychoeducation. Subsequently, compare against that high- low-quality human therapy, be modulated better reflect observed high-quality therapy. Our analysis GPT Llama-variants reveals these often resemble more commonly exhibited therapy rather than such offering higher degree problem-solving advice share emotions, typical recommendations. At same time, unlike upon clients' needs strengths. suggests despite ability generate anecdotal examples appear similar therapists, are currently not fully consistent care, thus require additional research ensure quality care.