HMM for discovering decision-making dynamics using reinforcement learning experiments

• DOI: 10.1093/biostatistics/kxae033
• Publication Date: 2024-09-04T01:56:01Z
• Authors: Xingche Guo; Donglin Zeng; Yuanjia Wang
• Keywords: Methodology (stat.ME); FOS: Computer and information sciences; Computer Science - Machine Learning; Statistics - Machine Learning; Applications (stat.AP); Machine Learning (stat.ML); Statistics - Applications; Statistics - Methodology; Machine Learning (cs.LG)

Major depressive disorder (MDD), a leading cause of years life lived with disability, presents challenges in diagnosis and treatment due to its complex heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as behavioral marker for MDD. To measure processing, patients perform computer-based tasks involve making choices or responding stimulants are associated different outcomes, such gains losses the laboratory. Reinforcement learning (RL) models fitted extract parameters various aspects (e.g. sensitivity) characterize how make decisions tasks. Recent findings suggest inadequacy characterizing solely based on single RL model; instead, there be switching decision-making processes between multiple strategies. An important scientific question is dynamics strategies affect ability individuals Motivated by probabilistic task within Establishing Moderators Biosignatures Antidepressant Response Clinical Care (EMBARC) study, we propose novel RL-HMM (hidden Markov model) framework analyzing reward-based decision-making. Our model accommodates strategy two distinct approaches under an HMM: subjects opting random choices. We account continuous state space allow time-varying transition probabilities HMM. introduce computationally efficient Expectation-maximization (EM) algorithm parameter estimation use nonparametric bootstrap inference. Extensive simulation studies validate finite-sample performance our method. apply approach EMBARC study show MDD less engaged compared healthy controls, engagement brain activities negative circuitry during emotional conflict task.