Updated benchmarking of variant effect predictors using deep mutational scanning

• DOI: 10.15252/msb.202211474
• Publication Date: 2023-06-13T13:11:03Z
• Authors: Benjamin J Livesey; Joseph A Marsh
• Keywords: Benchmarking; Benchmark (surveying)

Abstract The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top‐performing VEPs are unsupervised methods including EVE, DeepSequence and ESM‐1v, a language model that ranked first overall. However, the strong recent supervised in particular VARITY, shows developers taking circularity bias issues seriously. We also assess DMS discriminating between known pathogenic putatively benign missense variants. Our findings mixed, demonstrating some datasets perform exceptionally at classification, others poor. Notably, observe striking correlation VEP agreement identifying clinically relevant variants, strongly supporting validity rankings utility independent benchmarking.