DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels

• DOI: 10.7554/elife.63377
• Publication Date: 2021-09-02T12:00:56Z
• Authors: James P Bohnslav; Nivanthika K Wima...; Kelsey J Clausing; Yu Y Dai; David A Yarmolinsky; Tomás Cruz; Adam D Kashlan; M Eugenia Chiappe; Lauren L Orefice; Clifford J Woolf; Christopher D Harvey
• Keywords: Male; QH301-705.5; Science; Video Recording; Walking; Motor Activity; behavior analysis; computer vision; Pattern Recognition, Automated; Mice; 03 medical and health sciences; Image Processing, Computer-Assisted; Animals; Humans; Biology (General); Social Behavior; 0303 health sciences; Q; R; deep learning; Reproducibility of Results; Grooming; Mice, Inbred C57BL; Kinetics; Drosophila melanogaster; Medicine; Female; Neural Networks, Computer; Supervised Machine Learning; Neuroscience

Videos of animal behavior are used to quantify researcher-defined behaviors of interest to study neural function, gene mutations, and pharmacological therapies. Behaviors of interest are often scored manually, which is time-consuming, limited to few behaviors, and variable across researchers. We created DeepEthogram: software that uses supervised machine learning to convert raw video pixels into an ethogram, the behaviors of interest present in each video frame. DeepEthogram is designed to be general-purpose and applicable across species, behaviors, and video-recording hardware. It uses convolutional neural networks to compute motion, extract features from motion and images, and classify features into behaviors. Behaviors are classified with above 90% accuracy on single frames in videos of mice and flies, matching expert-level human performance. DeepEthogram accurately predicts rare behaviors, requires little training data, and generalizes across subjects. A graphical interface allows beginning-to-end analysis without end-user programming. DeepEthogram’s rapid, automatic, and reproducible labeling of researcher-defined behaviors of interest may accelerate and enhance supervised behavior analysis. Code is available at: https://github.com/jbohnslav/deepethogram.