Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements

• DOI: 10.1167/jov.21.7.9
• Publication Date: 2021-07-15T15:02:29Z
• Authors: Zachary J. Cole; Karl M. Kuntzelman; Michael D. Dodd; Matthew R. Johnson
• Keywords: Hyperparameter; Time line

Previous attempts to classify task from eye movement data have relied on model architectures designed emulate theoretically defined cognitive processes and/or that been processed into aggregate (e.g., fixations, saccades) or statistical fixation density) features. Black box convolutional neural networks (CNNs) are capable of identifying relevant features in raw and minimally images, but difficulty interpreting these has contributed challenges generalizing lab-trained CNNs applied contexts. In the current study, a CNN classifier was used two datasets (Exploratory Confirmatory) which participants searched, memorized, rated indoor outdoor scene images. The Exploratory dataset tune hyperparameters model, resulting architecture retrained, validated, tested Confirmatory dataset. were formatted timelines (i.e., x-coordinate, y-coordinate, pupil size) To further understand informational value each component data, timeline image broken down subsets with one more components systematically removed. Classification consistently outperformed data. Memorize condition most often confused Search Rate. Pupil size least uniquely informative when compared x- y-coordinates. general pattern results for replicated Overall, present study provides practical reliable black solution classifying