Evaluating Contextual Embeddings and their Extraction Layers for Depression Assessment

• DOI: 10.48550/arxiv.2112.13795
• Publication Date: 2021-01-01
• Authors: Matthew Matero; Albert M. Hung; H. Andrew Schwartz
• Keywords: Concatenation (mathematics); Depression; Degree (music)

Recent works have demonstrated ability to assess aspects of mental health from personal discourse. At the same time, pre-trained contextual word embedding models grown dominate much NLP but little is known empirically on how best apply them for assessment. Using degree depression as a case study, we do an empirical analysis which off-the-shelf language model, individual layers, and combinations layers seem most promising when applied human-level tasks. Notably, find RoBERTa effective and, despite standard in past work suggesting second-to-last or concatenation last 4 layer 19 (sixth-to last) at least good 23 using 1 layer. Further, multiple distributing across second half (i.e. Layers 12+), rather than 4, 24 yielded accurate results.