MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

• DOI: 10.18653/v1/2021.findings-emnlp.253
• Publication Date: 2021-12-28T12:24:05Z
• Authors: Matthew Matero; Nikita Soni; Niranjan Balasubr...; H. Andrew Schwartz
• Keywords: Context model

Much of natural language processing is focused on leveraging large capacity models, typically trained over single messages with a task predicting one or more tokens. However, modeling human at higher-levels context (i.e., sequences messages) under-explored. In stance detection and other social media tasks where the goal to predict an attribute message, we have contextual data that loosely semantically connected by authorship. Here, introduce Message-Level Transformer (MeLT) – hierarchical message-encoder pre-trained Twitter applied prediction. We focus prediction as benefiting from knowing message sequence previous messages). The model using variant masked-language modeling; instead tokens, it seeks generate entire masked (aggregated) vector via reconstruction loss. find applying this message-level transformer downstream achieves F1 performance 67%.