Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, Zhou. Findings of the Association for Computational Linguistics: EMNLP 2020.

We present a method that learns word embedding for Twitter sentiment classification in this paper.Most existing algorithms learning continuous representations typically only model the syntactic context of words but ignore text.This is problematic analysis as they usually map with similar opposite polarity, such good and bad, to neighboring vectors.We address issue by sentimentspecific (SSWE), which encodes information representation words.Specifically, we develop three neural networks...

We introduce a deep memory network for aspect level sentiment classification.Unlike feature-based SVM and sequential neural models such as LSTM, this approach explicitly captures the importance of each context word when inferring polarity an aspect.Such degree text representation are calculated with multiple computational layers, which is attention model over external memory.Experiments on laptop restaurant datasets demonstrate that our performs comparable to state-of-art feature based...

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of pre-trained language models. In this paper, we target on revisiting Chinese models examine their effectiveness in a non-English release model series community. We also propose simple but effective called MacBERT, which improves upon RoBERTa several ways, especially masking strategy that...

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of pre-trained language models.In this paper, we aim first introduce whole word masking (wwm) strategy for Chinese BERT, along with a series models.Then also propose simple but effective model called MacBERT, which improves upon RoBERTa in several ways.Especially, new MLM as correction...

Cloze-style reading comprehension is a representative problem in mining relationship between document and query. In this paper, we present simple but novel model called attention-over-attention reader for better solving cloze-style task. The proposed aims to place another attention mechanism over the document-level induces “attended attention” final answer predictions. One advantage of our that it simpler than related works while giving excellent performance. addition primary model, also...

Duyu Tang, Bing Qin, Ting Liu. Proceedings of the 53rd Annual Meeting Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

We propose learning sentiment-specific word embeddings dubbed sentiment in this paper. Existing embedding algorithms typically only use the contexts of words but ignore texts. It is problematic for analysis because with similar opposite polarity, such as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">good</i> and xmlns:xlink="http://www.w3.org/1999/xlink">bad</i> , are mapped to neighboring vectors. address issue by encoding information...

In this paper, we develop a deep learning system for message-level Twitter sentiment classification. Among the 45 submitted systems including SemEval 2013 participants, our (Coooolll) is ranked 2nd on Twitter2014 test set of 2014 Task 9. Coooolll built in supervised framework by concatenating sentiment-specific word embedding (SSWE) features with state-of-the-art hand-crafted features. We neural network hybrid loss function 1 to learn SSWE, which encodes information tweets continuous...

Sentiment analysis (also known as opinion mining) is an active research area in natural language processing. It aims at identifying, extracting and organizing sentiments from user generated texts social networks, blogs or product reviews. A lot of studies literature exploit machine learning approaches to solve sentiment tasks different perspectives the past 15 years. Since performance a learner heavily depends on choices data representation, many devote building powerful feature extractor...

In this paper, we explore the slot tagging with only a few labeled support sentences (a.k.a. few-shot). Few-shot faces unique challenge compared to other fewshot classification problems as it calls for modeling dependencies between labels. But is hard apply previously learned label an unseen domain, due discrepancy of sets. To tackle this, introduce collapsed dependency transfer mechanism into conditional random field (CRF) abstract patterns transition scores. few-shot setting, emission...

Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, Guoping Hu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

We present CodeBERT, a bimodal pre-trained model for programming language (PL) and nat-ural (NL). CodeBERT learns general-purpose representations that support downstream NL-PL applications such as natural codesearch, code documentation generation, etc. develop with Transformer-based neural architecture, train it hybrid objective function incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. This enables us utilize...

Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. The closely related the information of one task can benefit other. Previous studies either implicitly model with multi-task framework or only explicitly consider single flow from intent to slot. None prior approaches bidirectional connection between simultaneously in unified framework. In this paper, we propose Co-Interactive Transformer which considers cross-impact tasks. Instead...

Event detection remains a challenge due to the difficulty at encoding word semantics in various contexts.Previous approaches heavily depend on languagespecific knowledge and pre-existing natural language processing (NLP) tools.However, compared English, not all languages have such resources tools available.A more promising approach is automatically learn effective features from data, without relying resources.In this paper, we develop hybrid neural network capture both sequence chunk...

Deep pretrained language models have achieved great success in the way of pretraining first and then fine-tuning. But such a sequential transfer learning paradigm often confronts catastrophic forgetting problem leads to sub-optimal performance. To fine-tune with less forgetting, we propose recall learn mechanism, which adopts idea multi-task jointly learns tasks downstream tasks. Specifically, introduce Pretraining Simulation mechanism knowledge from without data, an Objective Shifting focus...

Recent work has shown success in using continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems, which is regarded a simple semi-supervised learning mechanism. However, fundamental problems on effectively incorporating the embedding within framework of linear models remain. In this study, we investigate and analyze three different approaches, including new proposed distributional prototype approach, for utilizing features. The presented approaches...

State-of-the-art studies have demonstrated the superiority of joint modeling over pipeline implementation for medical named entity recognition and normalization due to mutual benefits between two processes. To exploit these in a more sophisticated way, we propose novel deep neural multi-task learning framework with explicit feedback strategies jointly model normalization. On one hand, our method from general representations both tasks provided by learning. other successfully converts...

This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text Universal Dependencies. We base submission Stanford's winning for 2017 and make two effective extensions: 1) incorporating deep contextualized word embeddings into both part of speech tagger parser; 2) ensembling parsers trained with different initialization. also explore ways concatenating treebanks further improvements. Experimental results development data show...

We introduce N-LTP, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: lexical analysis (Chinese word segmentation, part-of-speech tagging, and named entity recognition), syntactic parsing (dependency parsing), semantic (semantic dependency role labeling). Unlike the existing state-of-the-art toolkits, such as Stanza, that adopt independent model for each task, N-LTP adopts multi-task framework by using a shared pre-trained model, which has...

Most pre-trained language models (PLMs) construct word representations at subword level with Byte-Pair Encoding (BPE) or its variations, by which OOV (out-of-vocab) words are almost avoidable. However, those methods split a into units and make the representation incomplete fragile.In this paper, we propose character-aware model named CharBERT improving on previous (such as BERT, RoBERTa) to tackle these problems. We first contextual embedding for each token from sequential character...

Research into the area of multiparty dialog has grown considerably over recent years. We present Molweni dataset, a machine reading comprehension (MRC) dataset with discourse structure built dialog. Molweni’s source samples from Ubuntu Chat Corpus, including 10,000 dialogs comprising 88,303 utterances. annotate 30,066 questions on this corpus, both answerable and unanswerable questions. also uniquely contributes dependency annotations in modified Segmented Discourse Representation Theory...

Xiachong Feng, Xiaocheng Libo Qin, Bing Ting Liu. Proceedings of the 59th Annual Meeting Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.