Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and consecutive variants have been proposed to further improve the performance of pre-trained language models. In this paper, we target on revisiting Chinese models examine their effectiveness in a non-English release model series community. We also propose simple but effective called MacBERT, which improves upon RoBERTa several ways, especially masking strategy that...

We address the problem of adversarial attacks on text classification, which is rarely studied comparing to image classification. The challenge this task generate examples that maintain lexical correctness, grammatical correctness and semantic similarity. Based synonyms substitution strategy, we introduce a new word replacement order determined by both saliency classification probability, propose greedy algorithm called probability weighted (PWWS) for attack. Experiments three popular...

Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of pre-trained language models.In this paper, we aim first introduce whole word masking (wwm) strategy for Chinese BERT, along with a series models.Then also propose simple but effective model called MacBERT, which improves upon RoBERTa in several ways.Especially, new MLM as correction...

Yang Xu, Yiheng Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Lidong Zhou. Proceedings of the 59th Annual Meeting Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, Ting Liu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

Semantic hierarchy construction aims to build structures of concepts linked by hypernym-hyponym ("is-a") relations.A major challenge for this task is the automatic discovery such relations.This paper proposes a novel and effective method semantic hierarchies based on word embeddings, which can be used measure relationship between words.We identify whether candidate pair has relation using word-embedding-based projections words their hypernyms.Our result, an F-score 73.74%, outperforms...

As an effective strategy, data augmentation (DA) alleviates scarcity scenarios where deep learning techniques may fail. It is widely applied in computer vision then introduced to natural language processing and achieves improvements many tasks. One of the main focuses DA methods improve diversity training data, thereby helping model better generalize unseen testing data. In this survey, we frame into three categories based on augmented including paraphrasing, noising, sampling. Our paper...

Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, Ting Liu. Proceedings of the 53rd Annual Meeting Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

In this paper, we explore the slot tagging with only a few labeled support sentences (a.k.a. few-shot). Few-shot faces unique challenge compared to other fewshot classification problems as it calls for modeling dependencies between labels. But is hard apply previously learned label an unseen domain, due discrepancy of sets. To tackle this, introduce collapsed dependency transfer mechanism into conditional random field (CRF) abstract patterns transition scores. few-shot setting, emission...

Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, Guoping Hu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

We focus on the study of conversational recommendation in context multi-type dialogs, where bots can proactively and naturally lead a conversation from non-recommendation dialog (e.g., QA) to dialog, taking into account user’s interests feedback. To facilitate this task, we create human-to-human Chinese dataset DuRecDial (about 10k 156k utterances), there are multiple sequential dialogs for pair seeker (user) recommender (bot). In each leads approach targets then makes recommendations with...

Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. The closely related the information of one task can benefit other. Previous studies either implicitly model with multi-task framework or only explicitly consider single flow from intent to slot. None prior approaches bidirectional connection between simultaneously in unified framework. In this paper, we propose Co-Interactive Transformer which considers cross-impact tasks. Instead...

This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text Universal Dependencies. We base submission Stanford's winning for 2017 and make two effective extensions: 1) incorporating deep contextualized word embeddings into both part of speech tagger parser; 2) ensembling parsers trained with different initialization. also explore ways concatenating treebanks further improvements. Experimental results development data show...

Deep pretrained language models have achieved great success in the way of pretraining first and then fine-tuning. But such a sequential transfer learning paradigm often confronts catastrophic forgetting problem leads to sub-optimal performance. To fine-tune with less forgetting, we propose recall learn mechanism, which adopts idea multi-task jointly learns tasks downstream tasks. Specifically, introduce Pretraining Simulation mechanism knowledge from without data, an Objective Shifting focus...

In this paper, we study the problem of data augmentation for language understanding in task-oriented dialogue system. contrast to previous work which augments an utterance without considering its relation with other utterances, propose a sequence-to-sequence generation based framework that leverages one utterance's same semantic alternatives training data. A novel diversity rank is incorporated into representation make model produce diverse utterances and these diversely augmented help...

Recent work has shown success in using continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems, which is regarded a simple semi-supervised learning mechanism. However, fundamental problems on effectively incorporating the embedding within framework of linear models remain. In this study, we investigate and analyze three different approaches, including new proposed distributional prototype approach, for utilizing features. The presented approaches...

Sentiment analysis, which addresses the computational treatment of opinion, sentiment, and subjectivity in text, has received considerable attention recent years. In contrast to traditional coarse-grained sentiment analysis tasks, such as document-level classification, we are interested fine-grained aspect-based that aims identify aspects users comment on these aspects' polarities. Aspect-based relies heavily syntactic features. However, reviews this task focuses natural spontaneous, thus...

Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, Ting Liu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

Multi-lingual contextualized embeddings, such as multilingual-BERT (mBERT), have shown success in a variety of zero-shot cross-lingual tasks. However, these models are limited by having inconsistent representations subwords across different languages. Existing work addresses this issue bilingual projection and fine-tuning technique. We propose data augmentation framework to generate multi-lingual code-switching fine-tune mBERT, which encourages model align from source multiple target...

This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text Universal Dependencies. We base submission Stanford's winning for 2017 and make two effective extensions: 1) incorporating deep contextualized word embeddings into both part of speech tagger parser; 2) ensembling parsers trained with different initialization. also explore ways concatenating treebanks further improvements. Experimental results development data show...

Cross-lingual model transfer has been a promising approach for inducing dependency parsers low-resource languages where annotated treebanks are not available. The major obstacles the two-fold: 1. Lexical features directly transferable across languages; 2. Target language-specific syntactic structures difficult to be recovered. To address these two challenges, we present novel representation learning framework multi-source parsing. Our allows parsing using full lexical straightforwardly. By...

Both entity and relation extraction can benefit from being performed jointly, allowing each task to correct the errors of other. Most existing neural joint methods extract entities relations separately achieve learning through parameter sharing, leading a drawback that information between output cannot be fully exploited. In this paper, we convert into directed graph by designing novel scheme propose transition-based approach generate incrementally, which decoding. Our method model...