Pretraining-Based Natural Language Generation for Text Summarization

• DOI: 10.48550/arxiv.1902.09243
• Publication Date: 2019-01-01
• Authors: Haoyu Zhang; Jianjun Xu; Ji Wang
• Keywords: Sequence (biology); Text generation; ENCODE

In this paper, we propose a novel pretraining-based encoder-decoder framework, which can generate the output sequence based on input in two-stage manner. For encoder of our model, encode into context representations using BERT. decoder, there are two stages first stage, use Transformer-based decoder to draft sequence. second mask each word and feed it BERT, then by combining representation generated predict refined for masked position. To best knowledge, approach is method applies BERT text generation tasks. As step direction, evaluate proposed summarization task. Experimental results show that model achieves new state-of-the-art both CNN/Daily Mail New York Times datasets.