DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization

• DOI: 10.48550/arxiv.2110.08168
• Publication Date: 2021-01-01
• Authors: Ziming Mao; Chen Wu; Ansong Ni; Yusen Zhang; Rui Zhang; Changyuan Yu; Budhaditya Deb; Chenguang Zhu; Ahmed Hassan Awad...; Dragomir Radev
• Keywords: Extractor; Grammaticality; Heuristics

Transformer-based models have achieved state-of-the-art performance on short-input summarization. However, they still struggle with summarizing longer text. In this paper, we present DYLE, a novel dynamic latent extraction approach for abstractive long-input DYLE jointly trains an extractor and generator treats the extracted text snippets as variable, allowing snippet-level attention weights during decoding. To provide adequate supervision, propose simple yet effective heuristics oracle well consistency loss term, which encourages to approximate averaged predicted by generator. We evaluate our method different long-document long-dialogue summarization tasks: GovReport, QMSum, arXiv. Experiment results show that outperforms all existing methods GovReport gains up 6.1 ROUGE, while yielding strong Further analysis shows proposed interpretability of generation process.