DIET: Lightweight Language Understanding for Dialogue Systems

• DOI: 10.48550/arxiv.2004.09936
• Publication Date: 2020-01-01
• Authors: Tanja Bunk; Daksh Varshneya; В. И. Власов; Alan Nichol
• Keywords: Language Understanding; Natural language understanding

Large-scale pre-trained language models have shown impressive results on understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods distributed representations (GloVe) purely supervised approaches. We introduce the Dual Intent Entity Transformer (DIET) architecture, study effectiveness of different intent entity prediction, two common dialogue tasks. DIET advances state art a complex multi-domain NLU dataset achieves similarly high performance simpler datasets. Surprisingly, we show that there is no clear benefit to using large for this task, in fact improves upon current even setup without any embeddings. Our best performing model outperforms fine-tuning BERT about six times faster train.