SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems

• DOI: 10.1609/aaai.v36i10.21341
• Publication Date: 2022-07-04T11:49:46Z
• Authors: Harrison Lee; Raghav Gupta; Abhinav Rastogi; Yuan Cao; Bin Zhang; Yonghui Wu
• Keywords: Schema (genetic algorithms); Robustness

Zero/few-shot transfer to unseen services is a critical challenge in task-oriented dialogue research. The Schema-Guided Dialogue (SGD) dataset introduced paradigm for enabling models support any service zero-shot through schemas, which describe APIs natural language. We explore the robustness of systems linguistic variations schemas by designing SGD-X - benchmark extending SGD with semantically similar yet stylistically diverse variants every schema. observe that two top state tracking fail generalize well across schema variants, measured joint goal accuracy and novel metric measuring sensitivity. Additionally, we present simple model-agnostic data augmentation method improve robustness.