Analyzing Compositionality-Sensitivity of NLI Models

• DOI: 10.1609/aaai.v33i01.33016867
• Publication Date: 2019-08-25T07:45:39Z
• Authors: Yixin Nie; Yicheng Wang; Mohit Bansal
• Keywords: Principle of compositionality; Shuffling; Tree (set theory)

Success in natural language inference (NLI) should require a model to understand both lexical and compositional semantics. However, through adversarial evaluation, we find that several state-of-the-art models with diverse architectures are over-relying on the former fail use latter. Further, this compositionality unawareness is not reflected via standard evaluation current datasets. We show removing RNNs existing or shuffling input words during training does induce large performance loss despite explicit removal of information. Therefore, propose compositionality-sensitivity testing setup analyzes examples from datasets cannot be solved features alone (i.e., which bag-of-words gives high probability one wrong label), hence revealing models’ actual awareness. only highlights limited ability NLI models, but also differentiates based design, e.g., separating shallow deeper, linguistically-grounded tree-based models. Our an important analysis tool: complementing currently linguistically driven diagnostic evaluations, exposing opportunities for future work evaluating understanding.