We propose a hypothesis only baseline for diagnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes inference is occurring based purely on the relationship between context and hypothesis, it follows that assessing entailment relations while ignoring provided degenerate solution. Yet, through experiments 10 distinct datasets, we find this approach, which refer to as hypothesis-only model, able significantly outperform majority-class across number of datasets. Our...

Answering natural language questions using the Freebase knowledge base has recently been explored as a platform for advancing state of art in open domain semantic parsing.Those efforts map to sophisticated meaning representations that are then attempted be matched against viable answer candidates base.Here we show relatively modest information extraction techniques, when paired with webscale corpus, can outperform these approaches by roughly 34% relative gain.

Rachel Rudinger, Jason Naradowsky, Brian Leonard, Benjamin Van Durme. Proceedings of the 2018 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building recent token-level probing work, we introduce novel edge task design construct broad suite sub-sentence tasks derived from the traditional structured pipeline. We probe word-level contextual representations four investigate how they encode sentence structure across range syntactic, semantic, local,...

Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch. Proceedings of the 53rd Annual Meeting Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2015.

We present a large-scale dataset, ReCoRD, for machine reading comprehension requiring commonsense reasoning. Experiments on this dataset demonstrate that the performance of state-of-the-art MRC systems fall far behind human performance. ReCoRD represents challenge future research to bridge gap between and comprehension. is available at http://nlp.jhu.edu/record.

We propose a novel approach to sentiment analysis for low resource setting. The intuition behind this work is that expressed towards an entity, targeted sentiment, may be viewed as span of across the entity. This representation allows us model detection sequence tagging problem, jointly discovering people and organizations along with whether there directed them. compare performance in both Spanish English on microblog data, using only lexicon external resource. By leveraging...

Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Statutory reasoning is the task of with facts and statutes, which are rules written in natural language by a legislature. It basic legal skill. In this paper we explore capabilities most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider variety approaches, including dynamic few-shot prompting, chain-of-thought zero-shot prompting. While achieve results that better than previous best published results, also identify several types...

Much work in knowledge extraction from text tacitly assumes that the frequency with which people write about actions, outcomes, or properties is a reflection of real-world frequencies degree to property characteristic class individuals. In this paper, we question idea, examining phenomenon reporting bias and challenge it poses for extraction. We conclude discussion approaches learning commonsense despite distortion.

Aaron Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, Benjamin Van Durme. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016.

Spoken term discovery is the task of automatically identifying words and phrases in speech data by searching for long repeated acoustic patterns. Initial solutions relied on exhaustive dynamic time warping-based searches across entire similarity matrix, a method whose scalability ultimately limited O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) nature search space. Recent strategies have attempted to improve efficiency using...

Humans have the capacity to draw common-sense inferences from natural language: various things that are likely but not certain hold based on established discourse, and rarely stated explicitly. We propose an evaluation of automated inference extension recognizing textual entailment: predicting ordinal human responses subjective likelihood holding in a given context. describe framework for extracting knowledge corpora, which is then used construct dataset this entailment task. train neural...

We present a novel document-level model for finding argument spans that fill an event’s roles, connecting related ideas in sentence-level semantic role labeling and coreference resolution. Because existing datasets cross-sentence linking are small, development of our neural is supported through the creation new resource, Roles Across Multiple Sentences (RAMS), which contains 9,124 annotated events across 139 types. demonstrate strong performance on RAMS other event-related datasets.

Existing models for social media personal analytics assume access to thousands of messages per user, even though most users author content only sporadically over time. Given this sparsity, we: (i) leverage from the local neighborhood a user; (ii) evaluate batch as function size and amount in various types neighborhoods; (iii) estimate time tweets required dynamic model predict user preferences. We show that when limited or no selfauthored data is available, language friend, retweet mention...

Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, Benjamin Van Durme. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, Benjamin Van Durme. Proceedings of the 2019 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

We analyze the Stanford Natural Language Inference (SNLI) corpus in an investigation of bias and stereotyping NLP data. The SNLI human-elicitation protocol makes it prone to amplifying stereotypical associations, which we demonstrate statistically (using pointwise mutual information) with qualitative examples.

We propose an attention-based model that treats AMR parsing as sequence-to-graph transduction. Unlike most parsers rely on pre-trained aligners, external semantic resources, or data augmentation, our proposed parser is aligner-free, and it can be effectively trained with limited amounts of labeled data. Our experimental results outperform all previously reported SMATCH scores, both 2.0 (76.3% LDC2017T10) 1.0 (70.2% LDC2014T12).

Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, Ellie Pavlick. Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019). 2019.

The narrative cloze is an evaluation metric commonly used for work on automatic script induction.While prior in this area has focused count-based methods from distributional semantics, such as pointwise mutual information, we argue that the can be productively reframed a language modeling task.By training discriminative model task, attain improvements of up to 27 percent over standard metrics.