We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on set Wikipedia articles, where answer to each question is segment text from corresponding passage. analyze understand types reasoning required questions, leaning heavily dependency and constituency trees. build strong logistic regression model, which achieves an F1 score 51.0%, significant improvement over simple baseline (20%). However,...

Extractive reading comprehension systems can often locate the correct answer to a question in context document, but they also tend make unreliable guesses on questions for which is not stated context. Existing datasets either focus exclusively answerable questions, or use automatically generated unanswerable that are easy identify. To address these weaknesses, we present SQuADRUn, new dataset combines existing Stanford Question Answering Dataset (SQuAD) with over 50,000 written adversarially...

Xiang Lisa Li, Percy Liang. Proceedings of the 59th Annual Meeting Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

In this paper, we train a semantic parser that scales up to Freebase. Instead of relying on annotated logical forms, which is especially expensive obtain at large scale, learn from question-answer pairs. The main challenge in setting narrowing down the huge number possible predicates for given question. We tackle problem two ways: First, build coarse mapping phrases using knowledge base and text corpus. Second, use bridging operation generate additional based neighboring predicates. On...

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these truly understand language remains unclear. To reward with real understanding abilities, we propose an adversarial evaluation scheme for Stanford Question Answering Dataset (SQuAD). Our method tests whether can answer questions about paragraphs contain adversarially inserted sentences, automatically generated distract computer without changing correct or misleading...

How can we explain the predictions of a black-box model? In this paper, use influence functions -- classic technique from robust statistics to trace model's prediction through learning algorithm and back its training data, thereby identifying points most responsible for given prediction. To scale up modern machine settings, develop simple, efficient implementation that requires only oracle access gradients Hessian-vector products. We show even on non-convex non-differentiable models where...

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on set Wikipedia articles, where answer to each question is segment text from corresponding passage. analyze understand types reasoning required questions, leaning heavily dependency and constituency trees. build strong logistic regression model, which achieves an F1 score 51.0%, significant improvement over simple baseline (20%). However,...

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer as emergent abilities large models. We consider ability be if it is not present in smaller but larger Thus, cannot predicted simply by extrapolating the The existence such emergence implies additional scaling could further expand capabilities

We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions total). The involve two crowd workers: (1) student who poses sequence of freeform to learn as much possible about hidden Wikipedia text, and (2) teacher answers the by providing short excerpts from text. QuAC introduces challenges not found existing machine comprehension datasets: its are often more open-ended, unanswerable, or only meaningful within dialog context,...

Suppose we want to build a system that answers natural language question by representing its semantics as logical forxm and computing the answer given structured database of facts. The core part such is semantic parser maps questions forms. Semantic parsers are typically trained from examples annotated with their target forms, but this type annotation expensive. Our goal instead learn question–answer pairs, where form modeled latent variable. We develop new formalism, dependency-based...

A central challenge in semantic parsing is handling the myriad ways which knowledge base predicates can be expressed.Traditionally, parsers are trained primarily from text paired with information.Our goal to exploit much larger amounts of raw not tied any base.In this paper, we turn on its head.Given an input utterance, first use a simple method deterministically generate set candidate logical forms canonical realization natural language for each.Then, paraphrase model choose that best...

Juncen Li, Robin Jia, He He, Percy Liang. Proceedings of the 2018 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Modeling crisp logical regularities is crucial in semantic parsing, making it difficult for neural models with no task-specific prior knowledge to achieve good results.In this paper, we introduce data recombination, a novel framework injecting such into model.From the training data, induce highprecision synchronous context-free grammar, which captures important conditional independence properties commonly found parsing.We then train sequence-to-sequence recurrent network (RNN) model...

Panupong Pasupat, Percy Liang. Proceedings of the 53rd Annual Meeting Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels scarce during training. An effective approach this challenge is pre-train related tasks where data abundant, and then fine-tune it downstream task interest. While pre-training has been in many language vision domains, remains an open question how effectively use graph datasets. In paper, we develop new strategy...

John Hewitt, Percy Liang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

We present an unsupervised approach to symmetric word alignment in which two simple asymmetric models are trained jointly maximize a combination of data likelihood and agreement between the models. Compared standard practice intersecting predictions independently-trained models, joint training provides 32% reduction AER. Moreover, efficient pair HMM aligners 29% AER over symmetrized IBM model 4 predictions.

Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication prototyping tools. In this paper, we introduce generative agents: computational software agents that simulate believable behavior. Generative wake up, cook breakfast, and head work; artists paint, while authors write; they form opinions, notice each other, initiate conversations; remember reflect on days past as plan the next day....

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. shows a sample complexity gap between standard robust classification. prove unlabeled data bridges this gap: learning procedure (self-training) achieves high accuracy using same number labels required for achieving accuracy. Empirically, augment CIFAR-10 with 500K images sourced 80 Million Tiny...

How do we build a semantic parser in new domain starting with zero training examples? We introduce methodology for this setting: First, use simple grammar to generate logical forms paired canonical utterances. The are meant cover the desired set of compositional operators, and utterances capture meaning (although clumsily). then crowdsourcing paraphrase these into natural resulting data is used train parser. further study role compositionality paraphrases. Finally, test our on seven domains...

Machine learning systems trained on user-provided data are susceptible to poisoning attacks, whereby malicious users inject false training with the aim of corrupting learned model. While recent work has proposed a number attacks and defenses, little is understood about worst-case loss defense in face determined attacker. We address this by constructing approximate upper bounds across broad family for defenders that first perform outlier removal followed empirical risk minimization. Our...

While neural networks have achieved high accuracy on standard image classification benchmarks, their drops to nearly zero in the presence of small adversarial perturbations test inputs. Defenses based regularization and training been proposed, but often followed by new, stronger attacks that defeat these defenses. Can we somehow end this arms race? In work, study problem for with one hidden layer. We first propose a method semidefinite relaxation outputs certificate given network input, no...

Overparameterized neural networks can be highly accurate on average an i.i.d. test set yet consistently fail atypical groups of the data (e.g., by learning spurious correlations that hold but not in such groups). Distributionally robust optimization (DRO) allows us to learn models instead minimize worst-case training loss over a pre-defined groups. However, we find naively applying group DRO overparameterized fails: these perfectly fit data, and any model with vanishing also already has...

A central problem in grounded language acquisition is learning the correspondences between a rich world state and stream of text which references that state. To deal with high degree ambiguity present this setting, we generative model simultaneously segments into utterances maps each utterance to meaning representation We show our generalizes across three domains increasing difficulty---Robocup sportscasting, weather forecasts (a new domain), NFL recaps.