Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint (EMNLP-IJCNLP). 2019.

ChatGPT, a chatbot developed by OpenAI, has gained widespread popularity and media attention since its release in November 2022. However, little hard evidence is available regarding perception various sources. In this paper, we analyze over 300,000 tweets more than 150 scientific papers to investigate how ChatGPT perceived discussed. Our findings show that generally viewed as of high quality, with positive sentiment emotions joy dominating social media. Its slightly decreased debut, however,...

We investigate neural techniques for end-to-end computational argumentation mining (AM). frame AM both as a token-based dependency parsing and sequence tagging problem, including multi-task learning setup. Contrary to models that operate on the argument component level, we find framing leads subpar performance results. In contrast, less complex (local) based BiLSTMs perform robustly across classification scenarios, being able catch long-range dependencies inherent problem. Moreover, jointly...

Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes. In this paper, we introduce: (i) an agreement score evaluate performance processes at instance-level; (ii) adaptive optimizer enhance reliability routing; (iii) compression partial improve networks. We validate our approach on two tasks, namely: multi-label text classification question answering. Experimental results...

We study unsupervised multi-document summarization evaluation metrics, which require neither human-written reference summaries nor human annotations (e.g. preferences, ratings, etc.). propose SUPERT, rates the quality of a summary by measuring its semantic similarity with pseudo summary, i.e. selected salient sentences from source documents, using contextualized embeddings and soft token alignment techniques. Compared to state-of-the-art SUPERT correlates better ratings 18- 39%. Furthermore,...

Christian Stab, Johannes Daxenberger, Chris Stahlhut, Tristan Miller, Benjamin Schiller, Christopher Tauchmann, Steffen Eger, Iryna Gurevych. Proceedings of the 2018 Conference North American Chapter Association for Computational Linguistics: Demonstrations. 2018.

Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, Iryna Gurevych. Proceedings of the 2019 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

Argument mining has become a popular research area in NLP. It typically includes the identification of argumentative components, e.g. claims, as central component an argument. We perform qualitative analysis across six different datasets and show that these appear to conceptualize claims quite differently. To learn about consequences such conceptualizations claim for practical applications, we carried out extensive experiments using state-of-the-art feature-rich deep learning systems,...

Activation functions play a crucial role in neural networks because they are the nonlinearities which have been attributed to success story of deep learning. One currently most popular activation is ReLU, but several competitors recently proposed or ‘discovered’, including LReLU and swish. While works compare newly on few tasks (usually from image classification) against ReLU), we perform first largescale comparison 21 across eight different NLP tasks. We find that largely unknown function...

Claudia Schulz, Steffen Eger, Johannes Daxenberger, Tobias Kahse, Iryna Gurevych. Proceedings of the 2018 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.

Evaluation of cross-lingual encoders is usually performed either via zero-shot transfer in supervised downstream tasks or unsupervised textual similarity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where directly compare source texts to (sometimes low-quality) system translations, which represents a natural adversarial setup for multilingual encoders. Reference-free holds the promise web-scale comparison MT systems. We systematically...

Cross-lingual representations have the potential to make NLP techniques available vast majority of languages in world. However, they currently require large pretraining corpora or access typologically similar languages. In this work, we address these obstacles by removing language identity signals from multilingual embeddings. We examine three approaches for this: (i) re-aligning vector spaces target (all together) a pivot source language; (ii) language-specific means and variances, which...

ChatGPT, a chatbot developed by OpenAI, has gained widespread popularity and media attention since its release in November 2022. However, little hard evidence is available regarding perception various sources. In this paper, we analyze over 300,000 tweets more than 150 scientific papers to investigate how ChatGPT perceived discussed. Our findings show that generally viewed as of high quality, with positive sentiment emotions joy dominating social media. Its slightly decreased debut, however,...

Recently, there has been a growing interest in designing text generation systems from discourse coherence perspective, e.g., modeling the interdependence between sentences. Still, recent BERT-based evaluation metrics are weak recognizing coherence, and thus not reliable way to spot discourse-level improvements of those systems. In this work, we introduce DiscoScore, parametrized metric, which uses BERT model different perspectives, driven by Centering theory. Our experiments encompass 16...

We consider two graph models of semantic change.The first is a time-series model that relates embedding vectors from one time period to previous periods.In the second, we construct for each word: nodes in this correspond points and edge weights similarity word's meaning across points.We apply our corpora three different languages.We find change linear senses.Firstly, today's (= meaning) words can be derived as combinations their neighbors periods.Secondly, self-similarity decays linearly...

Visual modifications to text are often used obfuscate offensive comments in social media (e.g., "!d10t") or as a writing style ("1337" "leet speak"), among other scenarios. We consider this new type of adversarial attack NLP, setting which humans very robust, our experiments with both simple and more difficult visual input perturbations demonstrate. then investigate the impact attacks on current NLP systems character-, word-, sentence-level tasks, showing that neural non-neural models are,...

Abstract Recently proposed BERT-based evaluation metrics for text generation perform well on standard benchmarks but are vulnerable to adversarial attacks, e.g., relating information correctness. We argue that this stems (in part) from the fact they models of semantic similarity. In contrast, we develop based Natural Language Inference (NLI), which deem a more appropriate modeling. design preference-based attack framework and show our NLI much robust attacks than recent metrics. On...

The vast majority of evaluation metrics for machine translation are supervised, i.e., (i) trained on human scores, (ii) assume the existence reference translations, or (iii) leverage parallel data. This hinders their applicability to cases where such supervision signals not available. In this work, we develop fully unsupervised metrics. To do so, similarities and synergies between metric induction, corpus mining, MT systems. particular, use an mine pseudo-parallel data, which remap deficient...

With the advent of large multimodal language models, science is now at a threshold an AI-based technological transformation. Recently, plethora new AI models and tools has been proposed, promising to empower researchers academics worldwide conduct their research more effectively efficiently. This includes all aspects cycle, especially (1) searching for relevant literature; (2) generating ideas conducting experimentation; (3) text-based (4) content (e.g., scientific figures diagrams); (5)...

Average word embeddings are a common baseline for more sophisticated sentence embedding techniques. However, they typically fall short of the performances complex models such as InferSent. Here, we generalize concept average to power mean embeddings. We show that concatenation different types considerably closes gap state-of-the-art methods monolingually and substantially outperforms these techniques cross-lingually. In addition, our proposed method recently baselines SIF Sent2Vec by solid...

Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, Yusuke Miyao. Proceedings of the 2019 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.

Generative large language models (LLMs) have seen many breakthroughs over the last year. With an increasing number of parameters and pre-training data, they shown remarkable capabilities to solve tasks with minimal or no task-related examples. Notably, LLMs been successfully employed as evaluation metrics in text generation tasks. Strategies this context differ choice input prompts, selection samples for demonstration, methodology used construct scores grading generations. Approaches often...