Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is member of target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA the pre-training data large language models (LLMs). We perform large-scale evaluation MIAs over suite (LMs) trained Pile, ranging from 160M 12B parameters. find that barely outperform random guessing for most settings across varying LLM sizes and domains. Our...

Distribution inference, sometimes called property infers statistical properties about a training set from access to model trained on that data. inference attacks can pose serious risks when models are private data, but difficult distinguish the intrinsic purpose of machine learning—namely, produce capture distribution. Motivated by Yeom et al.’s membership framework, we propose formal definition distribution general enough describe broad class distinguishing between possible distributions....

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership reconstruction attacks. Inspired by the success games (i.e. probabilistic experiments) study security properties cryptography, some authors describe privacy risks using similar game-based style. However, adversary capabilities and goals are often stated subtly ways one...

Federated Learning is vulnerable to adversarial manipulation, where malicious clients can inject poisoned updates influence the global model's behavior. While existing defense mechanisms have made notable progress, they fail protect against adversaries that aim induce targeted backdoors under different learning and attack configurations. To address this limitation, we introduce DROP (Distillation-based Reduction Of Poisoning), a novel mechanism combines clustering activity-tracking...

Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to generate grounded responses by leveraging external knowledge databases without altering model parameters. Although the absence of weight tuning prevents leakage via parameters, it introduces risk inference adversaries exploiting retrieved documents in model's context. Existing methods for membership and data extraction often rely on jailbreaking or carefully crafted unnatural queries, which can be easily detected...

Privacy attacks on Machine Learning (ML) models often focus inferring the existence of particular data points in training data. However, what adversary really wants to know is if a individual's (subject's) was included during training. In such scenarios, more likely have access distribution subject than actual records. Furthermore, settings like cross-silo Federated (FL), subject's can be embodied by multiple records that are spread across organizations. Nearly all existing private FL...

Existing Machine Learning techniques yield close to human performance on text-based classification tasks. However, the presence of multi-modal noise in chat data such as emoticons, slang, spelling mistakes, code-mixed data, etc. makes existing deep-learning solutions perform poorly. The inability systems robustly capture these covariates puts a cap their performance. We propose NELEC: Neural and Lexical Combiner, system which elegantly combines textual based methods for sentiment...

A distribution inference attack aims to infer statistical properties of data used train machine learning models. These attacks are sometimes surprisingly potent, but the factors that impact risk not well understood and demonstrated often rely on strong unrealistic assumptions such as full knowledge training environments even in supposedly black-box threat scenarios. To improve understanding risks, we develop a new outperforms best known white-box most settings. Using this attack, evaluate...

Extensive research has been conducted to identify, analyze and measure popular topics public sentiment on Online Social Networks (OSNs) through text, especially during crisis events. However, little work done understand such events pictures posted these networks. Given the potential of visual content for influencing users' thoughts emotions, we perform a large-scale analysis study compare themes across images textual Facebook terror attacks that took place in Paris 2015. We propose...

Transfer learning is a popular method for tuning pretrained (upstream) models different downstream tasks using limited data and computational resources. We study how an adversary with control over upstream model used in transfer can conduct property inference attacks on victim's tuned model. For example, to infer the presence of images specific individual training set. demonstrate which manipulate highly effective (AUC score > 0.9), without incurring significant performance loss main task....

Vighnesh Leonardo Shiv, Chris Quirk, Anshuman Suri, Xiang Gao, Khuram Shahid, Nithya Govindarajan, Yizhe Zhang, Jianfeng Michel Galley, Brockett, Tulasi Menon, Bill Dolan. Proceedings of the 57th Annual Meeting Association for Computational Linguistics: System Demonstrations. 2019.

Deep neural networks (DNNs) are vulnerable to malicious inputs crafted by an adversary produce erroneous outputs. Works on securing against adversarial examples achieve high empirical robustness simple datasets such as MNIST. However, these techniques inadequate when empirically tested complex data sets CIFAR-10 and SVHN. Further, existing designed target specific attacks fail generalize across attacks. We propose Adversarial Model Cascades (AMC) a way tackle the above inadequacies. Our...

Face recognition in the unconstrained environment is an ongoing research challenge. Although several covariates of face such as pose and low resolution have received significant attention, “disguise” considered onerous covariate recognition. One primary reasons for this scarcity large representative labeled databases, along with lack algorithms that work well multiple environments. In order to address problem presence disguise, paper proposes active learning framework termed A2-LINK....

Large Language Models (LLMs) are advancing at a remarkable pace, with myriad applications under development. Unlike most earlier machine learning models, they no longer built for one specific application but designed to excel in wide range of tasks. A major part this success is due their huge training datasets and the unprecedented number model parameters, which allow them memorize large amounts information contained data. This memorization goes beyond mere language, encompasses only present...

Having a bot for seamless conversations is much-desired feature that products and services today seek their websites mobile apps. These bots help reduce traffic received by human support significantly handling frequent directly answerable known questions. Many such have huge reference documents as FAQ pages, which makes it hard users to browse through this data. A conversation layer over raw data can lower great margin. We demonstrate QnAMaker, service creates conversational semi-structured...

In a poisoning attack, an adversary with control over small fraction of the training data attempts to select that in way induces corrupted model misbehaves favor adversary. We consider attacks against convex machine learning models and propose efficient attack designed induce specified model. Unlike previous model-targeted attacks, our comes provable convergence {\it any} attainable target classifier. The distance from induced classifier is inversely proportional square root number points....

Online Social Networks explode with activity whenever a crisis event takes place. Most content generated as part of this is mixture text and images, particularly useful for first responders to identify popular topics interest gauge the pulse sentiment citizens. While multiple researchers have used identify, analyze measure themes public during such events, little work has explored visual floating on networks in form inspired by them. Given potential influencing users' thoughts emotions, we...

Recent advancements in deep learning have significantly increased the capabilities of face recognition. However, recognition an unconstrained environment is still active research challenge. Covariates such as pose and low resolution received significant attention, but "disguise" considered onerous covariate One primary reason for this unavailability large representative databases. To address problem recognizing disguised faces, we propose framework A-LINK <sup...

Numerous works study black-box attacks on image classifiers. However, these make different assumptions the adversary's knowledge and current literature lacks a cohesive organization centered around threat model. To systematize in this area, we propose taxonomy over space spanning axes of feedback granularity, access interactive queries, quality quantity auxiliary data available to attacker. Our new provides three key insights. 1) Despite extensive literature, numerous under-explored spaces...

Numerous works study black-box attacks on image classifiers, where adversaries generate adversarial examples against unknown target models without having access to their internal information. However, these make different assumptions about the adversary's knowledge, and current literature lacks cohesive organization centered around threat model. To systematize knowledge in this area, we propose a taxonomy over space spanning axes of feedback granularity, interactive queries, quality quantity...

Membership inference attacks aim to infer whether an individual record was used train a model, serving as key tool for disclosure auditing. While such evaluations are useful demonstrate risk, they computationally expensive and often make strong assumptions about potential adversaries' access models training environments, thus do not provide very tight bounds on leakage from attacks. We show how prior claims around black-box being sufficient optimal membership hold most settings stochastic...

Deploying machine learning models in production may allow adversaries to infer sensitive information about training data. There is a vast literature analyzing different types of inference risks, ranging from membership reconstruction attacks. Inspired by the success games (i.e., probabilistic experiments) study security properties cryptography, some authors describe privacy risks using similar game-based style. However, adversary capabilities and goals are often stated subtly ways one...

Despite vast research in adversarial examples, the root causes of model susceptibility are not well understood. Instead looking at attack-specific robustness, we propose a notion that evaluates sensitivity individual neurons terms how robust model's output is to direct perturbations neuron's output. Analyzing models from this perspective reveals distinctive characteristics standard as adversarially-trained models, and leads several curious results. In our experiments on CIFAR-10 ImageNet,...