The cross-entropy loss commonly used in deep learning is closely related to the defining properties of optimal representations, but does not enforce some key properties. We show that this can be solved by adding a regularization term, which turn injecting multiplicative noise activations Deep Neural Network, special case common practice dropout. our regularized function efficiently minimized using Information Dropout, generalization dropout rooted information theoretic principles...

Using established principles from Information Theory and Statistics, we show that in a deep neural network invariance to nuisance factors is equivalent information minimality of the learned representation, stacking layers injecting noise during training naturally bias towards learning invariant representations. We then that, order avoid memorization, need limit quantity stored weights, which leads novel usage Bottleneck Lagrangian on weights as criterion. This also has an alternative...

We introduce a method to generate vectorial representations of visual classification tasks which can be used reason about the nature those and their relations. Given dataset with ground-truth labels loss function, we process images through "probe network" compute an embedding based on estimates Fisher information matrix associated probe network parameters. This provides fixed-dimensional task that is independent details such as number classes requires no understanding class label semantics....

We explore the problem of selectively forgetting a particular subset data used for training deep neural network. While effects to be forgotten can hidden from output network, insights may still gleaned by probing into its weights. propose method "scrubbing" weights clean information about set data. The does not require retraining scratch, nor access originally training. Instead, are modified so that any function is indistinguishable same applied network trained without forgotten. This...

We address the problem of scene layout generation for diverse domains such as images, mobile applications, documents, and 3D objects. Most complex scenes, natural or human-designed, can be expressed a meaningful arrangement simpler compositional graphical primitives. Generating new extending an existing re- quires understanding relationships between these To do this, we propose LayoutTransformer, novel framework that leverages self-attention to learn contextual elements generate layouts in...

We show that the influence of a subset training samples can be removed – or "forgotten" from weights network trained on large-scale image classification tasks, and we provide strong computable bounds amount remaining information after forgetting. Inspired by real-world applications forgetting techniques, introduce novel notion in mixed-privacy setting, where know "core" does not need to forgotten. While this variation problem is conceptually simple, working setting significantly improves...

Adapting pre-trained models with broad capabilities has become standard practice for learning a wide range of downstream tasks. The typical approach fine-tuning different each task is performant, but incurs substantial memory cost. To efficiently learn multiple down-stream tasks we introduce Task Adaptive Parameter Sharing (TAPS), simple method tuning base model to new by adaptively modifying small, task-specific subset layers. This enables multi-task while minimizing the resources used and...

Over the past few years, machine learning models have significantly increased in size and complexity, especially area of generative AI such as large language models. These require massive amounts data compute capacity to train, extent that concerns over training (such protected or private content) cannot be practically addressed by retraining model “from scratch” with questionable removed altered. Furthermore, despite significant efforts controls dedicated ensuring corpora are properly...

Similar to humans and animals, deep artificial neural networks exhibit critical periods during which a temporary stimulus deficit can impair the development of skill. The extent impairment depends on onset length window, as in animal models, size network. Deficits that do not affect low-level statistics, such vertical flipping images, have no lasting effect performance be overcome with further training. To better understand this phenomenon, we use Fisher Information weights measure effective...

We propose a novel deterministic method for preparing arbitrary quantum states. When our protocol is compiled into CNOT and single-qubit gates, it prepares an <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>N</mml:mi></mml:math>-dimensional state in depth xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>O</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mi>log</mml:mi><mml:mo>&amp;#x2061;</mml:mo><mml:mo stretchy="false">(</mml:mo><mml:mi>N</mml:mi><mml:mo...

Whatever information a deep neural network has gleaned from training data is encoded in its weights. How this affects the response of to future remains largely an open question. Indeed, even defining and measuring entails some subtleties, since trained deterministic map, so standard measures can be degenerate. We measure via optimal trade-off between accuracy complexity weights, measured by their coding length. Depending on choice code, definition reduce such as Shannon Mutual Information...

This paper explores the possibility of learning custom tokens for representing new concepts in Vision-Language Models (VLMs). Our aim is to learn that can be effective both discriminative and generative tasks while composing well with words form input queries. The targeted concept specified terms a small set images parent described using text. We operate on CLIP text features propose use combination textual inversion loss classification ensure learned token are aligned image embedding space....

Intelligent behaviour in the real-world requires ability to acquire new knowledge from an ongoing sequence of experiences while preserving and reusing past knowledge. We propose a novel algorithm for unsupervised representation learning piece-wise stationary visual data: Variational Autoencoder with Shared Embeddings (VASE). Based on Minimum Description Length principle, VASE automatically detects shifts data distribution allocates spare representational capacity knowledge, simultaneously...