Linear prediction methods, such as least squares for regression, logistic regression and support vector machines classification, have been extensively used in statistics machine learning. In this paper, we study stochastic gradient descent (SGD) algorithms on regularized forms of linear methods. This class related to online perceptron, are both efficient very simple implement. We obtain numerical rate convergence algorithms, discuss its implications. Experiments text data will be provided...

This paper develops a theory for group Lasso using concept called strong sparsity. Our result shows that is superior to standard strongly group-sparse signals. provides convincing theoretical justification sparse regularization when the underlying structure consistent with data. Moreover, predicts some limitations of formulation are confirmed by simulation studies.

This paper presents a new analysis for the orthogonal matching pursuit (OMP) algorithm. It is shown that if restricted isometry property (RIP) satisfied at sparsity level <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$O(\bar{k})$</tex></formula> , then OMP can stably recover Notation="TeX">$\bar{k}$</tex></formula> -sparse signal in 2-norm under measurement noise. For compressed sensing...

We propose a general method called truncated gradient to induce sparsity in the weights of online learning algorithms with convex loss functions. This has several essential properties: The degree is continuous -- parameter controls rate sparsification from no total sparsification. approach theoretically motivated, and an instance it can be regarded as counterpart popular $L_1$-regularization batch setting. prove that small rates result only additional regret respect typical guarantees. works...

In this paper, we propose the first framework (UCNet) to employ uncertainty for RGB-D saliency detection by learning from data labeling process. Existing methods treat task as a point estimation problem, and produce single map following deterministic pipeline. Inspired process, probabilistic network via conditional variational autoencoders model human annotation generate multiple maps each input image sampling in latent space. With proposed consensus are able an accurate based on these...

This paper investigates a new learning formulation called structured sparsity, which is natural extension of the standard sparsity concept in statistical and compressive sensing. By allowing arbitrary structures on feature set, this generalizes group idea that has become popular recent years. A general theory developed for with based notion coding complexity associated structure. It shown if target signal small, then one can achieve improved performance by using regularization methods,...

Given a large number of basis functions that can be potentially more than the samples, we consider problem learning sparse target function expressed as linear combination small these functions. We are interested in two closely related themes: <orderedlist continuation="restarts" numeration="bullet" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <listitem><para>feature selection, or identifying with nonzero coefficients;</para></listitem></orderedlist>

Visual salient object detection (SOD) aims at finding the object(s) that attract human attention, while camouflaged (COD) on contrary intends to discover hidden in surrounding. In this paper, we propose a paradigm of lever-aging contradictory information enhance ability both and detection. We start by exploiting easy positive samples COD dataset serve as hard SOD task improve robustness model. Then, introduce "similarity measure" module explicitly model contradicting attributes these two...

Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge adapting GCNs large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to uncontrollable neighborhood expansion across layers. In this paper, we accelerate training through developing an adaptive layer-wise sampling method. By constructing network layer by top-down passway, sample lower conditioned top one, where...

This paper investigates a new learning formulation called structured sparsity, which is natural extension of the standard sparsity concept in statistical and compressive sensing. By allowing arbitrary structures on feature set, this generalizes group idea. A general theory developed for with based notion coding complexity associated structure. Moreover, greedy algorithm proposed to efficiently solve problem. Experiments demonstrate advantage over sparsity.

We provide theoretical analysis of the statistical and computational properties penalized $M$-estimators that can be formulated as solution to a possibly nonconvex optimization problem. Many important estimators fall in this category, including least squares regression with regularization, generalized linear models regularization sparse elliptical random design regression. For these problems, it is intractable calculate global due formulation. In paper, we propose an approximate...

Large-scale distributed optimization is of great importance in various applications. For data-parallel based learning, the inter-node gradient communication often becomes performance bottleneck. In this paper, we propose error compensated quantized stochastic descent algorithm to improve training efficiency. Local gradients are reduce overhead, and accumulated quantization utilized speed up convergence. Furthermore, present theoretical analysis on convergence behaviour, demonstrate its...

This paper addresses the problem of cross-view image geo-localization, where geographic location a ground-level street-view query is estimated by matching it against large scale aerial map (e.g., high-resolution satellite image). State-of-the-art deep-learning based methods tackle this as deep metric learning which aims to learn global feature representations scene seen two different views. Despite promising results are obtained such methods, they, however, fail exploit crucial cue relevant...

Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Generative foundation models are susceptible to implicit biases that can arise from extensive unsupervised training data. Such produce suboptimal samples, skewed outcomes, and unfairness, with potentially serious consequences. Consequently, aligning these human ethics preferences is an essential step toward ensuring their responsible effective deployment in real-world applications. Prior research has primarily employed Reinforcement Learning Human Feedback (RLHF) address this problem, where...