Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video framework built upon Transformers, termed VisTR, which views VIS as direct end-to-end parallel sequence decoding/prediction problem. Given clip consisting multiple image frames input, VisTR outputs masks for each order directly. At core...

Detecting individual pedestrians in a crowd remains challenging problem since the often gather together and occlude each other real-world scenarios. In this paper, we first explore how state-of-the-art pedestrian detector is harmed by occlusion via experimentation, providing insights into problem. Then, propose novel bounding box regression loss specifically designed for scenes, termed repulsion loss. This driven two motivations: attraction target, surrounding objects. The term prevents...

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal dense prediction tasks due to the discrepancy between image-level pixel-level prediction. fill this gap, we aim design an effective, method that directly works at level of pixels (or local features) by taking into account correspondence features. We present contrastive (DenseCL), which implements optimizing a pairwise (dis)similarity loss...

In this work, we aim at building a simple, direct, and fast instance segmentation framework with strong performance. We follow the principle of SOLO method Wang et al. "SOLO: segmenting objects by locations". Importantly, take one step further dynamically learning mask head object segmenter such that is conditioned on location. Specifically, branch decoupled into kernel feature branch, which are responsible for convolution convolved features respectively. Moreover, propose Matrix NMS (non...

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated conditioned on the local neighborhood tokens. As result, can easily generalize to sequences that longer than what model has ever seen during training. Besides, keep desired translation-invariance in image classification task, resulting improved performance. implement with simple...

A 3D point cloud describes the real scene precisely and intuitively. To date how to segment diversified elements in such an informative is rarely discussed. In this paper, we first introduce a simple flexible framework instances semantics clouds simultaneously. Then, propose two approaches which make tasks take advantage of each other, leading win-win situation. Specifically, instance segmentation benefit from semantic through learning semantic-aware point-level embedding. Meanwhile,...

We launch EVA, a vision-centric foundation model to Explore the limits of Visual representation at scAle using only publicly accessible data. EVA is vanilla ViT pre-trained reconstruct masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up one billion parameters, and sets new records broad range representative downstream tasks, such as recognition, video action object detection, instance segmentation semantic...

We present a high-performance method that can achieve mask-level instance segmentation with only bounding-box annotations for training. While this setting has been studied in the literature, here we show significantly stronger performance simple design (e.g., dramatically improving previous best reported mask AP of 21.1% [13] to 31.6% on COCO dataset). Our core idea is redesign loss learning masks segmentation, no modification network itself. The new functions supervise training without...

Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data challenging due to position-sensitivity of requirement global context understanding. In this paper, we show that reliance on self-attention representation not necessary propose a new backbone...

We present SegGPT, a generalist model for segmenting everything in context. unify various segmentation tasks into in-context learning framework that accommodates different kinds of data by transforming them the same format images. The training SegGPT is formulated as an coloring problem with random color mapping each sample. objective to accomplish diverse according context, rather than relying on specific colors. After training, can perform arbitrary images or videos via inference, such...

In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt various tasks with only handful of prompts and examples. But computer vision, difficulties for in-context learning lie that vary significantly output representations, thus it is unclear how define general-purpose task vision can understand transfer out-of-domain tasks. In this work, we present Painter, generalist which addresses these obstacles an "image"-centric solution, is, redefine core images, specify also...

Instance segmentation is a fundamental vision task that aims to recognize and segment each object in an image. However, it requires costly annotations such as bounding boxes masks for learning. In this work, we propose fully unsupervised learning method learns class-agnostic instance without any annotations. We present FreeSOLO, self-supervised framework built on top of the simple SOLO. Our also presents novel localization-aware pre-training framework, where objects can be discovered from...

Contrastive language-image pre-training, CLIP for short, has gained increasing attention its potential in various scenarios. In this paper, we propose EVA-CLIP, a series of models that significantly improve the efficiency and effectiveness training. Our approach incorporates new techniques representation learning, optimization, augmentation, enabling EVA-CLIP to achieve superior performance compared previous with same number parameters but smaller training costs. Notably, our largest...

We present SegGPT, a generalist model for segmenting everything in context. unify various segmentation tasks into in-context learning framework that accommodates different kinds of data by transforming them the same format images. The training SegGPT is formulated as an coloring problem with random color mapping each sample. objective to accomplish diverse according context, rather than relying on specific colors. After training, can perform arbitrary images or videos via inference, such...

We launch EVA-02, a next-generation Transformer-based visual representation pre-trained to reconstruct strong and robust language-aligned vision features via masked image modeling. With an updated plain Transformer architecture as well extensive pre-training from open & accessible giant CLIP encoder, EVA-02 demonstrates superior performance compared prior state-of-the-art approaches across various representative tasks, while utilizing significantly fewer parameters compute budgets. Notably,...

Compared to many other dense prediction tasks, e.g., semantic segmentation, it is the arbitrary number of instances that has made instance segmentation much more challenging. In order predict a mask for each instance, mainstream approaches either follow "detect-then-segment" strategy (e.g., Mask R-CNN), or embedding vectors first then cluster pixels into individual instances. this paper, we view task from completely new perspective by introducing notion "instance categories", which assigns...

We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in context. This omnivore model take any single-modality or data input indiscriminately (e.g., interleaved image, text video) through one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, together with tokens form an sequence. Emu is then end-to-end trained unified objective of classifying the next token regressing embedding versatile...

Abstract High-quality broadband ultrasound transducers yield superior imaging performance in biomedical ultrasonography. However, proper design to perfectly bridge the energy between active piezoelectric material and target medium over operating spectrum is still lacking. Here, we demonstrate a new anisotropic cone-structured acoustic metamaterial matching layer that acts as an inhomogeneous with gradient impedance along propagation direction. When sandwiched unit medium, provides window...