Large scale Vision Language (VL) models have shown tremendous success in aligning representations between visual and text modalities. This enables remarkable progress zero-shot recognition, image generation & editing, many other exciting tasks. However, VL tend to over-represent objects while paying much less attention verbs, require additional tuning on video data for best action recognition performance. While previous work relied large-scale, fully-annotated data, this we propose an...

Vision-language foundation models have shown impressive capabilities across various zero-shot tasks, including training-free localization and grounding, primarily focusing on localizing objects in images. However, leveraging those to localize actions events videos is challenging, as less physical outline are usually described by higher-level concepts. In this work, we propose VideoGEM, the first spatial action grounding method based pretrained image- video-language backbones. Namely, adapt...

Disorders of ocular perfusion are associated with huge amout diseases, including such socially significant as diabetic retinopathy and glaucoma. To date, there is no gold standard for measuring perfusion. An innovative method two-dimensional assessment eye blood flow — laser speckle flowgraphy (LSFG) has been developed recent years implemented in ophthalmological practice. Purpose : to evaluate the possibilities LSFG determining find out age dependence obtained indicators flow. Materials...

Contrastive learning has become an important tool in representations from unlabeled data mainly relying on the idea of minimizing distance between positive pairs, e.g., views same images, and maximizing negative different images. This paper proposes a new variation contrastive objective, Group Ordering Constraints (GroCo), that leverages sorting distances pairs computing respective loss based how many have larger than thus are not ordered correctly. To this end, GroCo is differentiable...

Instructional videos are an excellent source for learning multimodal representations by leveraging video-subtitle pairs extracted with automatic speech recognition systems (ASR) from the audio signal in videos. However, contrast to human-annotated captions, both and subtitles naturally differ visual content of thus provide only noisy supervision learning. As a result, large-scale annotation-free web video training data remains sub-optimal text-video models. In this work, we propose leverage...

Large scale Vision-Language (VL) models have shown tremendous success in aligning representations between visual and text modalities. This enables remarkable progress zero-shot recognition, image generation & editing, many other exciting tasks. However, VL tend to over-represent objects while paying much less attention verbs, require additional tuning on video data for best action recognition performance. While previous work relied large-scale, fully-annotated data, this we propose an...

Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for languages other than English still lags. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to improve multilingual retrieval. Inspired by fact that outperforms languages, we train student model using input text different match cross-modal predictions from teacher models English. cross entropy based objective which forces distribution over student's similarity...

Large-scale noisy web image-text datasets have been proven to be efficient for learning robust vision-language models. However, transfer them the task of video retrieval, models still need fine-tuned on hand-curated paired text-video data adapt diverse styles descriptions. To address this problem without hand-annotated pairs, we propose a new setting, retrieval with uncurated & unpaired data, that uses only text queries together videos during training any data. end, an approach, In-Style,...

Spatio-temporal grounding describes the task of localizing events in space and time, e.g., video data, based on verbal descriptions only. Models for this are usually trained with human-annotated sentences bounding box supervision. This work addresses from a multimodal supervision perspective, proposing framework spatio-temporal action loose subtitle only, without human annotation. To end, we combine local representation learning, which focuses leveraging fine-grained spatial information,...

Self-supervised learning on large-scale multi-modal datasets allows semantically meaningful embeddings in a joint representation space without relying human annotations. These enable zero-shot cross-modal tasks like retrieval and classification. However, these methods often struggle to generalize well out-of-domain data as they ignore the semantic structure present modality-specific embeddings. In this context, we propose novel Semantic-Structure-Preserving Consistency approach improve...

Vision-language models trained on large, randomly collected data had significant impact in many areas since they appeared. But as show great performance various fields, such image-text-retrieval, their inner workings are still not fully understood. The current work analyses the true zero-shot capabilities of those models. We start from analysis training corpus assessing to what extent (and which of) test classes really and how this correlates with individual performance. follow up...

The task of multimodal learning has seen a growing interest recently as it allows for training neural architectures based on different modalities such vision, text, and audio. One challenge in models is that they need to jointly learn semantic concepts their relationships across input representations. Capsule networks have been shown perform well context capturing the relation between low-level features higher-level concepts. However, capsules so far mainly used only small-scale fully...

The paper shows evolution of scientific views in determining the intension 'investment' and its role development economy. main regularities historical process related to formation knowledge investment are examined. author ascertains interrelation between theoretical conceptions their impact on transformation economic processes. There four stages essence meaning category identified: Stage 1: from first commercial relations great geological discoveries. At that stage concept was not yet...

Contrastive learning has become an important tool in representations from unlabeled data mainly relying on the idea of minimizing distance between positive pairs, e.g., views same images, and maximizing negative different images. This paper proposes a new variation contrastive objective, Group Ordering Constraints (GroCo), that leverages sorting distances pairs computing respective loss based how many have larger than thus are not ordered correctly. To this end, GroCo is differentiable...

Self-supervised learning on large-scale multi-modal datasets allows semantically meaningful embeddings in a joint representation space without relying human annotations. These enable zero-shot cross-modal tasks like retrieval and classification. However, these methods often struggle to generalize well out-of-domain data as they ignore the semantic structure present modality-specific embeddings. In this context, we propose novel Semantic-Structure-Preserving Consistency approach improve...

Large-scale noisy web image-text datasets have been proven to be efficient for learning robust vision-language models. However, when transferring them the task of video retrieval, models still need fine-tuned on hand-curated paired text-video data adapt diverse styles descriptions. To address this problem without hand-annotated pairs, we propose a new setting, retrieval with uncurated & unpaired data, that during training utilizes only text queries together videos any data. end, an approach,...