Abstract The beneficial, complementary nature of visual and textual information to convey is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay image text form semantic meaning has been thoroughly studied linguistics communication sciences several decades, computer vision multimedia research remained on surface problem more less. An exception previous work that introduced two metrics Cross-Modal Mutual Information Semantic...

Gesture as language of non-verbal communication has been theoretically established since the 17th century. However, its relevance for visual arts expressed only sporadically. This may be primarily due to sheer overwhelming amount data that traditionally had processed by hand. With steady progress digitization, though, a growing number historical artifacts have indexed and made available public, creating need automatic retrieval art-historical motifs with similar body constellations or poses....

Two modalities are often used to convey information in a complementary and beneficial manner, e.g., online news, videos, educational resources, or scientific publications. The automatic understanding of semantic correlations between text associated images as well their interplay has great potential for enhanced multimodal web search recommender systems. However, is still an unsolved research problem. Recent approaches such image captioning focus on precisely describing visual content...

Iconography refers to the methodical study and interpretation of thematic content in visual arts, distinguishing it, e.g., from purely formal or aesthetic considerations. In iconographic studies, Iconclass is a widely used taxonomy that encapsulates historical, biblical, literary themes, among others. However, given hierarchical nature inherent complexity such taxonomy, it highly desirable use automated methods for (Iconclass-based) image classification. Previous studies either focused...

Event classification can add valuable information for semantic search and the increasingly important topic of fact validation in news. So far, only few approaches address image newsworthy event types such as natural disasters, sports events, or elections. Previous work distinguishes between a limited number relies on rather small datasets training. In this paper, we present novel ontology-driven approach images. We leverage large real-world news events to pursue two objectives: First, create...

The World Wide Web and social media platforms have become popular sources for news information. Typically, multimodal information, e.g., image text is used to convey information more effectively attract attention. While in most cases content decorative or depicts additional it has also been leveraged spread misinformation rumors recent years. In this paper, we present a web-based demo application that automatically quantifies the cross-modal relations of entities~(persons, locations, events)...

In this paper, we introduce iART: an open Web platform for art-historical research that facilitates the process of comparative vision. The system integrates various machine learning techniques keyword- and content-based image retrieval as well category formation via clustering. An intuitive GUI supports users to define queries explore results. By using a state-of-the-art cross-modal deep approach, it is possible search concepts were not previously detected by trained classification models....

The recognition of handwritten mathematical expressions in images and video frames is a difficult unsolved problem yet. Deep convectional neural networks are basically promising approach, but typically require large amount labeled training data. However, such dataset does not exist for the task formula recognition. In this paper, we introduce system that creates set synthesized examples which derived from LaTeX documents. For purpose, propose novel attention-based generative adversarial...

Event classification in images plays a vital role multimedia analysis especially with the prevalence of fake news on social media and Web. The majority approaches for event rely large sets labeled training data. However, image labels fine-grained instances (e.g., 2016 Summer Olympics) can be sparse, incorrect, ambiguous, etc. A few have addressed lack data but cover only events. Moreover, vision-language models that allow zero-shot few-shot prompting not yet been extensively exploited. In...

In this paper, we present a novel approach to estimate the relative depth of regions in monocular images. There are several contributions. First, task estimation is considered as learning-to-rank problem which offers advantages compared regression approaches. Second, clues human perception modeled systematic manner. Third, show that these can be and integrated appropriately Rankboost framework. For purpose, space-efficient version derived makes it applicable rank large number objects, posed...

Deep neural networks have been successfully applied to the task of visual concept classification. However, they require a large number training examples for learning. Although pre-trained deep are available some domains, usually be fine-tuned an envisaged target domain. Recently, approaches suggested that aimed at incrementally (or even endlessly) learning concepts based on Web data. Since tags images often noisy, normally filtering mechanisms employed in order remove ``spam'' not...

Monocular depth estimation is an essential but ill-posed (computer) vision task. While human visual perception of relies on several monocular clues, such as occlusion objects, relative height, usual object size, linear perspective, deep learning models have to implicitly learn these cues from labeled training data determine depth. In this paper, we investigate whether criteria are violated for certain image instances given a model's predictions. We consider the task ranking problem, i.e.,...

Video analysis platforms that integrate automatic solutions for multimedia and information retrieval enable various applications in many disciplines including film media studies, communication science, education. However, current video either focus on manual annotations or include only a few tools content analysis. In this paper, we present novel web-based platform called TIB AV-Analytics (TIB-AV-A). Unlike previous platforms, TIB-AV-A integrates state-of-the-art approaches the fields of...

arXiv is a popular pre-print server focusing on natural science disciplines (e.g. physics, computer science, quantitative biology). As platform with focus easy publishing services it does not provide enhanced search functionality -- but offers programming interfaces which allow external parties to add these services. This paper presents extensions of the open source framework Sanity Preserver (SP). With respect original framework, derestricts topical and allows for text-based visualisation...