A5036418431- Multimodal Machine Learning Applications
- Domain Adaptation and Few-Shot Learning
- Topic Modeling
- Human Pose and Action Recognition
- Advanced Image and Video Retrieval Techniques
- Parallel Computing and Optimization Techniques
- Advanced Vision and Imaging
- Video Analysis and Summarization
- Generative Adversarial Networks and Image Synthesis
- Advanced Data Storage Technologies
- Interconnection Networks and Systems
- Natural Language Processing Techniques
- Advanced Nanomaterials in Catalysis
- Advanced biosensing and bioanalysis techniques
- Metal-Organic Frameworks: Synthesis and Applications
- Computer Graphics and Visualization Techniques
- Advanced Image Processing Techniques
- Electrocatalysts for Energy Conversion
- Nanoplatforms for cancer theranostics
- Nanocluster Synthesis and Applications
- Distributed and Parallel Computing Systems
- Image Processing Techniques and Applications
- Nanoparticle-Based Drug Delivery
- Video Coding and Compression Technologies
- Handwritten Text Recognition Techniques
Nankai University
2019-2024
Nanjing University of Science and Technology
2021-2024
ShanghaiTech University
2023-2024
Chinese People's Armed Police Force
2024
University of California, Berkeley
2021-2024
JiangSu Armed Police General Hospital
2024
Fujian Institute of Research on the Structure of Matter
2023
Chinese Academy of Sciences
2023
Tan Kah Kee Innovation Laboratory
2023
Alpha Omega Alpha Medical Honor Society
2022
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as single unit, propose to decompose them into three modular components related subject appearance, location, and relationship other objects. This allows us flexibly adapt containing different types of information in end-to-end framework. our model, which call the Modular Attention Network (MAttNet), two attention are...
Recent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based 6 popular TV shows. TVQA consists of 152,545 pairs from 21,793 clips, spanning over 460 hours video. Questions are designed be compositional nature, requiring systems jointly localize relevant moments within clip, comprehend subtitle-based...
We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in hierarchical structure, where local context of video frame is captured by Cross-modal Transformer via fusion, and global Temporal Transformer. In addition to standard Masked Language Modeling (MLM) Frame (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), the model predicts both temporal alignment; (ii) Order (FOM), right...
Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose unified framework for the tasks of referring expression comprehension and generation. Our model is composed three modules: speaker, listener, reinforcer. The speaker generates expressions, listener comprehends reinforcer introduces reward function guide sampling more discriminative expressions. listener-speaker modules trained jointly in an end-to-end...
Hao Tan, Licheng Yu, Mohit Bansal. Proceedings of the 2019 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
We present the task of Spatio-Temporal Video Question Answering, which requires intelligent systems to simultaneously retrieve relevant moments and detect referenced visual concepts (people objects) answer natural language questions about videos. first augment TVQA dataset with 310.8K bounding boxes, linking depicted objects in answers. name this augmented version as TVQA+. then propose Answerer Grounded Evidence (STAGE), a unified framework that grounds evidence both spatial temporal...
In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs is collected using automatically produced fill-in-the-blank templates designed to gather targeted about: people and objects, their appearances, activities, interactions, as well inferences about general scene or its broader context. We provide several analyses demonstrate applicability two description generation tasks: generation,...
Traditional sparse image models treat color pixel as a scalar, which represents channels separately or concatenate monochrome image. In this paper, we propose vector representation model for images using quaternion matrix analysis. As new tool representation, its potential applications in several image-processing tasks are presented, including reconstruction, denoising, inpainting, and super-resolution. The proposed the matrix, where quaternion-based dictionary learning algorithm is...
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four datasets (COCO, Visual Genome, Conceptual Captions, SBU Captions), which can power heterogeneous downstream V+L tasks with multimodal embeddings. We design tasks: Masked Language Modeling...
Abstract High‐entropy materials are composed of five or more metal elements with equimolar near‐equimolar concentrations within one crystal structure, which offer remarkable structural properties for many applications. Despite previously reported entropy‐driven stabilization mechanisms, high‐entropy still tend to decompose produce a variety derivatives under operating conditions. In this study, we use transition‐metal (Ni, Co, Ni, Zn, V)‐based metal–organic frameworks (HE‐MOFs) as the...
In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs is collected using automatically produced fill-in-the-blank templates designed to gather targeted about: people and objects, their appearances, activities, interactions, as well inferences about general scene or its broader context. We provide several analyses demonstrate applicability two description generation tasks: generation,...
Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case pre-existing photographs from web. As an alternative, we propose a method that is able to compute 3D model human body and its outfit single photograph with little interaction. Our algorithm only capture global shape overall geometry it can also extract physical properties (i.e., material parameters needed for simulation) cloth. Unlike...
We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects most representative (summary) photos, and then composes natural language story for album. For this task, we make use Visual Storytelling dataset composed three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode album select compose story. Automatic human evaluations show achieves better performance on selection, generation, retrieval than baselines.
Abstract Inconvenient dual-laser irradiation and tumor hypoxic environment as well limited judgment of treating region have impeded the development combined photothermal photodynamic therapies (PTT PDT). Herein, Bi 2 Se 3 @AIPH nanoparticles (NPs) are facilely developed to overcome these problems. Through a one-step method, free radical generator (AIPH) phase transition material (lauric acid, LA, 44–46 °C) encapsulated in hollow bismuth selenide (Bi NPs). Under single 808-nm laser at area,...
Embodied Question Answering (EQA) is a relatively new task where an agent asked to answer questions about its environment from egocentric perception. EQA as introduced in [8] makes the fundamental assumption that every question, e.g., ``what color car?", has exactly one target (``car") being inquired about. This puts direct limitation on abilities of agent. We present generalization -- Multi-Target (MT-EQA). Specifically, we study have multiple targets them, such ``Is dresser bedroom bigger...
We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given clip with aligned subtitles as premise, paired natural language hypothesis based on the content, model needs to infer whether is entailed or contradicted by given clip. A large-scale dataset, named Violin (VIdeO-and-Language INference), introduced this which consists 95,322 video-hypothesis pairs from 15,887 clips, spanning over 582 hours video. These clips contain rich content...
Rigid 2D In-MOF nanosheets is investigated as electrocatalysts for nitrogen electroreduction in the entire pH range first time, which follows an enzymatic mechanism, with potential determining step being *H 2 NNH * → *NH + NH 3 .