Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition

• DOI: 10.48550/arxiv.2407.21033
• Publication Date: 2024-07-17
• Authors: Jielong Tang; Zhenxing Wang; Ziyang Gong; Jianxing Yu; Shuang Wang; Jian Yin
• Keywords: FOS: Computer and information sciences; Computer Science - Computation and Language; Artificial Intelligence (cs.AI); Computer Science - Artificial Intelligence; Computer Vision and Pattern Recognition (cs.CV); Computer Science - Computer Vision and Pattern Recognition; Computation and Language (cs.CL); Information Retrieval (cs.IR); Computer Science - Information Retrieval

Grounded Multimodal Named Entity Recognition (GMNER) is an emerging information extraction (IE) task, aiming to simultaneously extract entity spans, types, and entity-matched bounding box groundings in images from given sentence-image pairs data. Recent unified methods employing machine reading comprehension (MRC-based) frameworks or sequence generation-based models face challenges understanding the relationships of multimodal entities. MRC-based frameworks, utilizing human-designed queries, struggle model intra-entity connections. Meanwhile, outputs excessively rely on inter-entity dependencies due pre-defined decoding order. To tackle these, we propose a novel framework named Multi-grained Query-guided Set Prediction Network (MQSPN) learn appropriate at levels. Specifically, MQSPN consists Query (MQS) (MSP). MQS combines specific type-grained learnable entity-grained queries adaptively strengthen connections by explicitly aligning visual regions with textual spans. Based solid modeling, MSP reformulates GMNER as set prediction, enabling parallel prediction entities non-autoregressive manner, eliminating redundant preceding sequences, guiding establish global matching perspective. Additionally, boost better alignment two-level relationships, also incorporate Fusion Net (QFNet) work glue network between MSP. Extensive experiments demonstrate that our approach achieves state-of-the-art performances widely used benchmarks. Notably, method improves 2.83% F1 difficult fine-grained benchmark.