A5101481141- Topic Modeling
- Natural Language Processing Techniques
- Multimodal Machine Learning Applications
- Text and Document Classification Technologies
- Advanced Graph Neural Networks
- Domain Adaptation and Few-Shot Learning
- Machine Learning and Algorithms
- Advanced Neural Network Applications
- Machine Learning in Healthcare
- Video Analysis and Summarization
- Anomaly Detection Techniques and Applications
- Speech and dialogue systems
- Reinforcement Learning in Robotics
- Machine Learning in Bioinformatics
- Adversarial Robustness in Machine Learning
- Big Data and Digital Economy
- Explainable Artificial Intelligence (XAI)
- Advanced Text Analysis Techniques
- COVID-19 diagnosis using AI
- Human Pose and Action Recognition
- Evaluation and Performance Assessment
- Image and Video Stabilization
- Expert finding and Q&A systems
- Graph Theory and Algorithms
- Complex Network Analysis Techniques
Shanghai Artificial Intelligence Laboratory
2020-2024
Fudan University
2019-2023
Amazon (United States)
2020-2023
Hong Kong University of Science and Technology
2023
University of Hong Kong
2023
Adversarial attacks for discrete data (such as texts) have been proved significantly more challenging than continuous images) since it is difficult to generate adversarial samples with gradient-based methods. Current successful attack methods texts usually adopt heuristic replacement strategies on the character or word level, which remains find optimal solution in massive space of possible combinations replacements while preserving semantic consistency and language fluency. In this paper, we...
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, Xipeng Qiu. Proceedings of the 59th Annual Meeting Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing consider shallow, static, and separately entity embeddings, which limits performance gains these models. Few works explore potential deep contextualized representation when injecting knowledge. In this paper, we propose Contextualized Language Knowledge Embedding (CoLAKE), jointly learns for both with extended MLM objective. Instead only CoLAKE extracts context an from...
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, Zheng Zhang. Proceedings of the 2019 Conference North American Chapter Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.
The Transformer model is widely successful on many natural language processing tasks. However, the quadratic complexity of self-attention limit its application long text. In this paper, adopting a fine-to-coarse attention mechanism multi-scale spans via binary partitioning (BP), we propose BP-Transformer (BPT for short). BPT yields $O(k\cdot n\log (n/k))$ connections where $k$ hyperparameter to control density attention. has good balance between computation and capacity. A series experiments...
In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-head to capture features from different scales. Based on linguistic perspective and analysis of pre-trained (BERT) huge corpus, further design strategy control scale distribution for each layer. Results three kinds tasks (21 datasets) show our outperforms standard consistently significantly small moderate size datasets.
In this paper, we propose and study a novel visual object tracking approach based on convolutional networks recurrent networks. The proposed is distinct from the existing approaches to tracking, such as filtering-based ones tracking-by-detection ones, in sense that system explicitly trained off-line track anonymous objects noisy environment. model end-to-end trainable, minimizing any adversarial effect mismatches representation between true underlying dynamics learning dynamics. We...
While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of 1D RoPE to video, with complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential effective adaptation which have not been fully considered in prior work. As part our analysis, we introduce challenging V-NIAH-D (Visual Needle-In-A-Haystack Distractors) task,...
Adversarial attacks for discrete data (such as texts) have been proved significantly more challenging than continuous images) since it is difficult to generate adversarial samples with gradient-based methods. Current successful attack methods texts usually adopt heuristic replacement strategies on the character or word level, which remains find optimal solution in massive space of possible combinations replacements while preserving semantic consistency and language fluency. In this paper, we...
Two important tasks at the intersection of knowledge graphs and natural language processing are graph-to-text (G2T) text-to-graph (T2G) conversion. Due to difficulty high cost data collection, supervised available in two fields usually on magnitude tens thousands, for example, 18K WebNLG~2017 dataset after preprocessing, which is far fewer than millions other such as machine translation. Consequently, deep learning models G2T T2G suffer largely from scarce training data. We present CycleGT,...
Sentences produced by abstractive summarization systems can be ungrammatical and fail to preserve the original meanings, despite being locally fluent. In this paper we propose remedy problem jointly generating a sentence its syntactic dependency parse while performing abstraction. If word introduce an erroneous relation summary, behavior must discouraged. The proposed method thus holds promise for producing grammatical sentences encouraging summary stay true-to-original. Our contributions of...
Self-attention mechanism becomes more and popular in natural language processing (NLP) applications. Recent studies show the Transformer architecture which relies mainly on attention achieves much success large datasets. But a raised problem is its generalization ability weaker than CNN RNN many moderate-sized We think reason can be attributed to unsuitable inductive bias of self-attention structure. In this paper, we regard as matrix decomposition propose an improved module by introducing...
Data collection for the knowledge graph-to-text generation is expensive. As a result, research on unsupervised models has emerged as an active field recently. However, most have to use non-parallel versions of existing small supervised datasets, which largely constrain their potential. In this paper, we propose large-scale, general-domain dataset, GenWiki. Our dataset 1.3M text and graph examples, respectively. With human-annotated test set, provide new benchmark future from graphs.
Neural Code Intelligence -- leveraging deep learning to understand, generate, and optimize code holds immense potential for transformative impacts on the whole society. Bridging gap between Natural Language Programming Language, this domain has drawn significant attention from researchers in both research communities over past few years. This survey presents a systematic chronological review of advancements intelligence, encompassing 50 representative models their variants, more than 20...
Recent advancements in Chain-of-Thought prompting have facilitated significant breakthroughs for Large Language Models (LLMs) complex reasoning tasks. Current research enhances the performance of LLMs by sampling multiple chains and ensembling based on answer frequency. However, this approach fails scenarios where correct answers are minority. We identify as a primary factor constraining capabilities LLMs, limitation that cannot be resolved solely predicted answers. To address shortcoming,...
Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models, yet scalable training remains significant challenge. We introduce suite of 256 SAEs, trained on each layer and sublayer the Llama-3.1-8B-Base model, with 32K 128K features. Modifications to state-of-the-art SAE variant, Top-K are evaluated across multiple dimensions. In particular, we assess generalizability SAEs base models longer contexts fine-tuned models....
OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning ability.OpenAI has claimed the main techinique behinds is reinforcement learining. Recent works use alternative approaches like knowledge distillation to imitate o1's style, but their effectiveness limited by capability ceiling of teacher model. Therefore, this paper analyzes roadmap achieving from perspective learning,...
With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing consider shallow, static, and separately entity embeddings, which limits performance gains these models. Few works explore potential deep contextualized representation when injecting knowledge. In this paper, we propose Contextualized Language Knowledge Embedding (CoLAKE), jointly learns for both with extended MLM objective. Instead only CoLAKE extracts context an from...
Cycle-consistent training is widely used for jointly learning a forward and inverse mapping between two domains of interest without the cumbersome requirement collecting matched pairs within each domain. In this regard, implicit assumption that there exists (at least approximately) ground-truth bijection such given input from either domain can be accurately reconstructed successive application respective mappings. But in many applications no expected to exist large reconstruction errors...
Cheng Jiayang, Lin Qiu, Tsz Chan, Tianqing Fang, Weiqi Wang, Chunkit Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zheng Zhang. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Yu Sun, Qipeng Hang Yan, Xipeng Qiu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023.
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zheng Chenghu Zhou, Xinbing Wang, Luoyi Fu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.