LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking

• DOI: 10.48550/arxiv.2403.04303
• Publication Date: 2024-03-07
• Authors: Jialin Li; Qiang Nie; Weifu Fu; Yuhuan Lin; Guangpin Tao; Yong Liu; Chengjie Wang
• Keywords: Rank (graph theory)

Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape increasingly large depth can even reach dozens, further exacerbating issue. To mitigate problem, we introduce LORS (LOw-rank Residual Structure). allows modules share majority requiring much smaller unique ones per module match or surpass performance using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it decoders query-based object detector, conduct extensive experiments widely used MS COCO dataset. Experimental results demonstrate effectiveness method, as with 70\% reduction parameters decoder, still enables model achieve comparable