FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration

• DOI: 10.48550/arxiv.2307.16617
• Publication Date: 2023-01-01
• Authors: Zhijian Huang; Sihao Lin; Guiyu Liu; Mukun Luo; Chaoqiang Ye; Hang Xu; Xiaojun Chang; Xiaodan Liang
• Keywords: Benchmark (surveying); Modality (human–computer interaction); Modalities

Multi-modality fusion and multi-task learning are becoming trendy in 3D autonomous driving scenario, considering robust prediction computation budget. However, naively extending the existing framework to domain of multi-modality remains ineffective even poisonous due notorious modality bias task conflict. Previous works manually coordinate with empirical knowledge, which may lead sub-optima. To mitigate issue, we propose a novel yet simple multi-level gradient calibration across tasks modalities during optimization. Specifically, gradients, produced by heads used update shared backbone, will be calibrated at backbone's last layer alleviate Before gradients further propagated branches their magnitudes again same level, ensuring downstream pay balanced attention different modalities. Experiments on large-scale benchmark nuScenes demonstrate effectiveness proposed method, eg, an absolute 14.4% mIoU improvement map segmentation 1.4% mAP detection, advancing application learning. We also discuss links between tasks.