Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models

• DOI: 10.48550/arxiv.2303.06628
• Publication Date: 2023-01-01
• Authors: Zangwei Zheng; Mingyuan Ma; Kai Wang; Ziheng Qin; Xiangyu Yue; Yang You
• Keywords: Feature (linguistics); Transfer of learning; Code (set theory); Benchmark (surveying); Feature vector

Continual learning (CL) can help pre-trained vision-language models efficiently adapt to new or under-trained data distributions without re-training. Nevertheless, during the continual training of Contrastive Language-Image Pre-training (CLIP) model, we observe that model's zero-shot transfer ability significantly degrades due catastrophic forgetting. Existing CL methods mitigate forgetting by replaying previous data. However, since CLIP dataset is private, replay cannot access pre-training dataset. In addition, previously learned downstream tasks enhance their performance but comes at cost sacrificing performance. To address this challenge, propose a novel method ZSCL prevent degradation in both feature and parameter space. space, reference introduced for distillation between current initial models. The should have semantic diversity no need be labeled, seen pre-training, matched image-text pairs. large shift averaging weights training. We more challenging Multi-domain Task Incremental Learning (MTIL) benchmark evaluate different methods, where are from various domains instead class-separated single Our outperforms other traditional class-incremental setting MTIL 9.7% average score. code locates https://github.com/Thunderbeee/ZSCL.