On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learning

• DOI: 10.48550/arxiv.2210.06443
• Publication Date: 2022-01-01
• Authors: Lorenzo Bonicelli; M. Boschini; Angelo Porrello; Concetto Spampinato; Simone Calderara
• Keywords: Overfitting; Popularity; Code (set theory); Smoothness

Rehearsal approaches enjoy immense popularity with Continual Learning (CL) practitioners. These methods collect samples from previously encountered data distributions in a small memory buffer; subsequently, they repeatedly optimize on the latter to prevent catastrophic forgetting. This work draws attention hidden pitfall of this widespread practice: repeated optimization pool inevitably leads tight and unstable decision boundaries, which are major hindrance generalization. To address issue, we propose Lipschitz-DrivEn (LiDER), surrogate objective that induces smoothness backbone network by constraining its layer-wise Lipschitz constants w.r.t. replay examples. By means extensive experiments, show applying LiDER delivers stable performance gain several state-of-the-art rehearsal CL across multiple datasets, both presence absence pre-training. Through additional ablative highlight peculiar aspects buffer overfitting better characterize effect produced LiDER. Code is available at https://github.com/aimagelab/LiDER