Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models

• DOI: 10.48550/arxiv.2404.03827
• Publication Date: 2024-04-04
• Authors: Dennis Wu; Jerry Yao-Chieh Hu; Teng-Yun Hsiao; Han Liu
• Keywords: Hopfield network

We propose a two-stage memory retrieval dynamics for modern Hopfield models, termed $\mathtt{U\text{-}Hop}$, with enhanced capacity. Our key contribution is learnable feature map $\Phi$ which transforms the energy function into kernel space. This transformation ensures convergence between local minima of and fixed points within Consequently, norm induced by serves as novel similarity measure. It utilizes stored patterns learning data to enhance capacity across all models. Specifically, we accomplish this constructing separation loss $\mathcal{L}_\Phi$ that separates kernelized separating in Methodologically, $\mathtt{U\text{-}Hop}$ process consists of: \textbf{(Stage~I.)} minimizing more uniformed (local minimum) distribution, followed \textbf{(Stage~II.)} standard minimization retrieval. results significant reduction possible meta-stable states function, thus enhancing preventing confusion. Empirically, real-world datasets, demonstrate outperforms existing models SOTA measures, achieving substantial improvements both associative deep tasks.