Knowledge Distillation via Constrained Variational Inference

• DOI: 10.1609/aaai.v36i7.20786
• Publication Date: 2022-07-04T09:40:40Z
• Authors: Ardavan Saeedi; Yuria Utsumi; Li Sun; Kayhan Batmanghelich; Li-wei Lehman
• Keywords: Discriminative model; Graphical model; Representation; Feature (linguistics)

Knowledge distillation has been used to capture the knowledge of a teacher model and distill it into student with some desirable characteristics such as being smaller, more efficient, or generalizable. In this paper, we propose framework for distilling powerful discriminative neural network commonly graphical models known be interpretable (e.g., topic models, autoregressive Hidden Markov Models). Posterior latent variables in these proportions models) is often feature representation predictive tasks. However, posterior-derived features are have poor performance compared learned via purely approaches. Our constrains variational inference posterior similarity preserving constraint. This constraint distills by ensuring that input pairs (dis)similar also model. By adding scheme, guide reasonable density data while having which close possible those To make our applicable wide range build upon Automatic Differentiation Variational Inference (ADVI), black-box models. We demonstrate effectiveness on two real-world tasks disease subtyping trajectory modeling.