Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

• DOI: 10.48550/arxiv.2307.08013
• Publication Date: 2023-01-01
• Authors: Haobo Song; Soumajit Majumder; Tao Lin
• Keywords: Benchmarking; TRACE (psycholinguistics); Footprint; Through-the-lens metering; Trade-off

Implicit models such as Deep Equilibrium Models (DEQs) have garnered significant attention in the community for their ability to train infinite layer with elegant solution-finding procedures and constant memory footprint. However, despite several attempts, these methods are heavily constrained by model inefficiency optimization instability. Furthermore, fair benchmarking across relevant vision tasks is missing. In this work, we revisit line of implicit trace them back original weight-tied models. Surprisingly, observe that more effective, stable, well efficient on tasks, compared DEQ variants. Through lens simple-yet-clean models, further study fundamental limits capacity propose use distinct sparse masks improve capacity. Finally, practitioners, offer design guidelines regarding depth, width, sparsity selection demonstrate generalizability our insights other learning paradigms.