Unsupervised Flow-Aligned Sequence-to-Sequence Learning for Video Restoration

• DOI: 10.48550/arxiv.2205.10195
• Publication Date: 2022-01-01
• Authors: Jing Lin; Xiaowan Hu; Yuanhao Cai; Haoqian Wang; Youliang Yan; Xueyi Zou; Yulun Zhang; Luc Van Gool
• Keywords: Sequence (biology); Optical Flow; Deblurring

How to properly model the inter-frame relation within video sequence is an important but unsolved challenge for restoration (VR). In this work, we propose unsupervised flow-aligned sequence-to-sequence (S2SVR) address problem. On one hand, model, which has proven capable of modeling in field natural language processing, explored first time VR. Optimized serialization shows potential capturing long-range dependencies among frames. other equip with optical flow estimator maximize its potential. The trained our proposed distillation loss, can alleviate data discrepancy and inaccurate degraded issues previous flow-based methods. With reliable flow, establish accurate correspondence multiple frames, narrowing domain difference between 1D 2D misaligned frames improving model. S2SVR superior performance VR tasks, including deblurring, super-resolution, compressed quality enhancement. Code models are publicly available at https://github.com/linjing7/VR-Baseline