Global Spatial-Temporal Information-based Residual ConvLSTM for Video Space-Time Super-Resolution

• DOI: 10.48550/arxiv.2407.08466
• Publication Date: 2024-07-11
• Authors: Congrui Fu; Hui Yuan; Shiqi Jiang; Guanghui Zhang; Liquan Shen; Raouf Hamzaoui
• Keywords: Temporal resolution

By converting low-frame-rate, low-resolution videos into high-frame-rate, high-resolution ones, space-time video super-resolution techniques can enhance visual experiences and facilitate more efficient information dissemination. We propose a convolutional neural network (CNN) for super-resolution, namely GIRNet. To generate highly accurate features thus improve performance, the proposed integrates feature-level temporal interpolation module with deformable convolutions global spatial-temporal information-based residual long short-term memory (convLSTM) module. In module, we leverage convolution, which adapts to deformations scale variations of objects across different scene locations. This presents solution than conventional convolution extracting from moving objects. Our effectively uses forward backward feature determine inter-frame offsets, leading direct generation interpolated frame features. convLSTM first is used derive input features, second previously computed as its initial cell state. adopts connections preserve spatial information, thereby enhancing output Experiments on Vimeo90K dataset show that method outperforms state-of-the-art in peak signal-to-noise-ratio (by 1.45 dB, 1.14 0.02 dB over STARnet, TMNet, 3DAttGAN, respectively), structural similarity index(by 0.027, 0.023, 0.006 visually.