DreamVideo: Composing Your Dream Videos with Customized Subject and Motion

• DOI: 10.48550/arxiv.2312.04433
• Publication Date: 2023-01-01
• Authors: Yujie Wei; Shiwei Zhang; Zhiwu Qing; Hangjie Yuan; Zhi‐Heng Liu; Yu Liu; Yingya Zhang; Jingren Zhou; Hongming Shan
• Keywords: Adapter (computing)

Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory the challenging video task, as it requires controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from few static images desired subject target motion. DreamVideo decouples this task into two stages, learning motion learning, by leveraging pre-trained model. The aims accurately capture fine appearance provided images, which is achieved combining textual inversion fine-tuning our carefully designed identity adapter. In architect adapter fine-tune on given effectively model pattern. Combining these lightweight efficient adapters allows for flexible customization any with Extensive experimental results demonstrate superior performance over state-of-the-art methods customized generation. Our project page at https://dreamvideo-t2v.github.io.