Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation

• DOI: 10.48550/arxiv.2307.06940
• Publication Date: 2023-01-01
• Authors: Yingqing He; Menghan Xia; Haoxin Chen; Xiaodong Cun; Yuan Gong; Jinbo Xing; Yong Zhang; Xintao Wang; Chao Weng; Ying Shan; Qifeng Chen
• Keywords: Leverage (statistics)

Generating videos for visual storytelling can be a tedious and complex process that typically requires either live-action filming or graphics animation rendering. To bypass these challenges, our key idea is to utilize the abundance of existing video clips synthesize coherent by customizing their appearances. We achieve this developing framework comprised two functional modules: (i) Motion Structure Retrieval, which provides candidates with desired scene motion context described query texts, (ii) Structure-Guided Text-to-Video Synthesis, generates plot-aligned under guidance structure text prompts. For first module, we leverage an off-the-shelf retrieval system extract depths as structure. second propose controllable generation model offers flexible controls over characters. The are synthesized following structural appearance instruction. ensure consistency across clips, effective concept personalization approach, allows specification character identities through Extensive experiments demonstrate approach exhibits significant advantages various baselines.