Creating music videos using automatic media analysis

• DOI: 10.1145/641007.641119
• Publication Date: 2004-04-19T17:18:43Z
• Authors: Jonathan Foote; Matthew Cooper; Andreas Girgensohn
• Keywords: 0202 electrical engineering, electronic engineering, information engineering; 02 engineering and technology

We present methods for automatic and semi-automatic creation of music videos, given an arbitrary audio soundtrack and source video. Significant audio changes are automatically detected; similarly, the source video is automatically segmented and analyzed for suitability based on camera motion and exposure. Video with excessive camera motion or poor contrast is penalized with a high unsuitability score, and is more likely to be discarded in the final edit. High quality video clips are then automatically selected and aligned in time with significant audio changes. Video clips are adjusted to match the audio segments by selecting the most suitable region of the desired length. Besides a fully automated solution, our system can also start with clips manually selected and ordered using a graphical interface. The video is then created by truncating the selected clips (preserving the high quality portions) to produce a video digest that is synchronized with the soundtrack music, thus enhancing the impact of both.