Videogenic: Identifying Highlight Moments in Videos with Professional Photographs as a Prior

David Chuan-En Lin¹Fabian Caba Heilbron²Joon-Young Lee²Oliver Wang²Nikolas Martelaro¹

¹Carnegie Mellon University

²Adobe Research

C&C Paper NeurIPS Paper Video

Videogenic system overview — Videogenic leverages high-quality photographs as a prior to identify domain-specific highlights. For example, graduation photographs may frequently include people tossing their graduation caps.

Summary

This paper investigates the challenge of extracting highlight moments from videos. To perform this task, we need to understand what constitutes a highlight for arbitrary video domains while at the same time being able to scale across different domains. Our key insight is that photographs taken by photographers tend to capture the most remarkable or photogenic moments of an activity. Drawing on this insight, we present Videogenic, a technique capable of creating domain-specific highlight videos for a diverse range of domains. In a human evaluation study (N=50), we show that a high-quality photograph collection combined with CLIP-based retrieval (which uses a neural network with semantic knowledge of images) can serve as an excellent prior for finding video highlights. In a within-subjects expert study (N=12), we demonstrate the usefulness of Videogenic in helping video editors create highlight videos with lighter workload, shorter task completion time, and better usability.

Video

About 2 min read

Example Results

Videogenic identifies the most highlight-worthy moments of an activity or event. Examples include the officiant address of the wedding, the cars drifting, the skateboard kickflip, the graduation hat toss, the breakdance power move, the bird carrying its prey, and the weightlifter completing the clean and jerk.

Officiant address

Cars drifting

Skateboard kickflip

Graduation hat toss

Breakdance power move

Bird carrying prey

Completing clean and jerk

Rose blooming

Parkour jump

Sun rising

Solar eclipse peak

Fireworks burst

Placing snowman head

Peacock displaying feathers

Jumping from aircraft

Riding the wave

Video Sources

Highlight Graph

The highlight graph visualizes the distribution of predicted highlight scores across the video (a). The user may scrub through the graph to inspect a corresponding video frame and its highlight score (b).

The user may brush through the highlight graph to select an interval of the video to use for the highlight video (a). The interface displays a dashed line and a text label to indicate the average highlight value of the selected interval (b).

Example video frames and their corresponding highlight scores within a long skydiving video, using the keyword skydiving. The top-left corner displays the photograph collection used by Videogenic.

More Example Highlight Graphs

wedding

fireworks

breakdance

rafting