Volume 21 Issue 7 - March 2, 2012 PDF
Motion-Aware Temporal Coherence for Video Resizing
Yu-Shuen Wang1, Hongbo Fu2, Olga Sorkine3, Tong-Yee Lee1,*, Hans-Peter Seidel4
1 Department of Computer Science and Information Engineering, College of Electrical Engineering and Computer Science, National Cheng Kung University
2 Department of Computer Science and Engineering, City University of Hong Kong
3 Department of Computer Science, New York University
4 Max Planck Institute Informatik
Font Enlarge
Research on automatic resizing of media is becoming ever more important with the proliferation of display units, such as television, notebooks, PDAs and cell phones, which all come in different aspect ratios and resolutions. In this paper, we introduce a content aware technique (see figure 1) which considers the interior contents while resizing the videos. Specifically, we represent an image/frame with a grid mesh and then warp the mesh based on the saliency measure. Unlike the previous methods [1, 2, 3], which strove to preserve the prominent objects untouched, our method [3][4] allows them to be scaled uniformly, enabling the distortion propagation in multiple directions. In addition to the resizing of static images, we extend our resizing technique to videos [4]. The most important issue on this extension is the temporal coherence since the interior contents keep changing when the video is played. Due to the camera and object motions, simply preserving consistent resizing of temporally adjacent pixels cannot achieve temporal coherence and thus, resulting in flickering or waving artifacts. To solve this problem, we detect the camera motion based on the SIFT features and then decompose the scene into foreground and background regions. Obviously, the background motions depend on the camera while the foreground motions are arbitrary. We introduce different constraints to preserve their temporal coherences due to their different natures. All the criteria are formulated into energy terms and we solve for the resized videos by minimizing the objective function. Experimental results show our method outperforms previous methods for a variety of images (see figure 2). More results and our paper can be found in http://graphics.csie.ncku.edu.tw/VideoResizing/
Figure 1: Overview of our automatic content-aware video resizing framework. We align the original frames of a video clip to a common coordinate system by estimating interframe camera motion, so that corresponding components have roughly the same spatial coordinates. We achieve spatially and temporally coherent resizing of the aligned frames by preserving the relative positions of corresponding components within a grid-based optimization framework. The final resized video is reconstructed by transforming every video frame back to the original coordinate system.
Figure 2: From left to right columns: the original frame images, resizing results with homogeneous resizing, [1], [2], the na¨ıve extension of [3], and our method. Clearly, only our method can well preserve the visually prominent features while successfully retaining temporal coherence. Due to the motion-oblivious temporal coherence constraints, the previous content-aware resizing methods often cause inconsistent alteration of corresponding features across frames, e.g., the white bunny in the first example, the arch in the second example and the woman’s body in the third example.

  1. RUBINSTEIN, M., SHAMIR, A., AND AVIDAN, S. 2008. Improved seam carving for video retargeting. ACM Trans. Graph. 27, 3.
  2. WOLF, L., GUTTMANN, M., AND COHEN-OR, D. 2007. Non-homogeneous content-driven video-retargeting. In Proceedings of IEEE ICCV, 1–6.
  3. Yu-Shuen Wang, Chiew-Lan Tai, Olga Sorkine, and Tong-Yee Lee," Optimized Scale-and-Stretch for Image Resizing,"  ACM Transactions on Graphics (SIGGRAPH Asia 2008 issue), Vol. 27, No. 5, Dec. 2008, Article 118.
  4. Yu-Shuen Wang, Hongbo Fu, Olga Sorkine, Tong-Yee Lee and Hans-Peter Seidel, "Motion-Aware Temporal Coherence for Video Resizing," ACM Transaction on Graphics (Proceedings of SIGGRAPH Asia 2009), Vol. 28, No. 5, Dec. 2009, Article 118.
< Previous
Next >
Copyright National Cheng Kung University