1. Algorithmen zum Verkleinern von digitalen Videos
Wolfgang Effelsberg
Praktische Informatik IV
Universität Mannheim
Germany
2. Content
1. Motivation
2. Scaling und Cropping
3. Seam Carving for Still Images
4. An Improvement for Diagonal Lines
5. Seam Carving for Video
6. Conclusion and Outlook
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 2
von digitalen Videos 23. Juli 2012
3. 1. Motivation
• Mobile devices (smart phones, pad computers) are getting very popular for
showing videos.
• The representation of still images and videos on those devices requires an
adaptation to the screen size and the aspect ratio.
• Scaling and Cropping are easy to implement but they do not work very
well.
• Seam Carving is a promising technology for still images and videos.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 3
von digitalen Videos 23. Juli 2012
4. 2. Scaling and Cropping
Scaling
Scaling reduces the image linearly, without looking at its content.
„Letterboxing“ is used if the format does not fit on the screen.
Example:
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 4
von digitalen Videos 23. Juli 2012
5. Cropping
Cropping cuts the image parallel to the edges until the final format is reached,
without looking at the image content. Parts of the image can get lost.
Example:
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 5
von digitalen Videos 23. Juli 2012
6. 3. Seam Carving for Still Images
Removing of specific pixels from the image, based on the importance of the
content
Based on an „energy function“ describing the importance of different parts of the
content
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 6
von digitalen Videos 23. Juli 2012
7. Example
Reduce the width by 40 %
Energy is maximal on edge pixels in the image.
original image energy image
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 7
von digitalen Videos 23. Juli 2012
8. Naive Approach
Remove from each line the pixels with minimal energy
original image 200 pixels removed
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 8
von digitalen Videos 23. Juli 2012
9. Remove Entire Columns
Remove the 200 columns with the smallest energy
original image 200 columns with minimal
energy removed
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 9
von digitalen Videos 23. Juli 2012
10. Definition of a Seam
• A vertical seam is an 8-connected path of pixels, running from the top to the bottom
of the image and containing exactly one pixel per row.
• Horizontal seams are defined analogously.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 10
von digitalen Videos 23. Juli 2012
11. Advantage of Seams
Can be adapted better to the regions of interest (regions of high energy)
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 11
von digitalen Videos 23. Juli 2012
12. Seams for our Example
200 vertical seams with minimal energy removed
original image 200 vertical seams removed
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 12
von digitalen Videos 23. Juli 2012
13. Our Other Example
It os obvious that seam carving removes unimportant content first.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 13
von digitalen Videos 23. Juli 2012
14. Energy Functions (1)
A simple energy function based on the difference in light intensity between two pixels
δ δ
e( I ( x, y )) = I ( x, y ) + I ( x, y )
δx δy
Better (but more complicated) energy functions are possible, for example, based on
important objects found in the image.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 14
von digitalen Videos 23. Juli 2012
15. Energy Functions (2)
We use saliency and face recognition to identify regions of interest
δ δ
esal + face ( x, y ) = ws ⋅ saliency ( x, y ) + w f ⋅ face( x, y ) + I ( x, y ) + I ( x, y )
δx δy
original image saliency map face map seams with esal+face new
as energy function image
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 15
von digitalen Videos 23. Juli 2012
16. 4. An Improvement for Diagonal Lines
A problem with traditional seam carving: diagonal lines
original image width reduced to 40% by
seam carving
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 16
von digitalen Videos 23. Juli 2012
17. The Problem with Diagonal Lines (1)
When a seam crosses a diagonal line visible artefacts appear.
a diagonal line seams cutting that line seams removed
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 17
von digitalen Videos 23. Juli 2012
18. The Problem with Diagonal Lines (2)
Especially critical if several seams cross the line at the same place.
Idea: distribute the seams more evenly over the line.
Neigboring seams line after removing distribute seams evenly line after removing
cutting a line the seams the seams
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 18
von digitalen Videos 23. Juli 2012
19. Solution for Diagonal Lines (1)
Solution: modify the energy function between the computation of the seams: increase
the energy at intersection points
seam cutting a line modified energy energy function third seam
function and new modified again
seam
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 19
von digitalen Videos 23. Juli 2012
20. Solution for Diagonal Lines (2)
Result
original image traditional seam carving improved seam carving
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 20
von digitalen Videos 23. Juli 2012
21. 5. Seam Carving for Video
First idea: apply seam carving for each frame separately
the video gets shaky.
original video seam carved for each frame separately
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 21
von digitalen Videos 23. Juli 2012
22. Seam Carving for Video: First Idea
The video frames define a 3D block over time.
We remove 2D “seam surfaces”. The seam pixels of one surface are connected in
the third dimension.
We use a graph algorithm (min-cut max-flow) to discover optimal “seam surfaces”.
Problem: takes considerable time to compute.
time
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 22
von digitalen Videos 23. Juli 2012
23. Seam Carving for Videos: Second Idea
Fast seam carving for video
• Accumulate all energy values of all frames in the first frame.
• Compute the optimal seams there.
• Map those seams back to the block of frames.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 23
von digitalen Videos 23. Juli 2012
24. Example for the Second Idea
seams carved separately fast seam carving
for each frame
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 24
von digitalen Videos 23. Juli 2012
25. Another Example
scaling fast seam carving
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 25
von digitalen Videos 23. Juli 2012
26. 6. Conclusions and Outlook
• Seam Carving is a useful technique to reduce image sizes without losing
relevant content.
• Finding a good energy function is a real challenge.
• Improvements are possible for diagonal lines.
• Flicker removal is of critical importance for videos. Fast Seam Carving is
a useful algorithm for that problem.
• Seam carving does not work well if most of the content is relevant for the
viewer.
• Seam Carving can be combined with other techniques to create pleasant
smal-screen versions of video.
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 26
von digitalen Videos 23. Juli 2012
27. Vielen Dank …
… für Ihre Aufmerksamkeit!
Meine Mailadresse ist
effelsberg@informatik.uni-mannheim.de
Algorithmen zum Verkleinern Klagenfurt
Wolfgang Effelsberg 27
von digitalen Videos 23. Juli 2012