Invited panel presentation at the First Toronto International Stereoscopic 3D Conference

1. 2D to 3D conversion at CRC
A visual perception approach
Carlos Vazquez and Wa James Tam
3D-TV Research
Advanced Video Systems

2. 2D to S3D video conversion
Why? Where/When? How?
•Many S3D cinema screens available •Post-production: For cinema and •Fully manual
•Introduction of 3D-enabled TV sets TV production •Semi-automatic
•Large amount of 2D content •Broadcasting: For live content and •Automatic off-line
available legacy material •Automatic in real-time
•Limited content in original •TV set, STB: Legacy content from
stereoscopic format viewer library or 2D program
•Still difficult to capture
stereoscopic content
June 4, 2012 Advanced Video Systems -- 3D-TV Research 2

3. Real-time vs. off-line conversion
 On-line 2D-to-3D conversion
 Fully automatic
 Artefacts are not removed
 Speed of conversion is critical
 Non-veridical depth information frequently
used
 Off-line 2D-to-3D conversion
 Automatic, semi-automatic or manual
 Automatically generated content can be
corrected
 Normally human-assisted (post production)
 High quality and high cost
June 4, 2012 Advanced Video Systems -- 3D-TV Research 3

4. 2D to S3D conversion: General framework
 Strategy:
Define depth structure and
render new virtual view
 Input:
2D video sequence
 Output:
Stereoscopic/multi-view
video sequence
June 4, 2012 Advanced Video Systems -- 3D-TV Research 4

5. How to get the depth
Manually assignment of
• Human interpretation of the scene
depth by operator analysis of translates into depth information
the scene
Extracting depth from • Depth from static images
monocular cues in images • Depth from motion related cues
Assigning of depth from • Apply a know depth model
automatic analysis of the • Use surrogates for the depth
scene information
June 4, 2012 Advanced Video Systems -- 3D-TV Research 5

6. Automatic generation of depth information
 Two approaches are used:
 Extract depth information from pictorial
depth cues in images
 Assign depth based on general
assumptions about scenes
 Pictorial depth cues:
 Size, texture density, occlusion,
atmospheric haze, shadows
 Perspective
 Focus/defocus
 General assumptions
 Scene depth models Single image with several pictorial
 Surrogate depth based on colour and depth cues
shadings
June 4, 2012 Advanced Video Systems -- 3D-TV Research 6

8. Visual perception: An active process
 Content dependency:

Aoccdrnig to rscheearch at an Elingsh uinervtisy,
Past experience help in interpretation of visual content
it deosn't mttaer in waht oredr the ltteers in a
 Auto-correction:
wrod are, olny taht the frist and lsat ltteres are
 Fill-in missing information or minimize conflicting information
at the rghit pcleas. The rset can be a toatl mses
and you can sitll raed it wouthit a porbelm. Tihs
is bcuseae we do not raed ervey lteter by ilstef,
but the wrod as a wlohe.
http://kybele.psych.cornell.edu/~edelman/
June 4, 2012 Advanced Video Systems -- 3D-TV Research 8

9. Visual perception: Depth filling
June 4, 2012 Advanced Video Systems -- 3D-TV Research 9

10. Surrogate depth maps
Motivation
 Shadows offer relative depth
inside objects
 Small image features (rain drops,
snow flakes, dust) have depth
 Good separation between objects
in the scene
 Green and blue go back (Trees
and sky)
 Red goes to the front (Skin tones)
June 4, 2012 Advanced Video Systems -- 3D-TV Research 10

11. CRC-i3D: Real-time 2D to 3D video conversion
 Surrogate Depth Maps:
 Single Cue – using the Chroma component of the
images.
 Minimal adjustments to original depth map
 Rendering:
 Real-Time and no buffering required.
 Avoids disocclusion by smoothing depth map.
 Manual control of depth to ensure comfort
according to viewing conditions.
 Advantages:
 Very low processing requirements.
 No storage requirements.
 This all translates into less processing, heat and
power requirements.
June 4, 2012 Advanced Video Systems -- 3D-TV Research 11

12. Off-line conversion: depth-based approach
Depth generation Depth map editing New image generation
Generate Select Edit depth Render Fill-in
2D depth regions values image holes
S3D
 Generate initial depth: Automatic operation
 Edit depth map: Manual or semi-automatic
 Select objects: Semi-automatic operation
 Change depth of objects: Semi-automatic
 Render new view: Automatic operation
 Fill-in holes: Automatic operation with human
supervision.
June 4, 2012 Advanced Video Systems -- 3D-TV Research 12

13. Depth-based approach: Main characteristics
 Intermediate depth master: Can be used to render the
stereoscopic content for different targets
 Depth relations are easy to evaluate
 Depth is continuous: Slow variations in depth are easily
represented
 Depth information is represented as an image: Easy to
store and transmit
June 4, 2012 Advanced Video Systems -- 3D-TV Research 13

14. CRC-DMEG: Depth editing tool
June 4, 2012 Advanced Video Systems -- 3D-TV Research 14

15. CRC-DMEG: Depth editing tool
Simple histogram equalization of green channel
June 4, 2012 Advanced Video Systems -- 3D-TV Research 15

16. June 4, 2012 Advanced Video Systems -- 3D-TV Research 16