This was a talk I gave to a 4th year (senior-level) undergraduate class in Human Computer Interaction at Ryerson University. The talk focused on the different methods of displaying Stereoscopic 3D content, as well as the methods on generating such content. Technologies such as DLP 3DTVs, 3D theatres, and autostereoscopic displays are discussed. For the methods, 3D cameras, 2D to 3D conversion and other popular methods are discussed.
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Stereoscopic 3D: Generation Methods and Display Technologies for Industry and Home Entertainment
1. Stereoscopic 3D: Generation Methods
and Display Technologies for Industry
and Home Entertainment
Raymond Phan
Ph.D. Candidate
Multimedia and Distributed Computing (MDC) Research Laboratory
Department of Electrical and Computer Engineering
Ryerson University 1
Human Computer Interaction Guest Lecture
Thursday, March 8th, 2012
2. Outline of Presentation
• Introduction
– Stereoscopy / 3D Vision
• What is 3D all about??
• Depth and Disparity
• Some methods on generating 3D content
– Conversion from 2D imagery / video to 3D
• Cut and Paste Technique
• Depth Based Image Rendering – Recover Depth Maps
– Automated Methods – Using motion, focus cues, perspective
– Semi-Automated Methods
Human Computer Interaction Guest Lecture 2
Thursday, March 8th, 2012
3. Outline of Presentation – (2)
– Acquiring 3D content directly
• Stereo Rigs, Multi-camera Setup
• 3D cameras
• Displaying 3D Content
– Anaglyphs (very retro)
– 3D Theatres with polarized glasses
• RealD (most popular), IMAX
– Shutter glass technology
• nVidia 3D Vision, XpanD 3D, DLP projection systems,
DLP TVs
Human Computer Interaction Guest Lecture 3
Thursday, March 8th, 2012
4. Outline of Presentation – (3)
– Interference Filter Technology
• Based on projecting colours of different wavelengths to
each eye Dolby 3D, Panavision 3D
– Autostereoscopic Systems
• Technology without the use of glasses
– Parallax Barriers, Lenticular Arrays
– Single view vs. Multi-view systems
• Seen in the Nintendo 3DS, Fujifilm FinePix Real 3D
cameras, etc.
• Applications
• Conclusions
Human Computer Interaction Guest Lecture 4
Thursday, March 8th, 2012
5. Introduction
• So… what is stereoscopy / 3D vision?
– Creating the illusion of depth in an image or video
– Take images on flat displays, and make it “look real”
Human Computer Interaction Guest Lecture 5
Thursday, March 8th, 2012
6. Introduction – (2)
• Need to know some basic things first:
– Objects seen with the left eye are separated by
horizontal distances with the right eye disparity
– Greater/smaller the distance, the closer/farther the
object depth
Human Computer Interaction Guest Lecture 6
Thursday, March 8th, 2012
7. Generating 3D: 2D – 3D – (1)
• 1st Method: Cut and Paste Technique
– Used in IMAX’s 3D
DMR process
– 35 mm frames High res. digital Left-eye frames
– Right-eye frames Left frames objects are manually
shifted horizontally to create this new frame
Human Computer Interaction Guest Lecture 7
Thursday, March 8th, 2012
8. Generating 3D: 2D – 3D – (2)
– Remember disparity (close/far)! The closer the object,
the farther the shift needs to be
– Main Disadvantage: Very time consuming!
• Currently done on a frame-by-frame basis
• Due to this, only ~10 minutes of 35 mm video is 3D
converted Takes ~1 month to complete whole process
• Our MDC Project with IMAX: Goal Perform 2D to 3D
movie conversion faster
• Use a semi-automatic process to extract objects, and do
this every 10, or 20 frames or so
• In between frames, “guess” the best estimate of where the
objects are
Human Computer Interaction Guest Lecture 8
Thursday, March 8th, 2012
9. Generating 3D: 2D – 3D – (3)
• 2nd Method: Depth-Based Image Rendering (DBIR)
– 3D Content 1 2D Image + Depth Map
– Depth Map: Image containing depth of each image
• Closer Pixels == Light values, Farther Pixels == Dark values
– Orig. Image Left View. Right view Use depth
map (d(x,y)) to calculate shifted pixel from left view
Equation to
generate view
Right(x,y) =
Left(x+d(x,y),y)
Human Computer Interaction Guest Lecture 9
Thursday, March 8th, 2012
10. Generating 3D: 2D – 3D – (4)
• Commonly known as 2D to 3D Conversion
– Goal of 2D to 3D Conversion: Use an image and
determine what the best depth map is
– We use this depth map for conversion
– Use original single view image / frame as the left
view, and the depth map to create the right view
• There are two main methods to do this:
– Automated Methods Automatically examine
features in an image or frame and infer depth
Human Computer Interaction Guest Lecture 10
Thursday, March 8th, 2012
11. Generating 3D: 2D – 3D – (5)
– Semi-Automated Methods User-guided
• Mark certain areas of the image / frame on what you
think the depths should be at these locations
• Algorithm determines the rest of the depths
• Question: How do we know for sure that
we’re marking the proper depths?
– Been shown that as long as you mark depths in a
perceptually consistent way, perception is good
• Automated Methods:
– Popular Methods: Motion, Focus and Perspective
Human Computer Interaction Guest Lecture 11
Thursday, March 8th, 2012
12. Generating 2D: 2D – 3D – (6)
• Motion: Main Principle
– Objects that are closer
move faster
– Objects that are far
move slower
• Find motion vectors
– Find how much a pixel moves from one frame to
the next Calculate displacement vector
– Larger vector == Closer depth and vice-versa
Human Computer Interaction Guest Lecture 12
Thursday, March 8th, 2012
13. Generating 3D: 2D – 3D – (7)
• Potential Problems:
– Sometimes, far away objects move just as fast too
– Motion estimation (calculating motion vectors)
can be subject to error (i.e. very fast motion)
– If the image / frame is noisy, will corrupt
measurements
• Depth from focus: Main Principle
– Take multiple pictures of the same scene
– Each is taken with different camera parameters
Human Computer Interaction Guest Lecture 13
Thursday, March 8th, 2012
14. Generating 3D: 2D – 3D – (8)
• We basically change the focal length of the
camera
– Focal length : Distance from the image plane
to the surface to capture
– Crudely, we can change the focal length by
adjusting the zoom of your lens
• After, we find the amount of blur of an object
– In this aspect, sharper surfaces are closer, and
farther objects are more blurry
Human Computer Interaction Guest Lecture 14
Thursday, March 8th, 2012
15. Generating 3D: 2D – 3D – (9)
• We find a correlation between the depth, and
the amount of blur over the surfaces
– Finding multiple images at different focal lengths
is a must!
• Problems:
– Needs > 1 of same shot
• May not have such info
– Math is just too crazy
– Method rarely used now!
Human Computer Interaction Guest Lecture 15
Thursday, March 8th, 2012
16. Generating 3D: 2D – 3D – (10)
• Depth from perspective: Main Principle
– We use parallel lines and vanishing points in an
image or frame to give us a sense of depth
– Examples: Railroad Tracks, Tunnels, Roadsides
– These entities give us a sense of depth where they
appear to converge at a single point
– This single point would be the farthest point in the
image and the farthest depth
Human Computer Interaction Guest Lecture 16
Thursday, March 8th, 2012
17. Generating 3D: 2D – 3D – (11)
• Problems:
– Only a subset of
images / frames
fall into this
category
– Can only deal with
outdoor or with
scenes that have
perspective within them
• Not all images belong here!
Human Computer Interaction Guest Lecture 17
Thursday, March 8th, 2012
18. Generating 3D: 2D – 3D – (12)
• Semi-automatic methods:
– Mark some areas in an image / frame on what you
think the best depth should be
– Use this info to determine the rest of the depths
– This is the area that I am focusing on right now
• We can consider this as a case of multiple
object image segmentation
– Each “object” is a user-marked depth
– We decompose the rest of the image into
different objects i.e. different depths
Human Computer Interaction Guest Lecture 18
Thursday, March 8th, 2012
19. Generating 3D: 2D – 3D – (13)
•
• This method allows the user to fully control
the depth perception and experience
• Potential Problem:
– Takes more time because of user interaction and
computational complexity increases
Human Computer Interaction Guest Lecture 19
Thursday, March 8th, 2012
20. Generating 3D: 2D – 3D – (14)
• Another way to generate depth maps:
– Specialized hardware
• Example: ZCam Measures depth using bounced infra-red
light off of objects read in by a camera sensor
– Problem: Hardware is expensive!
Human Computer Interaction Guest Lecture 20
Thursday, March 8th, 2012
21. Direct 3D Acquisition – (1)
• Can directly acquire 3D information:
– Grabbing both left and right eye images / video
• 1st Method: Stereo Rigs
– Tripod with 2 cameras, separated by eye distance
– Drawbacks:
• Need 2 cameras! Synchronization!
• Difficult to separate cameras by eye distance
Human Computer Interaction Guest Lecture 21
Thursday, March 8th, 2012
22. Direct 3D Acquisition – (2)
• We can also use multi-camera stereo rigs
– Each pair of cameras is positioned at a different
point to capture the same scene
– Each viewpoint captures the objects in a different
way so that we can assemble all these together to
view a 3D object without glasses (more later)
Human Computer Interaction Guest Lecture 22
Thursday, March 8th, 2012
23. Direct 3D Acquisition – (3)
• Example: MERL 3DTV system (w/o glasses)
– 16 cameras and projectors for 16 viewpoints
– Depending on where you stand, you see a
different viewpoint Just like in real-life!
Human Computer Interaction Guest Lecture 23
Thursday, March 8th, 2012
24. Direct 3D Acquisition – (4)
• 2nd Method: 3D Cameras
– Specialized cameras specifically designed to take left
and right eye images
Human Computer Interaction Guest Lecture 24
Thursday, March 8th, 2012
25. Direct 3D Acquisition – (5)
• Non Digital 3D Cameras take left and right
images on two separate rolls of film
• Digital 3D Cameras (e.g. Fujifilm’s W1) take
left and right images and generate two
separate image files
• IMAX and specialized 3D video cameras
operate in the same way
– Two separate rolls of film
– For IMAX, the cameras are large as the film is
larger. Why? For higher resolution
Human Computer Interaction Guest Lecture 25
Thursday, March 8th, 2012
26. Displaying 3D Content – (1)
• Left & Right eye images are created
– How do we display these so we can perceive 3D?
– Many technologies exist to display 3D imagery and
video
• Let’s start off with the most basic one: Anaglyphs
– Left & right is filtered with separate colour filters
– Example: If you had a red colour filter, you determine
how much red a pixel has and that’s the output
– Each colour filter is chromatically different
• One filter cannot have any similarity in colour to the other
Human Computer Interaction Guest Lecture 26
Thursday, March 8th, 2012
27. Displaying 3D Content – (2)
• When one side is filtered with one colour, you
must choose the other filter to be a
contrasting colour
– How do we choose? Trichromacy theory states
that all colours are made up of Red, Green & Blue
– We basically choose the colour filters from this set
• Examples:
– Red and Cyan (Green + Blue) Filters
– Red and Green Filters
– Red and Blue Filters, etc.
Human Computer Interaction Guest Lecture 27
Thursday, March 8th, 2012
28. Displaying 3D Content – (3)
• After you filter each image separately, you
superimpose the results onto one image
• To view the images, you use anaglyph glasses,
where each side is of the same filter you used
– i.e. if you used Red for the left, and Cyan for the
right, we use anaglyph glasses that are of the
same order
– Here, the image with the red filter goes to the left
eye, and the cyan image goes to the right eye
Human Computer Interaction Guest Lecture 28
Thursday, March 8th, 2012
29. Displaying 3D Content – (4)
• As such, because we’re seeing two separate
images for two eyes, we thus perceive 3D
Human Computer Interaction Guest Lecture 29
Thursday, March 8th, 2012
30. Displaying 3D Content – (5)
• Advantages:
– Great for viewing without 3D technology
– Anaglyph glasses are pretty cheap
• Problems:
– Range of colours can be limited, as the
predominant colours in the images are of the
colour filters you applied
– Doesn’t work will if the range of colours in the
image are limited
Human Computer Interaction Guest Lecture 30
Thursday, March 8th, 2012
31. Displaying 3D Content – (6)
• 2nd method: 3D films in theatres with polarized
glasses
– 2 projectors Left & Right video projected
simultaneously on the theatre screen
– Views filtered with orthogonal polarizing filters
– Viewers wear low-cost polarized eyeglasses
– Each lens is orthogonally polarized with the other
Human Computer Interaction Guest Lecture 31
Thursday, March 8th, 2012
32. Displaying 3D Content – (7)
• What’s polarization!?
– Light can be viewed as a propagating wave
– Polarization determines the orientation of a wave’s
oscillations
– When passed through a
polarizing filter, orientation of
the light’s propagation changes
by forcing it through a slit
– Consequence – Not all light passes through
– Left view passed through a horizontal polarized filter
– Right view passed through a vertical polarized filter
Human Computer Interaction Guest Lecture 32
Thursday, March 8th, 2012
33. Displaying 3D Content – (8)
– Both views are shown simultaneously on a silver
perforated screen to preserve polarization
– Glasses Left lens has a horizontal filter
Right lens has a vertical filter
– Left blocks right view, and
right blocks left view!
• Drawbacks:
– Need to keep your head level
– Tilting your head causes the left and right views to
bleed into each other
– Image is darker, as only some of the light is sent
Human Computer Interaction Guest Lecture 33
Thursday, March 8th, 2012
34. Displaying 3D Content – (9)
• There is a way to combat “head level” issue
– Circular Polarization Used in RealD technology
– IMAX used former method Now they changed
– RealD is used in standard 3D theatres
– IMAX has the bigger screen, and better sound!
• What is circular polarization?
– We change the way the
wave propagates in a
circular motion
Human Computer Interaction Guest Lecture 34
Thursday, March 8th, 2012
35. Displaying 3D Content – (10)
• Each lens of the 3D projector continuously
changes polarization direction
• 3D glasses: Circularly polarized liquid crystal
that automatically adjust its polarization
– How is this possible?
– One lens is circularly polarizing clockwise, while
the other is polarizing counter-clockwise
– One lens is designed to filter clockwise images,
and counter-clockwise images for the other
– Each lens receives correct corresponding image
Human Computer Interaction Guest Lecture 35
Thursday, March 8th, 2012
36. Displaying 3D Content – (11)
• 3rd Method: Using shutter glasses
– Most popular in current 3DTVs on the market
– Also used in DLP Projection Systems
• Shutter glasses principle:
– Lenses are usually made of LCDs
– Used to separate the left and right views
– Lenses contain liquid crystals that block or pass
light in sync with an IR sensor, connected @ display
– Voltages are applied to the lenses so that one eye
blocks light, but the other one allows it through
Human Computer Interaction Guest Lecture 36
Thursday, March 8th, 2012
37. Displaying 3D Content – (12)
– Alternate this shutting off in sync with the image
displayed on the screen to show 3D, via IR sensor
– TV / monitor displays the left image, right lens is
blocked Allows left eye image to be seen
– After, we do for right image, with left lens blocked
Allows right eye image to be seen
Human Computer Interaction Guest Lecture 37
Thursday, March 8th, 2012
38. Displaying 3D Content – (13)
• Is used in nVidia 3D Vision Kit & XpanD 3D
– XpanD 3D: Company that markets shutter glass 3D
technology to homes and theatres
• Currently > 1000 theatres with shutter glass tech.
– nVidia 3D Vision: Kit for an nVidia video card
• IR sensor connected to video card to control views
• Only works with a compatible 3D monitor
• Advantages:
– No silver screen and keeping your head level
• Disadvantages: Shutter glasses are expensive!
– Need to replace batteries, high maintenance
Human Computer Interaction Guest Lecture 38
Thursday, March 8th, 2012
39. Displaying 3D Content – (14)
• DLP 3DTV technology further explained
– DLP: Digital Light Processing
– Backbone: Digital Micromirror Device
• Tiny mirrors direct light
• Device can have over 1 million mirrors!
– Each micromirror is either ON or OFF
• ON reflect light out towards screen
• OFF do not reflect out towards screen
(absorb it instead)
– Each mirror in the DLP 3DTV is
controlled by a pixel in the image
to display to the screen
Human Computer Interaction Guest Lecture 39
Thursday, March 8th, 2012
40. Displaying 3D Content – (15)
• For DLP 3DTVs, mirrors == diamond configuration
– One mirror displays two pixels of input data: How!?
• Each mirror shows one pixel, then does a half-pixel shift
downwards and shows the other pixel immediately below
• @ twice the normal frame so you can’t see the change
• Wait! Aren’t we losing 50% of the data?
Human Computer Interaction Guest Lecture 40
Thursday, March 8th, 2012
41. Displaying 3D Content – (16)
– No! The half-pixel shifting ensures same resolution
• Called SmoothPicture algorithm
– Saves bandwidth: Use same bandwidth for 3D images
– For a 2D image, the input data is the image itself
– For showing 3D, the left-eye image is shown first,
then the right-eye image is shown after ½-pix shifting
– LCD shutter glasses are in sync during each shift
• Drawbacks:
– Obviously, the TV is expensive
– Shutter glasses are high maintenance, and expensive
Human Computer Interaction Guest Lecture 41
Thursday, March 8th, 2012
42. Displaying 3D Content – (17)
• Next method: Interference Filter Technology
– Used in Dolby 3D and Panavision 3D systems
– A multispectral colour filter is used to filter
specific wavelengths of red, green and blue,
directed to the left eye
– Another colour filter used to filter different
wavelengths of red, green and blue, directed to
the right eye
– This uses glasses too Designed to filter the
same wavelengths in tune with each colour filter
Human Computer Interaction Guest Lecture 42
Thursday, March 8th, 2012
43. Displaying 3D Content – (18)
• This process is called: wavelength
multiplex visualization
• Advantages:
– No silver screen required
– Works with conventional screens
– Is not restricted to just theatres
• Disadvantages:
– Glasses are more expensive
– Colour filters must be very accurate
Human Computer Interaction Guest Lecture 43
Thursday, March 8th, 2012
44. Displaying 3D Content – (19)
• Last but not least: Autostereoscopic Displays
– View 3D content without glasses
– Currently seen in small gaming systems and small
commercial 3D cameras
• Nintendo 3DS and view screen of the Fujifilm W1
– Currently not available publicly for larger screens
– Common problem with autostereoscopic: Good
for viewing over small screens, but larger screens
tend to make people dizzy or cause discomfort
– Research currently performed to minimize this
Human Computer Interaction Guest Lecture 44
Thursday, March 8th, 2012
45. Displaying 3D Content – (20)
• Principle: Uses either lenticular sheets or
parallax barrier sheets
– Impose the left and right images
onto narrow alternating strips
– Half the columns show the
corresponding columns in the left
image, and other half show the
corresponding right image cols.
– In the figure, they’re represented
as green and pink respectively
Human Computer Interaction Guest Lecture 45
Thursday, March 8th, 2012
46. Displaying 3D Content – (21)
– After we use a screen that either
blocks every other strip
Parallax Barrier
– Or can use lenses of same size
as the strips so that we can bend
the left and right strips and make
it appear to fill the entire image
– Either of these will allow the left
and right images to be directed
to the correct eye
– You just need to stand in the right spot!
Human Computer Interaction Guest Lecture 46
Thursday, March 8th, 2012
47. Displaying 3D Content – (22)
• This can work with multi-view systems too
– The technology can be modified to
display a different viewpoint of the
scene
• Remember multi-view stereo rigs?
– When you stand in a different
position, you will get a different
perspective of the scene
• Just like what would happen in real-life!
– Achieve this directing the view of a particular
perspective to the right pairs of strips / lenses
47
48. Displaying 3D Content – (23)
• Current advocates for autostereoscopic tech.
– Sharp in 2004 designed their first
autostereoscopic LCD monitor
in 2004 Discontinued in 2007
– Similar Philips WOWvx series
• Discontinued in 2009
– Hitachi Designed autostereoscopic
mobile phone in 2009
– Nintendo 3DS Uses parallax barrier
– Fujifilm W1 Viewscreen
Uses lenticular sheets
Human Computer Interaction Guest Lecture 48
Thursday, March 8th, 2012
49. Displaying 3D Content – (24)
• Advantages:
– Glass-free: No maintenance req’d on equipment
– Ideal for delivering to a large group of people
• Co-ordination is required for glass-based technology
– Proven good for small screens / mobile phones
• Disadvantages:
– Larger screens still experimental and expensive
– Larger screens require you to stand far back to
appreciate 3D content
Human Computer Interaction Guest Lecture 49
Thursday, March 8th, 2012
50. Applications
• What can 3D be used for?
– Entertainment and Gaming (obviously!)
– Real-time 3D Video Teleconferencing
– Interactive Medical Surgery
– Interactive Training Sessions
– Virtual Model Exploration
– Robot Navigation
– Fine Art Appreciation
Human Computer Interaction Guest Lecture 50
Thursday, March 8th, 2012
51. Conclusions
• This presentation gave a basic overview of how
3D is made, and how we display 3D
• This presentation is not exhaustive!
– Many other methods to generate 3D material
• Much research performed in this area
– Several technical conferences in 3D: IEEE 3DTVCON,
IEEE 3DIM, SPIE Electronic Imaging
– Research group in Europe researching on
standardizing 3D to mobile phones:
http://sp.cs.tut.fi/mobile3dtv/
Human Computer Interaction Guest Lecture 51
Thursday, March 8th, 2012
52. Thank You!
Questions?
Human Computer Interaction Guest Lecture 52
Thursday, March 8th, 2012