1. Raskar, Camera Culture, MIT Media Lab
Computational Ph t
C t ti l Photography:
h
Epsilon to Coded Imaging
p g g
Camera Culture
Ramesh Raskar
Camera Culture
C C lt
Associate Professor, MIT Media Lab http://raskar.info

3. Tools
for
Visual
Computing
Shado Refracti Reflecti
w ve ve
Fernald, Science [Sept 2006]

4. How can we create an entirely
new class of imaging platforms
that have an understanding of the world that far exceeds
human ability
and produce meaningful abstractions that are well
within h
ithi human comprehensibility ?
h ibilit
Ramesh Raskar http://raskar.info

5. Raskar 2006
Computational Illumination Augmented Reality
Mitsubishi Electric Research Laboratories Spatial
Planar Non planar
Non-planar Curved Objects Pocket Proj
Pocket-Proj
1998 1997 2002 2002
Single
Projector
Use
r:T j
?
Projector
1998 1998 2002 1999 2003
Multiple
Projectors
Computational Camera and Photography

6. Motion Blurred Photo

8. Short
Sh t Traditional
T diti l MURA
Exposure Shutter
Captured
Single
Photo
Ph t
Deblurred
Result
Banding Artifacts and
Dark
some spatial frequencies
and noisy
d i
are lost

9. Blurring == Convolution
Fourier
Transform
Sharp
Sh Blurred
Bl d
Photo PSF == Sinc Function Photo
ω
Traditional Camera: Shutter is OPEN: Box Filter

10. Fourier
Transform
Sharp
Sh Blurred
Bl d
Photo PSF == Broadband Function Photo
Preserves High Spatial
Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED

11. Flutter Shutter Camera
Raskar, Agrawal, Tumblin [Siggraph2006]
LCD opacity switched
in coded sequence

12. Traditio Coded
nal Exposu
re
Deblurred Deblurred
Image
I Image
I
Image of
Static
Object

14. Coded Exposure Coded Aperture
Temporal 1-D broadband code: Spatial 2-D broadband mask:
Motion Deblurring Focus Deblurring

15. Coded Aperture Camera
The aperture of a 100 mm lens is modified
Insert a coded mask with chosen binary pattern
Rest of the camera is unmodified

16. LED
In Focus Photo

17. Out of Focus Photo: Open Aperture

18. Out of Focus Photo: Coded Aperture

19. Captured Blurred
Photo

20. Refocused on
Person

21. Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography
– Low-level Vision: Pixels
– Multiphotos by bracketing (HDR, panorama)
– ‘Ultimate camera’
2. Coded Photography
– Mid-Level Cues:
• Regions, Edges, Motion, Direct/global
g , g , , g
– Single/few snapshot
• Reversible encoding of data
– Additional sensors/optics/illum
3. Essence Photography
– Not mimic human eye
– Beyond single view/illum
– ‘New artform’

22. Raskar, Camera Culture, MIT Media Lab
• Ramesh Raskar and
Jack T bli
J k Tumblin
• Book Publishers: A K Peters

23. Less is More
Blocking Light == More Information
Coding in Time Coding in Space

24. Larval Trematode Worm Coded Aperture Camera

25. Shielding Light …
g g
Larval Trematode Worm Turbellarian Worm

26. Mask Sensor
Mask
?
Sensor
Mask
Full Resolution Digital 4D Light Field from
Refocusing: 2D Photo:
Coded Aperture Camera Heterodyne Light Field
d h ld
Camera

27. Light Field Inside a Camera

28. Light Field Inside a Camera
Lenslet-
Lenslet-based Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]

29. Stanford Plenoptic Camera [Ng et al 2005]
Contax medium format camera Kodak 16-megapixel sensor
Adaptive Optics microlens array 125μ square-sided microlenses
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

30. Digital Refocusing
g g
[Ng et al 2005]
Can we achieve this with a Mask alone?

31. Mask based Light Field Camera
Sensor
Mask
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

32. How to Capture
4D Light Field with
g
2D Sensor ?
What h ld be th
Wh t should b the
pattern of the mask ?

33. Cosine Mask Used
Mask Tile
1/f0

34. Captured 2D Photo
Encoding due to
Mask

35. Sensor Slice captures entire Light Field
fθ
fθ0
fx
fx0
Modulation
M d l i
Function Modulated Light Field

36. Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D
FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D
4D IFFT 200*200*9*9
planes
4D Light Field
200*200*9*9

37. Captured
Full resolution 2D image
2D Photo
of Focused Scene Parts
divide
Image of White Lambertian
Plane

38. Wavefront Sensing in Any Wavelength !
Sensor
Mask
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

39. Lens Flare Reduction/Enhancement using
4D Ray Sampling
Glare Captured Glare
Enhanced Reduced

40. Glare = low frequency noise in 2D
•But is high frequency noise in 4D
•Remove via simple outlier rejection
Sensor
i
j
u x

41. Rays = Waves for Propagation and Interface
Fresnel propagation Chirp (Lens) Fourier transform Fractional Fourier transform
x1 x2 x3 x4
x0
b
¡ a x0
u1 u2 u3
b u4
-ax
b 0
x0 x1 x0 x2 x3 -bx
a 0 x4
x0
- a
-x
a
0
I
x4

42. Imaging via volume hologram (Depth-specific Imaging)
KVH ( x =0, u =θ /λ; x , u )
4 4 s 3 3
-20
u4
1
u3
-15
0.8
-10
0.6
-5
L
u3 [mm-1]
0 0.4 x3 x4
5
0.2
10
0
15
20 -0.2
-0.4 -0.2 0 0.2 0.4
x3 [mm]
ZZ ½ µ ¶¾
0 0 µs
dx 0 dx 0 e¡ i 2¼( u 4 x 4 ¡ u 3 x 3 )
K V H (x 4 ; u4 ; x 3 ; u3 ) =3 4
0 0
exp ¡ i 2¼ zf (u3 + u4 ) ¡ u3 + u4 ¡
¸
¸
½ µ 0 0
¶µ 0
¶¾ ½ µ 0 0
¶µ ¶¾
u3 + u4 u4 µs u3 + u4 u0
4 µs
£ sinc L ¸ ¡ u3 + u4 + u4 + ¡ sinc L ¸ ¡ u3 + u4 ¡ u4 ¡ ¡
2 2 ¸ 2 2 ¸
½ ¾ ½ ¾
¼ zf 2 L
Derivation: h(x 2 ; x 1 ) = exp ¡ i 2
(x 1 + x 2 ¡ f µs) sinc (x 1 + x 2 ) (x 2 ¡ f µs)
¸ f ¸f2
K V H I (x 2 ; u2 ; x 1 ; u1 ) Parameters:
¸=05¹
0.5 ¹m
µs= 30°
K V H (x 4 ; u4 ; x 3 ; u3 ) L = 1 mm
zf = 50 mm

43. Camera Culture Group
Raskar, Camera Culture, MIT Media Lab Ramesh Raskar http://raskar.info
Computational Photography
1. Epsilon Photography Mask
Sensor
– Low-level Vision: Pixels
– Multiphotos by bracketing (HDR, panorama)
– ‘Ultimate camera’
2. Coded Photography
– Mid Level
Mid-Level Cues:
• Regions, Edges, Motion, Direct/global
• Coded Exposure
– Flutter Shutter Motion Deblurring
• Coded Aperture
– Defocus
• Optical Heterodyning -20 1
– Lightfield or Wavefront sensing -15
-10
0.8
• Coded Glare
0.6
-5
u3 [mm-1]
0 0.4
• p y
6D Display 5
10
0.2
0
• Femto-second Imaging
15
2D 2D 2D
1 20 -0.2
2 -0.4 -0.2 0 0.2 0.4
x3 [mm]
• Rays = Waves 1 1

44. How can we create an entirely
new class of imaging platforms
that have an understanding of the world that far exceeds
human ability
and produce meaningful abstractions that are well
within h
ithi human comprehensibility ?
h ibilit
Ramesh Raskar http://raskar.info