We propose a flexible light field camera architecture that is at the convergence of optics, sensor electronics, and applied mathematics. Through the co-design of a sensor that comprises tailored, Angle Sensitive Pixels and advanced reconstruction algorithms, we show that—contrary to light field cameras today—our system can use the same measurements captured in a single sensor image to recover either a high-resolution 2D image, a low-resolution 4D light field using fast, linear processing, or a high-resolution light field using sparsity-constrained optimization.
The Mariana Trench remarkable geological features on Earth.pptx
>A Switchable Light Field Camera Architecture with Angle SEnsitive Pixels and Dictionary-based Sparse Coding
1. A Switchable Light Field Camera
Using Angle Sensitive Pixels and
Dictionary-based Space Coding
Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2*
Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1
1MIT Media Lab 2Cornell University
*Shared first-authorship
4. Flexible Camera Vision
Visual Complexity
ComputeTime
High Resolution 2D Low Resolution Light Field
High Resolution Light Field
Prototype is grayscale only!
5. Computational Photography Touches...
Optics Algorithms Sensors
Lytro.com Dappled Photography, Veeraraghavan, et al.
?An angle-sensitive CMOS imager for single-sensor
3D photography, Wang et al.
7. Light Field Cameras
Lens Based Mask Based
barrier
lenslets
Veeraraghavan et al. 2007Ives, 1905Lippmann 1908Lytro 2014
Direct angle-space tradeoff
Limits Resolution
Direct angle-space tradeoff
Limits light transmission
Limits Light Field Resolution
More Flexible
8. Camera Arrays
Light Field Cameras
Sequential Acquisition
Wilburn et al. 2002,2005
Levoy and Hanrahan 1996
Liang et al. 2008
20. 2D ASP Tile
𝝆 𝜶,𝜷,𝜸
(𝛉) =
𝟏
𝟐
+
𝐦
𝟐
𝐜𝐨𝐬 𝛃 𝐜𝐨𝐬 𝜸 𝛉 𝒙 + 𝛃 𝐬𝐢𝐧 𝛄 𝛉 𝒚 + 𝛂
Physical Layout Impulse Response (2D)
Low
𝜷 𝟏𝟐
Med
𝜷 18
High
𝜷 24
Low
𝜷 𝟏𝟐
Med
𝜷 18
High
𝜷 24
𝜸 𝟎 𝜸 𝟎
𝜸 𝟒𝟓 𝜸 𝟒𝟓
𝜶 𝟎 𝜶 𝟎
𝜶 𝟗𝟎 𝜶 𝟗𝟎
Note: This prototype was made using a functionally similar variant of the ASP that
uses a stack of metal gratings instead of phase gratings
27. Fast/Linear Reconstruction
ASP responses are approximately orthonormal wavelents
Σ is used as a preconditioner for inverting capture equation
• Model the image capture process:
ASP Sensor Tile Impulse Response (2D)
31. Training light field
= is sparse
Sample 1,000,000 random 4D patches from training light fields
i i i
for all iCoefficient vectorDictionary
s.t.
Dictionary Learning
Courtesy Marwah et al.
32. Light Field “Atoms” in Dictionary
5,000 atoms, each 9x9 pixels and 5x5 views
Light fields can be represented by only a few of these atoms
Courtesy Marwah et al.
43. Conclusion
• We built a custom, ASP sensor
• New sensors, new opportunities
– Shoot now compute later!
– [Computation in] [quality out]
• Seeking new problems
– e.g. Illumination Multiplexing, Time of Flight
44. Support provided by:
NSF IIS-1218411 NSF Graduate Student Fellowship (Suren) MIT Media Lab Consortia Funding
NSF IIS-1116452 NSERC Postdoctural Fellowship (Gordon)
Matthew Hirsch1* Sriram Sivaramakrishnan2* Suren Jayasuriya2*
Albert Wang2 Alyosha Molnar2 Ramesh Raskar1 Gordon Wetzstein1
1MIT Media Lab 2Cornell University
A Switchable Light Field Camera