Recent advances in both computational photography and displays have given rise to a new generation of computational devices. Computational cameras and displays provide a visual experience that goes beyond the capabilities of traditional systems by adding computational power to optics, lights, and sensors. These devices are breaking new ground in the consumer market, including lightfield cameras that redefine our understanding of pictures (Lytro), displays for visualizing 3D/4D content without special eyewear (Nintendo 3DS), motion-sensing devices that use light coded in space or time to detect motion and position (Kinect, Leap Motion), and a movement toward ubiquitous computing with wearable cameras and displays (Google Glass). This short (1.5 hour) course serves as an introduction to the key ideas and an overview of the latest work in computational cameras, displays, and light transport.

1. Gordon Wetzstein
http://web.media.mit.edu/~gordonw
MIT Media Lab / Stanford University
Computational Cameras and Displays
Matthew O’Toole
www.dgp.toronto.edu/~motoole
University of Toronto
www.dgp.toronto.edu/~motoole/computationalcamerasanddisplays.html

2. evolution of digital imaging
2000 20142002 2004 2006 2008 2010 2012

3. evolution of digital imaging
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels

4. evolution of digital imaging
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels
EOS 10D
6.3 MPs
EOS D60
6.3 MPs
EOS 20D
8.2 MPs
EOS 30D
8.2 MPs
EOS 40D
10.1 MPs
EOS 50D
15.1 MPs
EOS 60D
17.9 MPs
Canon EOS 70D
20.2 megapixels
0
10
20
30
resolution
(megapixels)
2000 2014

5. evolution of digital imaging
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels
EOS 10D
6.3 MPs
EOS D60
6.3 MPs
EOS 20D
8.2 MPs
EOS 30D
8.2 MPs
EOS 40D
10.1 MPs
EOS 50D
15.1 MPs
EOS 60D
17.9 MPs
Canon EOS 70D
20.2 megapixels
0
10
20
30
resolution
(megapixels)
0
20
40
60
quantum
efficiency(%)
2000 2014
2000 2014
measurements by DxOMark.com

6. evolution of digital imaging
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels
EOS 10D
6.3 MPs
EOS D60
6.3 MPs
EOS 20D
8.2 MPs
EOS 30D
8.2 MPs
EOS 40D
10.1 MPs
EOS 50D
15.1 MPs
EOS 60D
17.9 MPs
Canon EOS 70D
20.2 megapixels
0
10
20
30
resolution
(megapixels)
0
20
40
60
quantum
efficiency(%)
2000 2014
2000 2014
measurements by DxOMark.com
nearing end of the megapixel race for cameras and displays…

7. evolution of digital imaging
“Retina” display: pixels imperceptible to the eye
nearing end of the megapixel race for cameras and displays
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels
EOS 10D
6.3 MPs
EOS D60
6.3 MPs
EOS 20D
8.2 MPs
EOS 30D
8.2 MPs
EOS 40D
10.1 MPs
EOS 50D
15.1 MPs
EOS 60D
17.9 MPs
Canon EOS 70D
20.2 megapixels

8. evolution of digital imaging
“Retina” display: pixels imperceptible to the eye
nearing end of the megapixel race for cameras and displays
2000 20142002 2004 2006 2008 2010 2012
Canon EOS D30
3.1 megapixels
EOS 10D
6.3 MPs
EOS D60
6.3 MPs
EOS 20D
8.2 MPs
EOS 30D
8.2 MPs
EOS 40D
10.1 MPs
EOS 50D
15.1 MPs
EOS 60D
17.9 MPs
Canon EOS 70D
20.2 megapixels
compressive displays
[Lanman et al. 2011]light field imaging [Ng et al. 05]

9. cameras and displays revolution
cameras
Lytro
Pelican Imaging
[Venkataraman et al. SIG Asia 2013]Raytrix

10. cameras and displays revolution
Nintendo 3DSGoogle Glass
displayscameras
Lytro
Pelican Imaging
[Venkataraman et al. SIG Asia 2013]Raytrix

11. cameras and displays revolution
Microsoft Kinect for Xbox OneLeap Motion
Nintendo 3DSGoogle Glass
displays
light
transportcameras
Lytro
Pelican Imaging
[Venkataraman et al. SIG Asia 2013]Raytrix

12. cameras and displays revolution
Microsoft Kinect for Xbox OneLeap Motion
Nintendo 3DSGoogle Glass
displays
light
transport
use computation to reform digital imaging (display, capture, …)
cameras
Lytro
Pelican Imaging
[Venkataraman et al. SIG Asia 2013]Raytrix

13. three domains of computation
computational
cameras

14. three domains of computation
computational
cameras
computational
displays

15. three domains of computation
computational
cameras
computational
displays
physical
world
computational
light transport

16. speakers
Gordon Wetzstein
• assistant professor at Stanford University (EE)
• received an award for his dissertation titled
“Computational Plenoptic Image Acquisition and
Display”
• spent last 3 years as a Postdoc at MIT Media Lab’s
camera culture group

17. speakers
Matthew O’Toole
Gordon Wetzstein
• Ph.D. student at the University of Toronto
supervised by Kyros Kutulakos
• focus on light transport analysis
• assistant professor at Stanford University (EE)
• received an award for his dissertation titled
“Computational Plenoptic Image Acquisition and
Display”
• spent last 3 years as a Postdoc at MIT Media Lab’s
camera culture group

18. overview
10 min | computational cameras (Wetzstein)

19. overview
30 min | computational displays (Wetzstein)
10 min | computational cameras (Wetzstein)

20. overview
35 min | computational light transport (O’Toole)
30 min | computational displays (Wetzstein)
10 min | computational cameras (Wetzstein)

21. overview
35 min | computational light transport (O’Toole)
30 min | computational displays (Wetzstein)
10 min | computational cameras (Wetzstein)
10 min | summary and Q&A (both)

22. takeaways from the course
• review current research topics and recent work
in computational sensing and displays
• introduce core concepts, including plenoptic cameras, compressive
displays and transport matrices
• provide a sense as to possible directions for future research

23. Computational Cameras
Part 1: