This is a ppt file for study meetings held in our lab, describing chapter 10 computational photography in the book of Szeliski's "Computer Vision: Algorithms and Applications." Aizawa-Yamasaki Lab. at The Univ. of Tokyo http://www.hal.t.u-tokyo.ac.jp/

1. Computer Vision: Algorithms and Applications
Chapter 10
Computational photography
Kazuhiro Sato

2. What is Computational Photography?

3. To enhance or extend the
capabilities of digital photography
enhance
extend

4. Contents –computational
photography-
1. Photometric calibration
2. High dynamic range imaging
3. Super-resolution and blur removal
4. Image matting and compositing
5. Texture analysis and synthesis

5. 1. Photometric calibration

7. imaging pipeline

8. First, we characterize
• the mapping functions
• the amounts of noise

9. The mapping functions consists of
• the radiometric response function
• vignetting
• the point spread function

10. 1.1 Radiometric Response
Function
often NOT linear
sensor plane
optics

11. How can we determine the function?
• calibration chart
• polynomial approximation
• least squares (explained later)

12. 1.2 Noise Level Estimation
[Liu et al. ‟08]
1. Segment the input image
2. Fit a linear function
3. Plot the standard deviation
pixel value
4. Fit a lower envelope estimated noise level

13. 1.3 Vignetting
・ Radial Gradient Symmetry [Zheng et al. ‟08]
Histogram (asymmetry)
0

16. 1.4 Optical Blur Estimation
・ PSF Estimation [Joshi et al. ‟08]

17. Solving a Bayesian framework
using a maximum a posteriori (MAP)

18. ・ Recovering the PSF without calibration
1. Fit step edges to the elongated ones
2. Apply the least squares for every surrounded pixel

19. 2. High Dynamic Range Imaging

20. Dynamic Range
0.01 1 100 10000
Luminance
[cd/m2]
moonlight indoor lighting sunlight
Human eye
CCD camera

21. The
natural
world is
too bright
to be
captured!

22. Creating a properly exposed
photo (High dynamic range
imaging)
different exposures
create
properly exposed photo

23. How to create such a photo?
1. Estimate the radiometric response function
2. Estimate a radiance map
3. Tone map the resulting HDR image into a 8-bit one

24. 1. Estimate the radiometric response function
[Debevec et al. ‟97]

26. 2. Estimate a radiance map [Mitsunaga et al. ‟99]
different
exposures
Merging the input images into a composite radiance
map.

27. merge
different
exposures radiance map (grayscale)

28. 3. Tone map the resulting HDR image into a 8-bit one
HDR image
8 bits / pixel
8-bit image

29. 2.1 Tone mapping
・ Global tone mapping using a transfer curve
[Larson et al. ‟97]
This global approach fails to preserve details in
regions with widely varying exposures.
gamma applied to gamma applied to
input HDR image
each channel luminance only

30. ・ Local tone mapping using bilateral filter
[Durand et al. ‟02]

31. This approach doesn‟t create visible halos around the
edges.
result with result with
low-pass filtering bilateral filtering
(visible halos) (no halos)

32. ・ Gradient domain tone mapping [Fattal et al. ‟02]
The new luminance is combined with the original color
image.

33. Attenuation map Tone-mapped result

34. 2.2 Flash photography [Petschnigg et al. 04]
Combine flash and non-flash images to achieve
better photos

35. Joint bilateral filter‟s kernel
domain kernel range kernel

36. 3. Super-resolution and blur removal

37. 3.1 Color image demosaicing
interpolate
Bayer RGB pattern
Full-color RGB
in a camera sensor

38. ・ Bayesian demosaicing with a two-color prior
[Bennett et al. ‟06]
Original Bilinear [Bennett et
al. ‟06]

39. Two-color model [Bennett et al. ‟06]

40. 4. Image matting and compositing

41. Image “matting” and “compositing”?
matting compositing
Input Alpha-matting Output

42. What is the problem in matting and
compositing?

43. Failed example of matting
Input
Alpha matte Composite Inset

44. 4.1 Blue screen matting [Smith et al. „96]
solve

45. ・ Two-screen matting[Smith et al. ‟96]
solve

46. 4.2 Natural image matting
Input Hand-drawn trimap Alpha map
foreground composite
[Chuang et al. ‟01]

47. ・ Knockout[Berman et al. ‟00]

48. ・ Bayesian approach [Chuang et al. ‟01]

50. ・ Comparison of natural image mattings

51. 4.3 Optimization-based matting
・ Border matting [Rother et al. ‟04]
1. Get a trimap by hard segmentation

52. data term smoothness term

53. ・ Color line (closed-form) matting[Levin et al. ‟08]
data term regularization term

55. 4.4 Smoke, shadow, and flash
matting
・ Smoke matting [Chuang et al. ‟02]
removing the
estimated alpha insertion of new
input frame foreground
matte objects
object

56. ・ Shadow matting [Chuang et al. ‟03]

57. 4.5 Video matting
[Chuang et al. ‟02]

58. 5. Texture analysis and synthesis

59. Texture synthesis is …
synthesize
Small patch
Similar-looking larger patch

60. Texture synthesis using non-parametric sampling
[Efros et al. ‟99]
Texture synthesis using non-parametric sampling
[Efros et al. ‟01]

61. 5.1 hole filling and inpainting
inpaint
Original Inpainted result

62. ・ Exemplar-based inpainting [Criminisi et al. ‟04]
confidence term data term

63. 3. Update confidence values

64. onion peel
[Criminisi et al. ‟04]
removed
original
region
Remain the gradient along
the region boundary

65. 5.2 Non-photorealistic rendering
Non-photorealistic ・ Texture transfer
rendering using
texture ・ Image
synthesis analogies

66. ・ Texture [Efros et al. ‟01]
transfer
texture transfer
input output
texture transfer
input output

68. 2. Find the minimum error boundary cut

69. ・ Image [Hertzmann et al. ‟01]
analogies
synthesize this image
?

70. unfiltered example NPR-filtered example
target result

71. unfiltered example filtered example
target result

72. Thank you!