O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Single Image Super-Resolution from Transformed Self-Exemplars (CVPR 2015)

Self-similarity based super-resolution (SR) algorithms are able to produce visually pleasing results without extensive training on external databases. Such algorithms exploit the statistical prior that patches in a natural image tend to recur within and across scales of the same image. However, the internal dictionary obtained from the given image may not always be sufficiently expressive to cover the textural appearance variations in the scene. In this paper, we extend self-similarity based SR to overcome this drawback. We expand the internal patch search space by allowing geometric variations. We do so by explicitly localizing planes in the scene and using the detected perspective geometry to guide the patch search process. We also incorporate additional affine transformations to accommodate local shape variations. We propose a compositional model to simultaneously handle both types of transformations. We extensively evaluate the performance in both urban and natural scenes. Even without using any external training databases, we achieve significantly superior results on urban scenes, while maintaining comparable performance on natural scenes as other state-of-the-art SR algorithms.

http://bit.ly/selfexemplarsr

  • Entre para ver os comentários

Single Image Super-Resolution from Transformed Self-Exemplars (CVPR 2015)

  1. 1. Single Image Super-Resolution from Transformed Self-Exemplars Jia-Bin Huang Narendra AhujaAbhishek Singh
  2. 2. Single Image Super-Resolution • Recovering high-resolution image from low-resolution one Spatial frequency Amplitude Super-Resolution Sharpening
  3. 3. Multi-image vs. Single-image Multi-image Source: [Park et al. SPM 2003] Single-image Source: [Freeman et al. CG&A 2002]
  4. 4. External Example-based Super-Resolution Learning to map from low-res to high-res patches • Nearest neighbor [Freeman et al. CG&A 02] • Neighborhood embedding [Chang et a. CVPR 04] • Sparse representation [Yang et al. TIP 10] • Kernel ridge regression [Kim and Kwon PAMI 10] • Locally-linear regression [Yang and Yang ICCV 13] [Timofte et al. ACCV 14] • Convolutional neural network [Dong et al. ECCV 14] • Random forest [Schulter et al. CVPR 15] External dictionary
  5. 5. Internal Example-based Super-Resolution Low-res and high-res example pairs from patch recurrence across scale • Non-local means with self-examples [Ebrahimi and Vrscay ICIRA 2007] • Unified classical and example SR [Glasner et al. ICCV 2009] • Local self-similarity [Freedman and Fattal TOG 2011] • In-place regression [Yang et al. ICCV 2013] • Nonparametric blind SR [Michaeli and Irani ICCV 2013] • SR for noisy images [Singh et al. CVPR 2014] • Sub-band self-similarity [Singh et al. ACCV 2014] Internal dictionary
  6. 6. Motivation • Internal dictionary • More “relevant” patches • Limited number of examples • High-res patches are often available in the transformed domain Symmetry Surface orientation Perspective distortion
  7. 7. Super-Resolution from Transformed Self-Exemplars
  8. 8. LR input image Matching error LR patch HR patch Translation Perspective Ground truth LR/HR patch
  9. 9. Translation Ground truth LR/HR patch Affine transform LR input image Matching error LR patch HR patch
  10. 10. Input low-res image All-frequency band low-frequency band Super-Resolution Scheme Multi-scale version of [Freedman and Fattal TOG 2011]
  11. 11. Input low-res image LR/HR example pairs Super-Resolution Scheme Multi-scale version of [Freedman and Fattal TOG 2011] low-frequency bandAll-frequency band
  12. 12. Input low-res image low-frequency bandAll-frequency band
  13. 13. Input low-res image low-frequency bandAll-frequency band
  14. 14. Super-Resolution as Nearest Neighbor Field Estimation Appearance cost Plane compatibility Scale cost [Huang et al. SIGGRAPH 2014] Scale
  15. 15. Search Patch Transformation • Generalized PatchMatch [Barnes et al. ECCV 2010] • Randomization • Spatial propagation • Backward compatible when planar structures were not detected Perspective Similarity Affine [Huang et al. SIGGRAPH 2014]
  16. 16. Results
  17. 17. Datasets – BSD 100 and Urban 100 Berkeley segmentation dataset (100 test images) Urban image dataset from Flickr (100 test images)
  18. 18. Dataset – Set5, Set14, and Sun-Hays 80 Set5 Set 14 Sun-Hays 80 [Sun and Hays ICCP 12]
  19. 19. Ground-truth HR SRCNN [Dong et al. ECCV 14] Glasner [Glasner et al. ICCV 2009] Our result SR Factor 4x Bicubic A+ [Timofte et al. ACCV 14]
  20. 20. SR Factor 4x Ground-truth HR SRCNN [Dong et al. ECCV 14] Glasner [Glasner et al. ICCV 2009] Our result Bicubic A+ [Timofte et al. ACCV 14]
  21. 21. SR Factor 4x Ground-truth HR SRCNN [Dong et al. ECCV 14] Glasner [Glasner et al. ICCV 2009] Our result Bicubic A+ [Timofte et al. ACCV 14]
  22. 22. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR Sub-band [Singh et al. ACCV 2014]
  23. 23. Ground-truth SRCNN [Dong et al. ECCV 14] Glasner [Glasner et al. ICCV 2009] Our result
  24. 24. Ground-truth HR SRCNN [Dong et al. ECCV 14] Glasner [Glasner et al. ICCV 2009] Our result
  25. 25. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR Sub-band [Singh et al. ACCV 2014]
  26. 26. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR Our resultSub-band [Singh et al. ACCV 2014]
  27. 27. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR Glasner [Glasner et al. ICCV 2009]
  28. 28. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR ScSR [Yang et al. TIP 10]
  29. 29. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR ScSR [Yang et al. TIP 10]
  30. 30. Bicubic SRCNN [Dong et al. ECCV 14] A+ [Timofte et al. ACCV 14] Our result Ground-truth HR ScSR [Yang et al. TIP 10]
  31. 31. BSD 100 Dataset – SR factor 4x
  32. 32. Quantitative Results – Urban 100 dataset Scale Bicubic ScSR Kim and Kwon Sub-band Glasner SRCNN A+ Ours 2x - PSNR 26.66 28.26 28.74 28.34 27.85 28.65 28.87 29.38 4x - PSNR 23.14 24.02 24.20 24.19 23.58 24.14 24.34 24.82 2x - SSIM 0.8408 0.8828 0.8940 0.8820 0.8709 0.8909 0.8957 0.9032 4x - SSIM 0.6573 0.7024 0.7104 0.7115 0.6736 0.7047 0.7195 0.7386 ~ 0.5 dB averaged PSNR improvement over the state-of-the-art method
  33. 33. Quantitative Results – BSD 100 dataset On par of the state-of-the-art method Scale Bicubic ScSR Kim Sub-band Glasner SRCNN A+ Ours 2x - PSNR 29.55 30.77 31.11 30.73 30.28 31.11 31.22 31.18 3x - PSNR 27.20 27.72 28.17 27.88 27.06 28.20 28.30 28.30 4x - PSNR 25.96 26.61 26.71 26.60 26.17 26.70 26.82 26.85 2x - SSIM 0.8425 0.8744 0.8840 0.8774 0.8621 0.8835 0.8862 0.8855 3x - SSIM 0.7382 0.7647 0.7788 0.7714 0.7368 0.7794 0.7836 0.7843 4x - SSIM 0.6672 0.6983 0.7027 0.7021 0.6747 0.7018 0.7089 0.7108
  34. 34. Ground truth HR image Input LR image 128 x 96
  35. 35. Bicubic SR Factor 8x
  36. 36. Internet-scale scene matching [Sun and Hays ICCP 12] SR Factor 8x #Training images 6.3 millions
  37. 37. SRCNN [Dong et al. ECCV 14] SR Factor 8x #Training images 395,909 from ImageNet
  38. 38. Our result SR Factor 8x #Training image 1 LR input
  39. 39. Our result: coarse-to-fine super-resolution
  40. 40. Ground truth HR image Input LR image 128 x 96
  41. 41. Bicubic SR Factor 8x
  42. 42. Sparse coding [Yang et al. TIP 10] SR Factor 8x
  43. 43. SRCNN [Dong et al. ECCV 14] SR Factor 8x
  44. 44. Our result SR Factor 8x
  45. 45. Our result: coarse-to-fine super-resolution
  46. 46. Ground truth HR image Input LR image 128 x 96
  47. 47. Bicubic SR Factor 8x
  48. 48. SR Factor 8xInternet-scale scene matching [Sun and Hays ICCP 12]
  49. 49. SR Factor 8xSRCNN [Dong et al. ECCV 14]
  50. 50. Our result SR Factor 8x
  51. 51. Our result: coarse-to-fine super-resolution
  52. 52. Bicubic SR Factor 8x
  53. 53. SRCNN [Dong ECCV 2014] SR Factor 8x
  54. 54. Ours SR Factor 8x
  55. 55. Bicubic SR Factor 8x
  56. 56. SRCNN [Dong ECCV 2014] SR Factor 8x
  57. 57. Ours SR Factor 8x
  58. 58. Low-Res TI-DTV [Fernandez-Granda and Candes ICCV 2013] Ours SR Factor 4x
  59. 59. Low-Res TI-DTV [Fernandez-Granda and Candes ICCV 2013] Ours SR Factor 4x
  60. 60. Limitations – Blur Kernel Model • Suffer from blur kernel mismatch • Blind SR to estimate kernel [Michaeli and Irani ICCV 2013] [Efrat et al. ICCV 2013] • With ground truth kernel, we can get significantly improvement • External example-based method would need to retrain the model
  61. 61. Limitations • Slow computation time • On average, 40 seconds for super-resolving 2x on an image in BSD 100 dataset on a 2.8Ghz PC, 12G RAM PC SRF 4x Ground truth HR Our result A+ [Timofte et al. ACCV 14]SRCNN [Dong et al. ECCV 14]
  62. 62. Conclusions • Super-resolution based on transformed self-exemplars • No training data, no feature extraction, no complicated learning algorithms • Works particularly well on urban scenes • On par with state-of-the-art on natural scenes Code and data available: http://bit.ly/selfexemplarsr See us on poster #82
  63. 63. Single Image Super-Resolution from Transformed Self-Exemplars http://bit.ly/selfexemplarsr

×