O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Image Completion using Planar Structure Guidance (SIGGRAPH 2014)

We propose a method for automatically guiding patch-based image completion using mid-level structural cues. Our method first estimates planar projection parameters, softly segments the known region into planes, and discovers translational regularity within these planes. This information is then converted into soft constraints for the low-level completion algorithm by defining prior probabilities for patch offsets and transformations. Our method handles multiple planes, and in the absence of any detected planes falls back to a baseline fronto-parallel image completion algorithm. We validate our technique through extensive comparisons with state-of-the-art algorithms on a variety of scenes.

Project page: https://sites.google.com/site/jbhuang0604/publications/struct_completion

  • Entre para ver os comentários

Image Completion using Planar Structure Guidance (SIGGRAPH 2014)

  1. 1. Image Completion using Planar Structure Guidance Jia-Bin Huang Johannes KopfNarendra AhujaSing Bing Kang
  2. 2. The image completion problem Credit: ©Flickr user remonrijper
  3. 3. The image completion problem Credit: ©Flickr user remonrijper
  4. 4. The image completion problem Credit: ©Flickr user remonrijper
  5. 5. Large Hole Small Hole Non-stationaryStationary
  6. 6. Large Hole Small Hole Non-stationary Texture Synthesis Stationary
  7. 7. Texture Synthesis Large Hole Small Hole Non-stationary Image Inpainting Stationary
  8. 8. Texture Synthesis Large Hole Small Hole Non-stationary Image Completion Stationary
  9. 9. Input Image Credit: ©Flickr user remonrijper
  10. 10. Region to fill
  11. 11. Photoshop Content Aware Fill [Wexler et al. 2007] [Barnes et al. 2009]
  12. 12. Statistics of Patch Offsets [He and Sun ECCV 2012] Translational patches are not sufficient!
  13. 13. Searching patch transformation space is HARD! Image Melding [Darabi et al. 2012]
  14. 14. Our Result We use mid-level information
  15. 15. • Based on non-parametric framework of [Wexler et al. 2007]
  16. 16. • Based on non-parametric framework of [Wexler et al. 2007] • The basic form
  17. 17. • Based on non-parametric framework of [Wexler et al. 2007] • The basic form • Patch matching cost • PatchMatch [Barnes et al. 2009] for efficient nearest neighbor field search.
  18. 18. • Low-level processing is not sufficient
  19. 19. • Low-level processing is not sufficient • Idea: guide the completion using mid-level information!
  20. 20. • Low-level processing is not sufficient • Idea: guide the completion using mid-level information! • Transformation of source patches
  21. 21. • Low-level processing is not sufficient • Idea: guide the completion using mid-level information! • Transformation of source patches • Translational (Photoshop CAF)
  22. 22. • Low-level processing is not sufficient • Idea: guide the completion using mid-level information! • Transformation of source patches • Translational (Photoshop CAF) • Rotation and scale (Image Melding)
  23. 23. • Low-level processing is not sufficient • Idea: guide the completion using mid-level information! • Transformation of source patches • Translational (Photoshop CAF) • Rotation and scale (Image Melding) • Projective transform (Ours)
  24. 24. Mid-Level cues for guidance •Planes: Cues for how to deform patches •Regularity: Cues for where to sample patches
  25. 25. Mid-level cues – planes Credit: ©Flickr user micromegas
  26. 26. Plane detection – vanishing point detection
  27. 27. Plane detection – vanishing point detection
  28. 28. Plane detection – localization Support lines for VPs
  29. 29. Plane detection – localization Support lines for VPs Support line density
  30. 30. Plane detection – localization ⊙ Support lines for VPs Support line density Plane location density
  31. 31. Plane detection – localization ⊙ ⊙ Support lines for VPs Support line density Plane location density
  32. 32. Plane detection – localization ⊙ ⊙ ⊙ Support lines for VPs Support line density Plane location density
  33. 33. Plane detection – posterior probability
  34. 34. Our Result
  35. 35. Photoshop Content Aware Fill Statistics of Patch Offsets Image Melding Our Result
  36. 36. Photoshop Content Aware Fill Statistics of Patch Offsets Image Melding Our Result
  37. 37. Input and Hole Credit: ©Flickr user theenmoy
  38. 38. Our Result (with planar guidance only) Need regularity guidance!
  39. 39. Extracting mid-level cues - regularity detection
  40. 40. Regularity detection – feature detection and matching
  41. 41. Regularity detection – mode detection • For fronto-parallel scenes, similar to [He and Sun ECCV 2012] • Can handle perspectively distorted planes and multiple surfaces
  42. 42. Regularity detection without rectification
  43. 43. Plane and regularityWith planar guidance only
  44. 44. Guided sampling and propagation • Slight modification of PatchMatch algorithm • Plane guided sampling and propagation • Draw a plane index based on posterior probability • Draw random samples using rejection-based sampling
  45. 45. Guided sampling and propagation • Slight modification of PatchMatch algorithm • Plane guided sampling and propagation • Draw a plane index based on posterior probability • Draw random samples using rejection-based sampling • Regularity guided sampling • Randomly draw a offset vector in rectified space
  46. 46. Results
  47. 47. Input and Hole Credit: ©Flickr user danielfoster
  48. 48. Photoshop Content Aware Fill Statistics of Patch Offsets Image Melding Our Result
  49. 49. Input and hole Credit: ©Flickr user the-o
  50. 50. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result [Wexler et al. 2007] [Barnes et al. 2009] [He and Sun 2012] [Darabi et al. 2012]
  51. 51. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result [Wexler et al. 2007] [Barnes et al. 2009] [He and Sun 2012] [Darabi et al. 2012]
  52. 52. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result [Wexler et al. 2007] [Barnes et al. 2009] [He and Sun 2012] [Darabi et al. 2012]
  53. 53. Input and Hole Credit: ©Flickr user chrisschoenbohm
  54. 54. Photoshop Content Aware Fill Statistics of Patch Offsets Image Melding Our Result
  55. 55. Input and Hole Credit: ©Flickr user 99667320@N06
  56. 56. Photoshop Content Aware Fill Statistics of Patch Offsets Image Melding Our Result
  57. 57. Input and hole Credit: ©Flickr user sunshinepictures
  58. 58. Photoshop CAF Statistics of Patch Offsets Image Melding Our Result [Wexler et al. 2007] [Barnes et al. 2009] [He and Sun 2012] [Darabi et al. 2012]
  59. 59. What if there are no structures? • Reduces to baseline image completion algorithms which use translational patches (same as Photoshop content- aware fill)
  60. 60. What if there are no structures? • Reduces to baseline image completion algorithms which use translational patches (same as Photoshop content- aware fill) • Works well with man-made scenes • Backward-compatible with natural scenes • Tested on 25 natural images from [Kopf et al. 2012]
  61. 61. Input and Mask Completion
  62. 62. Input and Mask Completion
  63. 63. Input and Mask Completion
  64. 64. Check out our project website! • http://bit.ly/planarcompletion • Full comparison of 110 images • 85 urban scenes + 25 natural • Code will be released soon!
  65. 65. Failure modes • Difficult to find correct demarcation lines (boundary between regions) • Need higher-level information • Missed, false, or inaccurate detection of plane parameters or regularity • Computer vision algorithms are not perfect
  66. 66. Input Credit: ©Flickr user loungerie
  67. 67. Mask
  68. 68. Our result
  69. 69. Photoshop Priority BPImage Melding Statistics of Patch Offset
  70. 70. Input Credit: ©Flickr user addictive_picasso
  71. 71. Mask
  72. 72. Our result
  73. 73. Photoshop CAF [Wexler et al. 2007] [Barnes et al. 2009] Image Melding [Darabi et al. 2012] Statistics of Patch Offset [He and Sun 2012]
  74. 74. Conclusions • Mid-level constraints for the win!
  75. 75. Conclusions • Mid-level constraints for the win! • Our work: planes and regularity
  76. 76. Conclusions • Mid-level constraints for the win! • Our work: planes and regularity • Great for man-made scenes
  77. 77. Conclusions • Mid-level constraints for the win! • Our work: planes and regularity • Great for man-made scenes • Backward-compatible with natural scenes
  78. 78. Thank You!

×