Super resolution

Introduction Image Registration Imaging Process IBP Irani and Peleg Algorithm Gradient-like Method Color SR
Super Resolution
Federico D’Amato Roberto Medico
University of Florence
June 9, 2014
F. D’Amato, R. Medico Super Resolution

Outline
1 Introduction
2 Image Registration
3 Imaging Process
4 IBP
5 Irani and Peleg Algorithm
6 Gradient-like Method
7 Color SR

Super Resolution Techniques
Super Resolution is a class of techniques that enhance the
resolution of an imaging system. There are 3 main approaches
to SR reconstruction of an high-resolution image from lower
resolution image(s):
• Interpolation-based
• Example-learning-based
• Multi-image-based

Interpolation-based
Figure: Interpolaton methods try to achieve a best approximation of a
pixel’s color and intensity based on the values at surrounding pixels

Example-learning-based
Correspondences between low-resolution and high-resolution
images are learned from a set of training images. The training
set consists of high-resolution / low-resolution pairs.

Multi-Image Super Resolution
Super-Resolution from image sequences attempts to
reconstruct the original scene image with high resolution given
a set of observed images at lower resolution.

Why Super Resolution?
Limit of camera resolution:
• Spatial limit → determined by spatial density of optical
sensor
• Optical blur → determined by the lens
How to improve camera resolution?
• Direct method: improving imaging system by
manufacturing technique (pixel density, lens size)
• Use of Super-resolution reconstruction:
• Use of spatial sub-pixel movement information between
frame
• Reconstruction from low-resolution image sequences to
high-resolution image

Digital Imaging System
Key components:
1 the sensor ⇒ limit on highest spatial frequency
2 the lens ⇒ optical blur
Figure: Image acquisition process

Spatial Aliasing
Spatial aliasing is an effect that causes different signals to
become indistinguishable (or aliases of one another) when
spatially-sampled. When a digital image is recorded, a
reconstruction is performed by the imaging device → if the
image data is not properly processed during sampling or
reconstruction, the reconstructed image will differ from the
original image (it’s called an ’alias’ of the original scene)
Figure: One signal and its alias

Aliasing components
• Sensor is constructed from a ﬁnite number of discrete
pixels → reconstruction of real world scene is affected by
aliasing effects
• It’s impossible to completely remove aliasing components
using anti-aliasing ﬁlters ⇒ information in the aliased
components is used to recover spatial frequencies
beyond sensor resolution
• It’s the possible to use information to improve the image
resolution

Aliasing effect on patterns of increasing frequency → poor (or
completely wrong) image reconstruction

Figure: Sub-Pixel shifted signals

Naive approach
• How can we compute the value
of pixel X?
• By applying some interpolation
technique (e.g. bilinear) to
neighbours A,B,C,D of X

Multi-image Approach
• LR image resolution: MxN
• Images displacement: half a
pixel
• Combining the pixel of the LR
images in a more dense grid
2Mx2N returns an image at
higher resolution.

Registration
• Computation of the changes (displacements) between
images is known as registration
• 2D Rotation matrix:
• Displacement are computed between one image g0 (taken
as reference image) and all the others image.
Displacement between gk and g0 can be written as:
g0(x, y) = gk (x cos(Θ)−y sin(Θ)+a, y cos(Θ)+x sin(Θ)+b)

Figure: Example of rigid registration

• Expand sin Θ and cos Θ to the ﬁrst two terms of their Taylor
series:
g0(x, y) ≈ gk (x + a − yΘ − x
Θ2
2
, y + b + xΘ − y
Θ2
2
)
• Expand gk to the ﬁrst term of its Taylor series:
g0(x, y) ≈ gk (x, y)+(a−yΘ−x
Θ2
2
)
∂gk
∂x
+(b+xΘ−y
Θ2
2
)
∂gk
∂y
• The error function between gk and g0 is:
E(a, b, Θ) = [gk (x, y) + (a − yΘ − x
Θ2
2
)
∂gk
∂x
+
+(b + xΘ − y
Θ2
2
)
∂gk
∂y
− g0(x, y)]2

• ∂E
∂a = 0, ∂E
∂b = 0, ∂E
∂Θ = 0
• Ignoring non-linear terms and small coefﬁcients we get the
following system of linear equations, whose solution
(a, b, Θ) minimizes the difference between g0 and gk
warped by (a, b, Θ):
g2
x a + gx gy b + Agx Θ = gx gt
g2
y b + gx gy a + Agy Θ = gy gt
A2
Θ + Agy b + Agx a = Agt
where gx = ∂gk
∂x , gy = ∂gk
∂y , gt = g0 − gk and A = xgy − ygx

Iterative refinement
When it’s not possibile to assume that the displacements
between g0 and gk are sufficiently small, an iterative
refinement algorithm is used:
1 Assume no motion between frames
2 for n=0,1,..
• Compute (a(n)
, b(n)
, Θ(n)
) and add the computed motion
to the current estimate (a, b, Θ)
• Warp frame gk towards g0 using (a, b, Θ) and return to 2.
The process ends when (a(n), b(n), Θ(n)) ≈ 0.

Receptive Field
The receptive ﬁeld of a LR pixel (m, n) of the kth LR image is
deﬁned by its center (x, y) and its shape, determined by the
region of support of hPSF (·) in the high resolution grid. The
center (x, y) can be computed by:
x = ak + sx m cos Θk − sy n sin Θk
y = bk + sx m sin Θk + sy n cos Θk
where
• (ak , bk ) is the translation of the kth image from g0
• Θk is the rotation between the kth image and g0
• sx and sy are the upscaling factors in x and y directions

Figure: Receptive Field

• Imaging process can be modeled as:
gk (m, n) = σk (hPSF
(f(x, y)) + ηk (x, y))
where
• gk is the kt h observed LR image
• f is the original image
• hPSF
is a blurring operator
• ηk is an additive noise term
• σk is a non-linear function that digitizes and quantizes
image into pixels (including displacement)
• (x, y) is the center of the receptive ﬁeld of the detector
whose output is gk (m, n)

Figure: Simulated Imaging Process

Blurring Operator Estimate
Given a generic imaging device, we can empyrically estimate
its blurring function h(·) analyzing the output of the imaging
process of well-known sample scenes.

Iterative Back Projection
• Iterative algorithm based on a set of K low resolution
images of the same scene with known displacements
• Goal: to improve an initial guess of the HR image
iteratively minimizing an error function using
back-projection

Hypothesis
• Assumptions:
• displacement between images can be described by three
parameters:
• a, horizontal shift
• b, vertical shift
• Θ, rotation angle
• ignores acceleration of the camera while imaging a single
frame

Data:
• f0: initial guess of the HR image
• gk : set of LR observed images
• hPSF , (ak , bk , Θk ) ∀k = 1, .., K
for n = 0, 1, .. do
1 Compute the set of K simulated LR images {g
(n)
k } from f(n)
2 Compute en between gk and gk
(n)
if en > then
Update guess f(n+1) by back-projecting the error on f(n)
else
return f(n)
end
end
Algorithm 1: Iterated Back Projection

Simulated Imaging Process
How can we programmatically simulate the device imaging
process?
Def.
A low resolution pixel y is influenced by a high resolution pixel
x if x ∈ y’s receptive field
Def.
A low resolution image g is influenced by a high resolution
pixel x if ∃y ∈ g influenced by x

g(n)
(y) =
x
f(n)
(x)hPSF
(x − zy )
where
• hPSF is the point-spread kernel of the imaging blur
• x is an HR pixel
• y is a LR pixel inﬂuenced by x
• z is the center of y’s the receptive ﬁeld

Idea
• Given the g
(n)
k simulated LR images, the goal is to
minimize the error between {g
(n)
k } and {gk }.
• The minimization is obtained with the iterative
back-projection scheme, where ek = gk − g
(n)
k is
weighted computing the influence of every LR pixel y on
HR pixel x , using:
hBP(x − zy )
y∈ k Yk,x
hBP(x − zy )
where Yk,x is the set: {y ∈ gk | y is influenced by x}
• y has more influence when x is close to zy , center of y’s
receptive field
• The error is then multiplied by a factor:
hBP (x−zy )
c

Guess Improvement
• The value of x in the next guess f(n+1) is calculated
summing up the weighted errors on all the LR pixel y it
inﬂuences
f(n+1)
(x) = f(n)
(x)+
y∈ k Yk,x
(gk (y)−g
(n)
k (y))
(hBP(x − zy ))2
c y∈ k Yk,x
hBP
xy

Back-Projection Kernel
• hBP
xy
affects how much the error on LR pixel y (inﬂuenced
by x) contributes to the value of HR pixel x in the next
guess f(n+1)
• hBP affects the characteristics of the solution image, e.g.
its smoothness
• A possible choice is hBP = hPSF

Initial Guess Choice
• Initial guess f(0) can inﬂuence the output of the algorithm,
i.e. which HR image is reached ﬁrst
• One possibile choice of f(0) is taking the average of the
upscaled LR images gk
• This choice doesn’t affect performance

Algorithm Complexity
The complexity is O(KN min{M, log N}) where:
• K is the number of LR images
• N is the size of the HR image f
• M is the size of HPSF kernel
Parallelism can be used to compute the contributions of LR
pixels indipendently.

Improvements
• Stable Pixels: HR pixels that don’t change value for 2
consecutive iterations won’t be considered in the following
iterations
• Noise reduction: minimal and maximal values of
gk (y) − g
(n)
k (y) are ignored in computing the weighted
average of the contributions of the LR pixels in the iterative
back-projection scheme

Error Function
The error function to minimize is given by the MSE between the
simulated images g
(n)
k and the observed images gk :
e(n)
= k (gk − g
(n)
k )2
K

Algorithm
f(n+1)
= f(n)
− λG
where
• G = k HBP(g
(n)
k − gk )
• g
(n)
k is the kth simulated LR image at the nth iteration

YIQ representation
• Y component represents the luminance information
• I and Q represent the chrominance information
Most of the energy is concentrated in the Y component.

Color Super Resolution
It’s possible to use the SR algorithm even on color images,
going through 4 steps:
1 transform the color images in YIQ representation
2 apply the SR algorithm to the Y component images
3 register the images at the two (I,Q) chrominance image
sequences using parameters found in 2. Create an
average for each of the I and Q components
4 fuse the HR Y component and LR I and Q components to
generate a HR RGB image

References
• ’Image sequence enhancement using sub-pixel
displacements’, Keren, D., Peleg, S. ; Brada, R.; Dept. of
Comput. Sci., Hebrew Univ., Jerusalem, Israel
• ’Video Super-resolution Reconstruction Based on
Sub-pixel Registration and Iterative Back Projection’,
Journal of Electronic Imaging,Vol. 18, No. 1, 2009,
Feng-Qing Qin, Xiao-HaiHe, Wei-Long Chen, Xiao-Min
Yang, and Wei Wu
• ’Improving resolution by image registration’, Michal Irani,
Shmuel Peleg
• ’Super-Resolution’, Pradeep Gaidhani

Super resolution

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Super resolution

Semelhante a Super resolution (20)

Último

Último (20)

Super resolution