iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation

Accuracy Improvements and
Artifacts Removal in Edge Based
Image Interpolation
Nicola Asuni Andrea Giachetti
nicola.asuni@tecnick.com andrea.giachetti@univr.it

UNIVERSITY OF CAGLIARI - ITALY UNIVERSITY OF VERONA – ITALY
Faculty of Mathematical, Physics and Natural Sciences Department of Computer Science
Department of Computer Science Vision, Image Processing & Sound Lab

UNIVERSITY OF CAGLIARI
Faculty of Mathematical, Physics and Natural Sciences
Department of Computer Science

Summary
Introduction
interpolation and image upscaling, short literature review

NEDI - New Edge Directed Interpolation
how it works, limits

iNEDI – improved NEDI
modified method to reduce NEDI artifacts

Experimental Results
iNEDI provides the best results in a large set of objective and
subjective tests.

Conclusions

N. Asuni, A. Giachetti: Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation
VISAPP 2008 - 22/25 January, 2008 - Funchal, Madeira - Portugal
2


Interpolation
Constructing new data points from a discrete set of
known data points.
Construct a function which closely fits a discrete
set of known data points (curve fitting or
regression analysis).

Generally, it is an ill-conditioned problem.
The selection of the adequate interpolation method
depends by the nature of know data points.
3


Digital Image Interpolation (1)
Process that allows to obtain an High Resolution
image form its Low Resolution counterpart.

Images ©2005 Nicola Asuni - Alamy
LR

upsizing, resampling,
zooming, magnification, HR
resolution enhancement
4


Digital Image Interpolation (2)

LR

HR
The interpolation procedure tries to recover missing
information by assuming that there is a known
relationship between a low resolution image and the
same image acquired with an high resolution sensor.

5


Why do we need to resample?
Image upscaling through pixel interpolation is used in
different fields and devices to create high resolution
images with a "natural" appearance from low
resolution acquired data.

Applications of this procedure can be found
in image viewing or processing software,
photographic printing and Computer
Graphics.

Icons ©2005 www.tpdkdesign.net
Real time algorithms can
also be applied to increase
the perceived quality of
video streaming or textures
in virtual navigation tools.
6


Interpolation Artifacts
General upsampling algorithms do not add real
information. Interpolation cannot reconstruct original
high frequencies but edge geometries could be
preserved. Sharpness and freedom from artifacts in
edges are two critical factors in the perceived quality
of images.

aliasing (jaggies) blurring edge halo frequency artifacts
7


Interpolation Methods
Several algorithms have been proposed in literature to
obtain better results and several patents have been
obtained for "smart" interpolation techniques.
Few systematic comparisons have been, however,
presented and it is difficult to determine which method
is the best suited for a selected application.

Main categories
Linear Filtering
Non-linear methods
Statical methods
Edge-directed methods

8


Linear Filtering
The simplest image interpolation algorithms are based
on linear filtering.

OR NN BL BC

These methods are computationally efficient and
especially the bicubic interpolation provides visually
good images, that do not appear, however, "natural"
due to blur and jagged contours.

9


Non-Linear Methods
Non linear methods are usually based on an implicit or
explicit search of local image features and on a
subsequent local adaptation of the interpolation
function to the (low resolution) extracted features.
Lu, X., Hong, P. S., and Smith, M. J. T. (2003). An efficient directional image
interpolation method. In Proc. IEEE Int. Conf. Acoustics Speech Signal
Processing, volume 3, pages 97–100.
Schultz, R. R. and Stevenson, R. L. (1994). A bayesian approach to image
expansion for improved definition. IEEE Trans. Image Processing, 3:233–
242.
Takahashi, Y. and Taguchi, A. (2002). An enlargement method of digital
images with the prediction of highfrequency components. In Proc. IEEE Int.
Conf. Ac. Speech Signal Proc., volume 4, pages 3700–3703.
Morse, B. and Schwartzwald, D. (2001). Image magnification using level-
set reconstruction. In Proc. IEEE Conf. Computer Vision Pattern
Recognition, volume 3, pages 333–340.
Muresan, D. and Parks, T. (2004). Adaptively quadratic (aqua) image
interpolation. IEEE Transactions on Image Processing, 13(5):690–698.

10


Statistical Methods
Some authors have tried to exploit pixel or texture
statistics or databases of example images to obtain
good high resolution images.
The huge variety of natural textures and scales makes,
however, quite difficult a general purpose use of
similar techniques, though they can be efficiently
applied to particular tasks such as searching patterns
like faces, trees, etc.

Atkins, C. B., Bouman, C. A., and Allebach, J. P. (2001). Optimal image
scaling using pixel classification. In Proc. IEEE Int. Conf. Image
Processing, volume 3, pages 864–867.
Freeman, W. T., Jones, T. R., and Pasztor, E. C. (2002). Example-based
super-resolution. IEEE Computer Graphics and Applications, 22(2):56–65.
Sun, J., Zheng, N., Tao, H., and Shum, H. (2003). Image hallucination
with primal sketch priors. In Proceedings IEEE conf. on Computer Vision
and Pattern Recognition, volume 2.

11


Edge-Directed Methods
These methods try to improve the accuracy of the
interpolation characterizing the edge features in a
larger region around the point to interpolate. They
aims at interpolating along edges rather than across
them to prevent blurring. They usually produce images
with sharper edges.
J. Allebach and P. W. Wong, "Edge-directed interpolation,"
Proc. IEEE Int. Conf. Image Processing, vol. 3, pp. 707–710,
September 1996.
X. Li and M. T. Orchard, "New Edge-Directed Interpolation,"
IEEE Trans. on Image Processing, Vol. 10, No. 10, pp.
1521-1527, October 2001.

NEDI seems to provide the best results for natural
images, even in the case of large scale factors. This is
why we start our analysis describing this technique
and then proposing several improvements.
12


NEDI (New Edge-Directed Interpolation)
Edge-directed adaptative method for natural
images. Its behavior depends on what it is
interpolating. It uses bilinear interpolation for
smooth regions.
This method first estimates local covariance
coefficients from a low-resolution image and then
use these covariance estimates to adapt the
interpolation at a higher resolution based on the
geometric duality between the low-resolution
covariance and the high-resolution covariance.
The image size is doubled in each direction at
each cycle.

13


NEDI – 4th order interpolation (1)

The darker circles
represent the pixels
from the original Low
Resolution grid, the
white circles represent
the pixels that are to be
interpolated and the
central circle is the pixel
to be interpolated in
this first step.

14


NEDI – 4th order interpolation (2)

The central point is
calculated using a fourth-
order interpolation algorithm
that simply sums the
contribute of each diagonal
neighbor multiplied by its
alpha coefficient.

15


NEDI – Estimation of Coefficients (1)
NEDI estimates α
coefficients by solving
an unconstrained
system of linear
equations.
The system is
obtained by assuming
that the coefficients
linking each pixel with
its four diagonal
neighbors do not
change with scale and
that they are constant
in a squared window
W centered in the
pixel to interpolate.
16


NEDI – Estimation of Coefficients (2)
The y vector contains the
values of the pixels
contained on the window W.
The C matrix contains in
each row the values of the
four diagonal neighbors of
each pixel listed on y vector.

NEDI estimates α coefficients
by minimizing the Mean
Square Error (MSE).

17


NEDI – Second Step
The second NEDI step fills
the remaining gaps in the
same way of the first step
after a π/4 rotation of the
grid.

The darker circles represent
the pixels from the original
Low Resolution grid, the
white circles represent the
pixels that are to be
interpolated and the gray
circles represent the pixels
interpolated on the first
step.

18


NEDI – Example 8X

NEAREST NEIGHBOR BICUBIC NEDI

Jagged contours are evident in nearest neighbor and
bicubic interpolation.
The NEDI interpolation presents sharp edges, even if
introduces different artifacts and performs often worser
than the bicubic method in quantitative comparisons.
19


NEDI – Limits

The assumption of local stationarity of the
covariance (local regularity in curvature) is
violated in several cases.
Introduces several artifacts on high-frequency
regions.
The system is often ill-conditioned. The use of
larger windows improves the conditioning of the
CTC matrix but produces blurred images.
Interpolated pixel values change with the global
brightness.
Very high computational complexity.

20


iNEDI (improved NEDI)
iNEDI (improved New Edge-Directed Interpolation)
is the proposed technique to fix/reduce the original
NEDI problems and increase the interpolation
accuracy.

Main improvements:
Non edge pixels handling
Windows shape and dynamic windows
Edge "segmentation"
Value Adjust
Matrix conditioning, error propagation and
minimum norm solution

21


iNEDI – Non edge pixels handling
When the four pixels used to calculate the interpolated
ones have a similar gray level, there is no need to
compute the NEDI coefficients. If the covariance is
stationary, a small error causes a bad conditioning of
the solution.
If the local gray level variation is
below a fixed threshold NEDI uses
the bilinear interpolation, while
iNEDI uses bicubic interpolation.
This choice gives a good trade-off
between edge direction
preservation, accuracy and
speed, using higher values of the
threshold (using iNEDI only for
strong edges).

22


iNEDI – Windows shape
The squared window W
used by NEDI introduces
directional artifacts and
makes the algorithm non
isotropic. This has been
solved by iNEDI using a
circular window.
To account for various
frequencies regions, iNEDI
can dynamically increase
the window radius from a
minimum to a maximum
value while the normalized
residual of the least
squares fit decreases.

23


iNEDI – Edge "segmentation"

iNEDI tries to exclude from
the circular window all
pixels that do not belong to
the local edge using a sort
of region growing method
that excludes all uniform
areas and pixels that are
not connected to the edge.
Two pixels are connected if
they are adjacent and their
values are included in a
predefined range.

24


iNEDI – Value Adjust

iNEDI increases the probability of obtaining a good
interpolation, but there is still the possibility of
having unwanted high frequencies.
For this reason we put a further constraint by
replacing any interpolated value outside the intensity
range of the four neighbors with the closest of the
values delimiting that range.

25


iNEDI – Matrix conditioning, error
propagation and minimum norm solution
The overconstrained system Cα = y is almost
always ill-conditioned and a small error in y can
cause a large error in the estimated α.
A simple trick is to add a constant value to the
gray levels, in order to have all values far from
zero. This change is effective in reducing artifacts
and wrong estimates.
The solution to the least squares problem is not
unique. On iNEDI we rewrite the system as α = C+y
where C+=(CTC)-1CT is the Moore-Penrose
pseudoinverse matrix, and find the minimum norm
solution using the Singular Value Decomposition. In
this way we avoid the direct inversion of CTC
matrix.
26


iNEDI – Implementation for Testing
iNEDI prototype has been implemented in
matlab language (compatible with MATLAB®
and Octave software applications) and released
as a FLOSS by adopting GPL v.2 license.
http://inedi.tecnick.com
iNEDI code includes:
Simple edge detection on threshold
Padding procedures to preserve image borders
Cropping and Rotation procedures
RGB image handling

27


iNEDI vs NEDI – 16X Example

NEDI iNEDI
28


Experimental Results
iNEDI has been widely tested and compared with other
methods found in literature, as well as with the original
NEDI. We have used both objective and subjective
evaluation tests.
Objective evalutation
A simple test often used in literature consists of
generating low resolution images by filtering and
subsampling high resolution ones and then measure
the difference between the differently re-upsized
images and the original one.
Subjective evalutation
We have asked to a group of 24 people to sort from
the worse to the best a set of 12 RGB images
enlarged by a 8x factor with four different algorithms
(iNEDI, NEDI, bicubic and bilinear).
29


Results – Objective Metrics
PSNR (Peak-to-Signal Noise Ratio) is used to compare
the differences between interpolated and reference
images. PSNR measures the accuracy of reconstruction.

The constant MAXPIX is the end scale value of the pixel
intensity. MSE is the Mean Square Error between pixel
values of the original and interpolated image.
Typical PSNR values are between 20 and 40 dB. Higher
values indicates a better image reconstruction.
Improvements of 0.1 dB are usually considered
significative.
30


Results – Test images
9 8bit 1025x1025 natural
images, subsampled at
256x256 and 128x128 and
then upscaled respectively
two times and four times
using different algorithms.
Different target images
(512x512 and 511x511) were
used to account the different
image shifts introduced by
the various techniques.
We also crop a 50 pixel frame
before comparison to account
for the different behavior of
the selected methods near
the image border.

31


Results – 2X PSNR Measures
iNEDI improvements vs NEDI are between 0.54
and 1.63 dB with an average value of 0.85 dB.

32


Results – 4X PSNR Measures
iNEDI improvements vs NEDI are between 0.52
and 1.41 dB with an average value of 0.85 dB.

33


Results – PSNR Measures
The iNEDI accuracy of the
reconstruction is higher than those
obtained with the other techniques.
NOTE: The good results of the bicubic
interpolation does not mean that it is
surely better than other methods.
Original NEDI, as well as the other edge
based method tested, are effective in
removing the typical artifacts of the
bicubic and bilinear interpolation. The
lower PSNR is probably due to the other
kinds of artifacts affecting NEDI and the
excessive smoothing of the other
approaches.

34


Results – 8X Artificial Image

+3.43 dB
iNEDI removes
the jagged
lines effects of
the linear
methods and
the directional
artifacts of
NEDI.
35


Results – Extended Comparison
To compare the iNEDI technique
with other techniques available on
commercial software, we have
extended the test provided by the
SAR Image Processor Software:
http://www.general-cathexis.com/interpolation.html

The iNEDI algorithm provided a
PSNR 1 dB higher than the best one
of the 42 methods included (DDL
with SuperRez postprocessing).

36


Results – Subjective evaluation
Traditional Signal-to-Noise
Ratio measures cannot be
compared with the perceived
quality by humans.
A group of 24 people have
been asked to give a
qualitative judgment on 12
color images originally of
80x60 pixels and enlarged of
a factor 8 with iNEDI, NEDI,
Bicubic and Bilinear
interpolation.
The qualitative judgment has
been performed sorting the
images from the worst (1) to
the best (4).
37


Results – Subjective measures

The subjective results confirmed the results
of the analysis based on the PSNR.

38


Conclusions
Images enlarged with the proposed technique appear
more natural and less smoothed than those obtained
with other approaches presented in literature, and
both psychological and quantitative tests confirm this
fact.
For selected applications, such as printing or off line
extrapolation of high resolution textures from low
resolution data, the relevant computational effort is
not a problem, while for applications requiring a fast
image processing, such as improving quality of video
streaming, different methods should be applied, even
if the algorithm can be optimized and parallelized.

39


Thanks for your attention.
Any Questions?

40

iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation

Semelhante a iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation (20)

Mais de Tecnick.com LTD

Mais de Tecnick.com LTD (6)

Último

Último (20)

iNEDI - Accuracy Improvements and Artifacts Removal in Edge Based Image Interpolation