The document presents a novel method for contextual classification of very high resolution images. The method combines adaptive texture extraction using semivariograms, multiscale segmentation, and Markov random field modeling for spatial information fusion. It was shown to achieve very accurate classification results on an IKONOS image, correctly classifying both structured and textured classes without border artifacts. This was an improvement over previous methods and addressed limitations of existing approaches. Future work could generalize the method by integrating additional information sources like edges.

1. IGARSS-2011
Vancouver, Canada, July 24-29, 2011
Contextual High-Resolution Image
Classification by Markovian Data Fusion,
Adaptive Texture Extraction, and
Multiscale Segmentation
Gabriele Moser
Sebastiano B. Serpico
University of Genoa Department of Biophysical
and Electronic Engineering

2. 2
Outline
• Introduction
– Contextual very high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

3. 3
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

4. 4
Introduction
• Very high-resolution (VHR) optical remote-
sensing images:
– Very interesting in land-use / land-cover mapping,
especially in urban and built-up area analysis.
– 0.5 ÷ 5-m resolution available thanks to current
(e.g., IKONOS, QuickBird, WorldView-2, GeoEye-
1) and forthcoming (e.g., Pleiades) missions.
– Increased need to model spatial information due to
limited spectral information (few spectral channels)
• A novel contextual classification method is
proposed for HR optical images, based on:
– Adaptive texture extraction by semivariogram;
– Multiscale segmentation;
QuickBird, panchromatic, 1 m
– Markov random fields for spatial information fusion.
University of Genoa Department of Biophysical
and Electronic Engineering

5. 5
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

6. 6
The Proposed Approach
How to incorporate spatial information?
– Region-based approaches: usually effective for classes
with geometrical structures (e.g., urban).
– Texture analysis: effective for natural and artificial textured
classes, especially for images with few spectral channels;
– Texture analysis: often introduce artifacts at the object
borders (due to moving-window processing).
Key-ideas
– Integrating segmentation and texture information by
incorporating semivariogram features into a previous
multiscale region-based MRF model.
– Applying spatially adaptive texture extraction to prevent
border artifacts.
University of Genoa Department of Biophysical
and Electronic Engineering

7. 7
Overview of the Proposed Method
Initialization phase
Generate a preliminary classification map L0 by applying a
previous region-based MRF classifier [5] to the input image X.
Iterative phase
Extract a set Ft of texture features by applying to X the
proposed adaptive semivariogram method, based on the class
borders in the current map Lt.
Stack together X and Ft and generate a set St of Q
t=t+1
segmentation maps, each related to a different spatial scale, by
applying a scale-dependent segmentation method to (X, Ft).
Generate the updated map Lt + 1 by applying a previous region-
based MRF classifier [5] to the multiscale segmentation St.
yes no
convergence?
STOP
University of Genoa Department of Biophysical
and Electronic Engineering

8. 8
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

9. 9
Adaptive Semivariogram Extraction
γ i (h ) =
1
2
{
E ( xi − x j ) 2 i−j 2 }
=h (h ≥ 0) Semivariogram
– Local 2nd order statistics γi(h)

∑ δ (ℓti , ℓtj ) xi − x j 2
2

for a single-channel image.
1 j∈Rihw
γ i (h | w , L ) =
t
ˆ Multispectral extension by


2 ∑ δ (ℓti , ℓtj )
j∈Rihw
–
(possibly weighted) Euclidean
 distance.
R =  j : i − j = h, i − j < w 
 ihw 
  1 ∞ 
2 – Usually estimated with a w × w
moving window.
Proposed adaptive estimation
– Use, for each pixel i, the pixels
that both belong to the related
i w × w moving window and
w×w share the same label as i in the
window current map.
– 1-norm on the pixel grid for
Current map Lt:
colors denote class labels; yellow convenience.
borders denote pixels used to estimate semivariogram
University of Genoa Department of Biophysical
and Electronic Engineering

10. 10
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

11. 11
Markov Random Fields
• MRF model for the spatial context
– Representation of the statistical interactions between the pixel
labels in an image by using only local relationships:
( ) (
P ℓi ℓ j , j ≠ i = P ℓi ℓ j , j ∼ i )
Labels in the neighborhood
(here, 3 × 3)
i
• MRF-based classification
– Minimization of a (posterior) energy function U(·), thanks to
the Hammersley-Clifford theorem. Here:
Q
U (L | S ) = − ∑∑ α q ln P (siq | ℓ i ) − α 0 ∑ δ (ℓ i , ℓ j )
t t
i q =1 i∼j
Pixelwise probability mass function (PMF) of the segment labels in the
segmentation map at each scale and each iteration, conditioned to each class
University of Genoa Department of Biophysical
and Electronic Engineering

12. 12
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

13. 13
Segmentation and PMF Estimation
• Felzenszwalb & Huttenlocherm segmentation method
– Graph-based region-growing method depending on a scale
parameter.
– Segmentation at different scales by varying the scale
parameter.
• Class-conditional PMF estimation
– Extension of a previous method that computes relative-
frequency estimate [5], based, at each t-th iteration, on a
preliminary intermediate map Mt obtained classifying (X, Ft).
– To generate Mt from the HR stacked image (X, Ft), a non-
parametric contextual method is desirable.
– Here, a recent (non-region-based) method that combines
MRFs and support vector machines (SVMs) is used [9].
University of Genoa Department of Biophysical
and Electronic Engineering

14. 14
Parameter Estimation
and Energy Minimization
• Weight parameters α in the MRF
– Extension of a recent method based on the Ho-Kashyap
algorithm.
• Energy minimization: iterated conditional mode (ICM)
– Initialized with the intermediate preliminary map Mt.
– Converges to a local energy minimum.
– Usually good tradeoff between accuracy and processing time.
University of Genoa Department of Biophysical
and Electronic Engineering

15. 15
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

16. 16
Data Set and Experimental Set-up
• Data set
– Itaipu (Brazil/Paraguay), IKONOS, 3
channels, 1999 × 1500 pixels
• Set-up
– Q = 5 scales, 7 × 7 window (w = 7).
– Preliminary experiments suggested
limited sensitivty of the accuracy to
(w, Q) for 5 ≤ w ≤ 31 e 2 ≤ Q ≤ 5.
– SVM applied with Gaussian kernel.
– Kernel and regularization parameters
RGB false color
in the SVM optimized by a recent
method based on the numerical
minimization of the span bound.
urban
herbaceous rangeland
schrub and brush rangeland
forest land
barren land
built-up (non-urban)
Training map Test map water
University of Genoa Department of Biophysical
and Electronic Engineering

17. 17
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

18. 18
Classification Accuracies
– Very high test-set accuracies by the proposed method.
– Very similar test-set accuracies also by the previous method in [5]
(multiscale segmentation and MRFs, no textures) and by an SVM applied to
spectral and standard (non-adaptive) semivariogram features.
– Much lower test-set accuracies for an SVM applied only to the spectral
channels (expected result: no spatial information used).
– But... test samples located only inside homogeneous areas and not at the
class borders (usual in remote sensing).
University of Genoa Department of Biophysical
and Electronic Engineering

19. 19
Classification Maps: Previous Methods
RGB false color Method in [5]
SVM , spectral +
– Relevant visual differences between the semivariogram
benchmark considered methods.
– Errors for “herbaceous” (textured class;
e.g., white circle), but no border artifacts
by the method in [5].
– Correct classification of “herbaceous,” but
irregular behavior at the class borders by
SVM with standard semivariogram.
University of Genoa Department of Biophysical
and Electronic Engineering

20. 20
Classification Maps: Proposed Method
Proposed method Method in [5]
SVM , spectral +
– Correct classification of “herbaceous” semivariogram
– no border artifacts by the proposed
method.
– This suggests:
• effectiveness of the proposed adaptive
semivariogram
• capability of the proposed classifier to fuse
multiscale segmentation and texture
University of Genoa Department of Biophysical
and Electronic Engineering

21. 21
Classification Maps: Further Comments
RGB false color Proposed method
SVM , only spectral
– Visually noisy map by the SVM applied
only to the spectral bands (as expected).
– Spatially regular result, but no appreciable
oversmoothing by the proposed method.
– Time < 50 minutes for all considered
methods on a 2.33-GHz, 4-GB RAM pc
(usually acceptable time for land-cover
mapping).
University of Genoa Department of Biophysical
and Electronic Engineering

22. 22
Outline
• Introduction
– Contextual high-resolution image classification
• The proposed method
– Key ideas and overview of the method
– Adaptive semivariogram extraction
– Region-based multiscale MRF
– Segmentation, estimation, and optimization
• Experimental results
– Data set and experimental set-up
– Results evaluation and comparisons
• Conclusion
University of Genoa Department of Biophysical
and Electronic Engineering

23. 23
Conclusion
• Novel MRF-based VHR image classifier combining the
multiscale segmentation and texture to model spatial
information.
– Very accurate results for both textured and geometrically-
structured classes.
– No border artifacts, thanks to adaptive semivariogram.
– Improvement in class discrimination and/or border precision,
compared to previous methods.
• Possible future generalizations
– Integrating edge information (e.g., line processes).
– Approaching global energy minimization (e.g., graph-cuts).
– Comparisons with other methods for VHR image classification
– Experiments with other VHR data sets.
University of Genoa Department of Biophysical
and Electronic Engineering

24. 24
References
1. S. Li, Markov random field modeling in image analysis, Springer, 2009.
2. X. Descombes and J. Zerubia, “Marked point process in image analysis,” IEEE Signal Processing Magazine, vol. 19,
no. 5, pp. 77–84, 2002.
3. Q. Chen and P. Gong, “Automatic variogram parameter extraction for textural classification of the panchromatic ikonos
imagery,” IEEE Trans. Geosci. Remote Sensing, vol. 42, no. 4, pp. 1106–1115, 2004.
4. M. De Martino, F. Causa, and S. B. Serpico, “Classification of optical high-resolution images in urban environment
using spectral and textural information,” in Proc. of IGARSS-2003, Toulouse, France, 2003, vol. 1, pp. 467–469.
5. G. Moser and S. B. Serpico, “Classification of high-resolution images based on MRF fusion and multiscale
segmentation,” in Proc. of IGARSS-2008, Boston, USA, 2008, vol. II, pp. 277–280.
6. A. H. S. Solberg, T. Taxt, and A. K. Jain, “A Markov random field model for classification of multisource satellite
imagery,” IEEE Trans. Geosci. Remote Sensing, vol. 34, no. 1, pp. 100–113, 1996.
7. P. Li, T. Cheng, G. Moser, S. B. Serpico, and D. Ma, “Multitemporal change detection by spectral and multivariate
texture information,” in Proc. of IGARSS-2007, Barcelona (Spain), 23-28 July 2007, 2007, pp. 1922–1925.
8. P. F. Felzenszwalb and D. Huttenlocherm, “Efficient graph-based image segmentation,” Int. J. Comp. Vis., vol. 59, pp.
167–181, 2004.
9. G. Moser and S. B. Serpico, “Contextual remote-sensing image classification by support vector machines and markov
random fields,” in Proc. of IGARSS-2010, Honolulu (USA), 25-30 July 2010, 2010, pp. 3728–3731.
10. S. B. Serpico and G. Moser, “Weight parameter optimization by the Ho-Kashyap algorithm in MRF models for
supervised image classification,” IEEE Trans. Geosci. Remote Sensing, vol. 44, pp. 3695–3705, 2006.
University of Genoa Department of Biophysical
and Electronic Engineering