RESEARCH

1. A New Model for the Segmentation of Multiple,
Overlapping, Near-Circular Objects
Csaba Molnar∗, Zoltan Kato∗† and Ian H. Jermyn‡
∗Institute of Informatics, University of Szeged, Szeged, Hungary,
Email: mcsaba@inf.u-szeged.hu, kato@inf.u-szeged.hu
†Department of Mathematics and Informatics, J. Selye University, Komarno, Slovakia
‡Department of Mathematical Sciences, Durham University, Durham, United Kingdom
Email: i.h.jermyn@durham.ac.uk
Abstract—Some of the most difficult image segmentation
problems involve an unknown number of object instances that
can touch or overlap in the image, e.g. microscopy imaging of cells
in biology. In an important set of cases, the nature of the objects
and the imaging process mean that when objects overlap, the
resulting image is approximately given by the sum of intensities
of individual objects; and, in addition, the objects of interest
are ‘blob-like’ or near-circular. We propose a new model for the
segmentation of the objects in such images. The posterior energy
is the sum of a prior energy modelling shape and a likelihood
energy modelling the image. The prior is a multi-layer nonlocal
phase field energy that favours configurations consisting of a
number of possibly overlapping or touching near-circular object
instances. The likelihood energy models the additive nature of
image intensity in regions corresponding to overlapping objects.
We use variational methods to compute a MAP estimate of the
object instances in an image. We test the resulting model on
synthetic data and on fluorescence microscopy images of cell
nuclei.
I. INTRODUCTION
Key to many applications of image processing and com-
puter vision is the ability to segment objects of interest for
subsequent analysis, i.e. to find the region in the image
domain corresponding to each object instance. In all except the
simplest cases, this is not possible without including significant
prior knowledge about the objects involved. Knowledge is
needed concerning both the shape of the object instances and
their disposition in the image, encoded in a prior distribution,
and the appearance of object instances in the image, encoded
in a likelihood; the more difficult the problem, the greater the
quantity of prior knowledge required.
Some of the most difficult segmentation problems arise
when there is an unknown number of instances of a particular
object present, and these instances touch or overlap in the
image. Many examples arise in microscopic imaging, in both
biological (cells, lipid droplets, sub-cellular objects e.g. nuclei)
and physical sciences (e.g. nanoparticles). In an important
subset of these examples, including those just mentioned, the
objects involved have near-circular shapes. We focus on these
cases in this paper.
It is clear that shape information is particularly important
in these situations, in order to separate overlapping objects.
However, classical shape modelling techniques based on tem-
plates or reference shapes [1]–[8] do not extend well to the
case of an unknown number of instances. This is partly a
question of efficient representation: typically each new object
increases the dimensionality of the representation space; and
partly a question of modelling: typical approaches achieve
geometric invariance via mixture models over a group, which
requires hidden alignment variables to be estimated for each
instance. The way around the first difficulty is to use a level set
representation, but this does not combine well with template-
based shape modelling methods unless only one instance
is involved. A different approach to the inclusion of shape
information is therefore needed.
One way to do this is to encode shape information as in-
teractions of the object boundary with itself, as with boundary
smoothness terms in active contour models. In the latter case,
the interaction is local, involving derivatives, but less generic
shape information requires nonlocal interactions, leading to
higher-order active contours [9]. This approach has the advan-
tage that no templates are used and invariance is intrinsic: no
alignment variables are needed. Level set methods can there-
fore be used to represent multiple object instances efficiently. It
turns out, however, that the standard distance function level set
representation causes severe computational difficulties when
such interactions are introduced. These difficulties can be
surmounted by using a phase field representation, and using a
model energy expressed in terms of the phase field rather than
a contour or distance function [10]. Multiple object instances
can therefore be represented at no extra cost, while shape
information is included in an efficient manner.
Examples of such nonlocal phase field models have
been used to address precisely the problem class of interest
here [11]. However, these models were not suited to cases
in which object instances touch or overlap, first because the
model could only represent subsets of the image domain, and
second because inter-object interactions tended to keep objects
a certain distance apart. To overcome these issues, a multi-
layered generalization of the model in [11] was introduced
in [12], with an equivalent binary MRF formulation described
in [13].
It is not just shape information that is required to solve
these difficult segmentation problems, however: sophisticated
image models are needed too. In conventional photographic
imaging, occlusion means that we need only be concerned
with the object closest to the camera in any given ray direc-
tion. In many imaging modalities, however, and in particular
in transmission (light microscopy, transmission electron mi-
croscopy) and emission (fluorescent microscopy) based imag-
978-1-4673-6795-0/15/$31.00 ©2015 IEEE

2. ing techniques, the image intensity in areas corresponding to
overlapping object instances is approximately given by the sum
of the intensities corresponding to individual objects, and this
must be taken into account if segmentation is to be successful.
In this paper, we introduce a new image model, formulated as
a likelihood energy constructed in the phase field framework,
which models the additive properties of such images.
We combine the prior model of a ‘gas’ of possibly over-
lapping, near-circular object instances formulated as a multi-
layer, nonlocal phase field energy, described in section II,
with the phase field formulation of the new image model,
described in section III, and compute the MAP estimate of
the object instances in a given image using gradient descent.
In section IV, we report on the model’s performance, both on a
synthetic data set and on real fluorescence microscopy images.
II. SHAPE MODEL
In this section, we review the phase field framework and
describe the prior, shape model, formulated as a multi-layer,
nonlocal phase field energy.
A. Phase field model of a gas of near-circles
A phase field function ϕ is a level set representation: given
a threshold z it determines a region R in the image domain
D: R = {x ∈ D : ϕ(x) ≥ z}. Unlike distance function level
sets, however, ϕ has no a priori form; rather it is controlled by
the model itself, which gives rise to a number of advantages,
particularly when nonlocal interactions are present [10].
The nonlocal phase field energy used to model a gas of non-
touching, non-overlapping near-circular objects in [11] has the
following form:
Ef (ϕ) =
∫
D
{
Df
2
|∇ϕ|2
+ αf
(
ϕ −
ϕ3
3
)
+ λf
(ϕ4
4
−
ϕ2
2
)}
−
βf
2
∫∫
D×D′
∇ϕ · ∇ϕ′
G(x − x′
) , (1)
where (un)primed functions are evaluated at (x ∈ D) x′
∈
D′
≡ D. The interaction function G : R2
→ R is
G(z) =



1
2
(
2 −
|z|
d
−
1
π
sin
(
π(|z| − d)
d
))
, |z| < 2d ,
0 else ,
(2)
where d controls the range of interaction and H is the
Heaviside step function.
It can be shown that ϕR, the minimizing ϕ for a fixed region
R, takes the value +1 in R and −1 in its complement, away
from the boundary, with a smooth ‘boundary layer’ transition
between the two values. The quantity (ϕR + 1)/2 is thus a
smoothed version of the indicator function of R. When βf = 0,
the energy of ϕR is given (to a very good approximation) by
a linear combination of region boundary length and interior
area. The model can therefore be used in place of a classic
active contour, but with the concomitant advantages of the
phase field framework. When βf ̸= 0, the model is equivalent
to a higher-order active contour model [9]. A stability analysis
of the model [14], translated to the phase field framework [11],
then shows that for appropriate ranges of the parameters, the
energy favours configurations consisting of a number of near-
circular shapes, a ‘gas’ of near-circles. The stability analysis
also reduces the number of free prior parameters from four to
two, and places constraints on these remaining values.
B. Multi-layer model
The above model is appropriate when object instances are
well-separated, but it has a severe limitation for the problems
of interest here: it cannot represent touching or overlapping
object instances, because a phase field function represents
subsets of D; and the nonlocal term in the energy, as well as
generating the desired near-circular shapes, also causes object
instances with small separation to have high energy.
To remove these difficulties, a multi-layer version of the
above model was developed in [12]. Let ϕ̃ =
{
ϕ(i)
}
i∈[1..ℓ]
:
[1..ℓ] × D → R, where ℓ is the number of layers. The energy
Ẽf (ϕ̃) of the model is simply the sum of energies of indepen-
dent layers extended with a term that penalizes overlapping
pairs of object instances by an amount proportional to overlap
area:
Ẽf (ϕ̃) =
ℓ
∑
i=1
Ef (ϕ(i)
) +
κ
4
∑
i̸=j
∫
D
(1 + ϕ(i)
)(1 + ϕ(j)
) , (3)
where κ controls the overlap penalty. This model solves the
two issues mentioned above. Overlapping object instances
can now be represented by appearing in different layers,
and the inter-object repulsion is now removed because it is
energetically favourable for nearby objects to be represented
on different layers, thus incurring no energy penalty.
The overlap penalty served two purposes in [12]. First, it
had a genuine modelling role to play when overlapping objects
were genuinely less probable than non-overlapping objects.
Second, it served to prevent degenerate configurations in which
the same object instance was represented in every layer. This
was needed because the image model used in [12] coupled
independently to each layer. We now describe a new image
model, that, in addition to being a more accurate model of
the images we deal with, obviates the need for the overlap
penalty. From now on, we therefore set κ = 0 in Ẽf . This is a
significant advantage of the new model, since finding a good
value for κ was difficult.
III. IMAGE MODEL
In many applications, e.g. microscopy using transmission-
or emission-based imaging techniques, the image intensity in
areas corresponding to overlapping objects is approximately
equal to the sum of the intensities of individual objects. Failure
to take this into account leads to segmentation errors, and
subsequent errors in the shape and number of object instances.
We now describe a model of such images designed to avoid
such errors.
We model the image intensities in the background and in
the single-object foreground (e.g. a cell) as having fixed (but
different) means and variances, leading to Gaussian distribu-
tions with independent pixel intensities by maximum entropy.
When several objects overlap, we model the resulting image

3. (a) (b)
Fig. 1. Illustration of the proposed image model: (a) shows a synthetic image;
in (b), the red and green surfaces are the phase field functions in two different
layers, while the blue surface shows the resulting ϕ+.
as the sum of the intensities from the background and each of
the overlapping objects, so that the resulting model is again
Gaussian with independent pixels, but with mean and variance
equal to the sum of the means and variances of the background
and the objects.
We define ϕ+ =
∑ℓ
i=1
(ϕ(i)
+1)
2 , which represents the num-
ber of overlapping objects at each point. Then the likelihood
energy is
Eintensity(I, ϕ+) =
∫
(I − (µ− + ∆µϕ+))2
2(σ2
− + ∆σ2ϕ+)
, (4)
where I is image intensity; µ− and σ2
− are the mean and
variance of the background; and ∆µ and ∆σ2
are the changes
in mean and variance brought about by each new overlapping
object.
Note that when ϕ+(x) ≃ 0, x is in the background, and
the mean and variance of the intensity are µ−, σ2
−. When
ϕ+(x) ≃ n, there are n overlapping objects at x. The mean
is then µ− + n∆µ and the variance is given similarly. Thus
this model implements the type of additive image model
discussed in section I. Fig. 1 illustrates the representation and
the functioning of the image term on a synthetic image.
A. Curing a problem with the energy
The expression for the energy in Eq. (4), although it
appears sensible, has a fundamental problem: it is not bounded
below as ϕ+ 7→ −∆σ2
σ2
−
. Although the prior energy may
discourage such a value, no finite amount of prior energy can
offset the divergence in Eintensity. As a result, the minimum of
E will be −∞, and the MAP estimate will be any assignment
of values to the {ϕ(i)
} that achieve this bound; the data, and the
prior, will be irrelevant. Needless to say, this is not desirable.
Luckily it is easy to cure: we replace ϕ+ in Eq. (4) by
ϕ̃+ =
∑l
i=1(tanh(ϕ(i)
) + 1)/2; the value of each term in ϕ̃+
is confined to [0, 1], thereby curing the divergence. In practice,
because Ef encourages ϕ(i)
to be close to ±1 anyway, each
term takes on a value very close to 0 or 1; the interpretation
of ϕ̃ as the number of object instances overlapping a point is
therefore preserved. The likelihood energy becomes
Ẽintensity(I, ϕ̃+) =
∫
(I − µ− − ∆µϕ̃+)2
2(σ2
− + ∆σ2ϕ̃+)
. (5)
B. Functional Derivative
The posterior energy is given up to an additive constant
by E = Ẽintensity + Ẽf . Note that there is in theory a
contribution coming from the normalization constant of the
likelihood energy. However, this depends only weakly on ϕ+,
and we ignore it here. To compute MAP estimates of the object
instances given an input image, we will use gradient descent.
We therefore need the functional derivative of E with respect
to each of the phase field components ϕ(i)
. For Ẽf , this reduces
to the derivative of the ith
term in the sum, which is the same
as for the single-layer model; this result can be found in [10].
The derivative of the new likelihood energy Ẽintensity is found
as follows. Under an infinitesimal variation in ϕ̃+, the change
in Ẽintensity is
δẼintensity =
1
2
∫
D
[2(I − µ− − ∆µϕ̃+)(−∆µ)(σ2
− + ∆σ2
ϕ̃+)
(σ2
− + ∆σ2ϕ̃+)2
−
(I − µ− − ∆µϕ̃+)2
∆σ2
(σ2
− + ∆σ2ϕ̃+)2
]
δϕ̃+ . (6)
Using δϕ̃+ = 1
2
∑
i sech2
(ϕ(i)
) δϕ(i)
, expanding the brackets
and dividing by δϕ(k)
, and using that δϕ(i)
(x)
δϕ(k)(y)
= δikδ(x, y), the
functional derivative of Ẽintensity with respect to ϕ(k)
becomes
δẼintensity
δϕ(k)
=
1
4
sech2
(ϕ(k)
)
[∆µ2
∆σ2
ϕ̃2
+ + 2∆µ2
σ2
−ϕ̃+
(σ2
− + ∆σ2ϕ̃+)2
+
−2∆µσ2
−(I − µ−) − ∆σ2
(I − µ−)2
(σ2
− + ∆σ2ϕ̃+)2
]
. (7)
IV. EXPERIMENTAL RESULTS
To segment the object instances in an image, we will
compute a MAP estimate. This will be done using gradient
descent based on a simple forward Euler scheme, with the
functional derivatives as given in the previous section. We
first present the quantitative results of applying this estimation
procedure to a set of synthetic images designed to fit the
image model, in order to study computation time and the
behaviour of the phase field during optimization. We then
present a comparison with two other methods on fluorescence
microscopy images of prostate cell nuclei.
A. Initialization
Gradient descent methods can only find local minima,
meaning that the initialization of the phase field layers is an
important part of the optimization. We tested two different
initializations. The ‘neutral’ initialization consists of Gaussian
noise with mean αf /λf (the local maximum of the potential)
and a very small variance. The ‘seeded’ initialization consists
of small circular regions, one in the interior of each object. We
construct the seeded initialization using an adaptive threshold-
ing method to find local image maxima, which then serve as
the seeds. The seeds are distributed between the layers so as
to maximize the minimum distance between seeds in the same
layer.

4. B. Synthetic results
The first test database contains 200 images. There are
smaller (150 × 150) images containing 4–10 circles, and
larger (400 × 400) images containing 30, 35, 40 circles of
radius 15. The background and foreground intensities were
chosen randomly from the sets {30, 40, 50} and {90, 100, 110}
respectively, with 10 dB signal-to-noise ratio. The goals of the
tests were, first, to check that the likelihood energy functions
as planned, and, second, to compare the neutral and seeded
initializations of the phase field. For the neutral initialization,
three layers were used, and for the seeded initialization,
the seeds were distributed in a simple way between layers,
resulting in 2–5 layers in the experiments.
In Fig. 2, the tests use the neutral initialization. The
likelihood energy works as expected: brighter areas are indeed
covered by multiple objects, although the result is not always
a perfect segmentation. Fig. 3 compares the results from the
two initializations. The left-hand plot shows the proportion
of correctly detected objects as a function of the relative
weight of the likelihood energy. The right-hand plot shows the
‘segmentation error’, also as a function of the relative weight
of the likelihood energy. The segmentation error measures
the pixel misclassification rate, computed by comparing well-
detected object instances to their ground truth equivalents:
SErr(GT, SEG) = |∆(GT,SEG)|
|GT |+|SEG| , where GT and SEG
denote the ground truth object instance and its segmentation
respectively, and ∆ is the symmetric difference operator. It is
quite clear that the seeded initialization improves the results
obtained by the model (some examples of segmentations are
shown in Fig. 4), with this initialization resulting in very
accurate segmentations.
Fig. 2. Results on synthetic images using three layers and the neutral
initialization.
0 0.02 0.04 0.06 0.08 0.1
0.2
0.4
0.6
0.8
1
relative weight of likelihood energy
proportion
of
well
detected
objects
neutral
seeded
0 0.02 0.04 0.06 0.08 0.1
0.02
0.04
0.06
0.08
0.1
relative weight of likelihood energy
segmentation
error
neutral
seeded
Fig. 3. Evaluation of the results on synthetic images using the two
initializations, as a function of the relative weight of the likelihood energy.
Left: proportion of correctly detected objects; right: segmentation error of
correctly detected objects.
Fig. 4. Results on synthetic images using three or four layers and the seeded
initialization.
The configuration used for the experiments was an Intel
Core i7 CPU 2.93 GHz, with 6GB RAM, running on Windows
7 64-bit operating system. The optimization on an image of
size 400×400 using three phase field layers takes ∼ 70 seconds
if stopped after 500 iterations (with a mean stopping error of
δE/δϕ(k)
< 10−
7). The running time for each iteration is
linear in the number of pixels and number of layers.
C. Comparison with other methods
We compared the new method to CellProfiler [15], which
is a threshold-based method, and a Hough transform method
that uses image gradient. CellProfiler is used worldwide for
cell segmentation, but cannot handle overlapping objects. The
Hough transform method does allow overlapping objects, but
can only represent perfect circles. Fig. 5 shows comparative
results on fluorescence microscopy images containing many
touching and overlapping prostate cell nuclei. Both methods
are faster than the proposed method, but give visibly lower
quality results, both in terms of correctly detected objects and
segmentation error.
V. CONCLUSION
The contribution of this work is a new model for the
segmentation of touching and overlapping near-circular objects
in images, and in particular a new image model that takes
into account the additive nature of the image intensity corre-
sponding to overlapping objects in many imaging modalities,
particularly transmission- and emission-based microscopy. The
new model enables both the separate detection of multi-
ple overlapping objects and their accurate segmentation, and
proves successful in performing this task on both synthetic
and fluorescence microscopy images. The main open question
is efficient estimation of those prior parameters that are not
fixed by the stability analysis.
ACKNOWLEDGMENT
This research was partially supported by the European
Union and the State of Hungary, co-financed by the European
Social Fund through project TAMOP-4.2.2.A-11/1/KONV-
2012-0073 (Telemedicine-focused research activities in the
fields of Mathematics, Informatics and Medical sciences).
REFERENCES
[1] M. E. Leventon, W. E. L. Grimson, and O. Faugeras, “Statistical
shape influence in geodesic active contours,” in Proc. International
Conference on Computer Vision and Pattern Recognition (CVPR),
Hilton Head Island, South Carolina, USA, Jun. 2000.
[2] Y. Chen, H. D. Tagare, S. Thiruvenkadam, F. Huang, D. Wilson, K. S.
Gopinath, R. W. Briggs, and E. A. Geiser, “Using prior shapes in
geometric active contours in a variational framework,” International
Journal of Computer Vision, vol. 50, no. 3, pp. 315–328, 2002.
[3] A. Foulonneau, P. Charbonnier, and F. Heitz, “Geometric shape priors
for Region-Based active contours,” in Proc. International Conference
on Image Processing (ICIP), Barcelona, Spain, Oct. 2003.
[4] R. Huang, V. Pavlovic, and D. N. Metaxas, “A graphical model
framework for coupling MRFs and deformable models,” in Proc.
International Conference on Computer Vision and Pattern Recognition
(CVPR), vol. 2, Jun. 2004.
[5] N. Paragios and M. Rousson, “Shape priors for level set representa-
tions,” in Proc. European Conference on Computer Vision (ECCV),
Copenhagen, Denmark, May 2002.

5. Fig. 5. Results of applying the new model and two other methods to fluorescence microscopy images of prostate cell nuclei. From left to right: original image,
CellProfiler, Hough, proposed method.
[6] D. Cremers, F. Tischhauser, J. Weickert, and C. Schnorr, “Diffusion
snakes: Introducing statistical shape knowledge into the Mumford-Shah
functional,” International Journal of Computer Vision, vol. 50, no. 3,
pp. 295–313, 2002.
[7] S. H. Joshi and A. Srivastava, “Intrinsic bayesian active contours for
extraction of object boundaries in images,” International Journal of
Computer Vision, vol. 81, no. 3, pp. 331–355, 2009.
[8] A. Srivastava and I. H. Jermyn, “Looking for shapes in two-dimensional
cluttered point clouds,” IEEE Transactions on Pattern Analysis and
Machine Intelligence, vol. 31, no. 9, pp. 1616–1629, 2009.
[9] M. Rochery, I. H. Jermyn, and J. Zerubia, “Higher order active
contours,” International Journal of Computer Vision, vol. 69, no. 1,
pp. 27–42, 2006.
[10] ——, “Phase field models and higher-order active contours,” in Proc.
International Conference on Computer Vision (ICCV), Beijing, China,
2005.
[11] P. Horvath and I. H. Jermyn, “A ‘gas of circles’ phase field model
and its application to tree crown extraction,” in Proc. European Signal
Processing Conference (EUSIPCO), M. Domanski, R. Stasinski, and
M. Bartkowiak, Eds., Poznan, Poland, Sep. 2007.
[12] C. Molnar, Z. Kato, and I. H. Jermyn, “A multi-layer phase field model
for extracting multiple near-circular objects,” in Proc. International
Conference on Pattern Recognition (ICPR), Tsukuba Science City,
Japan, Nov. 2012.
[13] J. Nemeth, Z. Kato, and I. H. Jermyn, “A multi-layer ‘gas of circles’
markov random field model for the extraction of overlapping near-
circular objects,” in Proc. Advanced Concepts for Intelligent Vision
Systems, J. Blanc-Talon, W. Philips, D. Popescu, P. Scheunders, and
R. Kleihorst, Eds., Ghent, Belgium, Aug. 2011.
[14] P. Horvath, I. H. Jermyn, Z. Kato, and J. Zerubia, “A higher-order active
contour model of a ‘gas of circles’ and its application to tree crown
extraction,” Pattern Recognition, vol. 42, no. 5, pp. 699–709, 2009.
[15] A. E. Carpenter, T. R. Jones, M. R. Lamprecht, C. Clarke, I. H. Kang,
O. Friman, D. A. Guertin, J. H. Chang, R. A. Lindquist, J. Moffat et al.,
“Cellprofiler: image analysis software for identifying and quantifying
cell phenotypes,” Genome biology, vol. 7, no. 10, p. R100, 2006.