Automatic ground truth generation for image sequences

1. INTERNATIONAL JOURNAL OF ELECTRONICS AND
International Journal of Electronics and Communication Engineering & Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
COMMUNICATION
ENGINEERING TECHNOLOGY (IJECET)
ISSN 0976 – 6464(Print)
ISSN 0976 – 6472(Online)
Volume 5, Issue 9, September (2014), pp. 24-34
© IAEME: http://www.iaeme.com/IJECET.asp
Journal Impact Factor (2014): 7.2836 (Calculated by GISI)
www.jifactor.com
24
IJECET
© I A E M E
AUTOMATIC GROUND TRUTH GENERATION FOR IMAGE SEQUENCES
Shivank Bhagotra
Electronics Communication Engineering, BKBIET, Pilani, Rajasthan, India
ABSTRACT
This paper presents an approach for generating efficient Ground Truth for image sequences
related to Traffic Signals and Direction Signs by making suitable annotations around the area of
interest and saving corresponding information in the form of XML files. This feature can be readily
used in the cars to detect the speed limit signals and automatically optimize their speed in case of
driverless cars. The Graphic User Interface (GUI) has been designed in GUIDE Matlab and the
Image Database has been provided by the MIPS-MIAM laboratory, Ensisa, Mulhouse, France.
The entire project deals with making a driverless car wherein the real time images as
perceived by the camera placed in front of the car are processedand annotations are formed around
Direction signs and Traffic Speed Limit Signals. The specifications such as Content,
Representativeness, Occlusion, Image Identifier, and Uncertainty get stored in the form of XML
files. This stored data is processed and subsequent changescan be made in the speed, direction of the
car.
Keywords: Annotation, Content, GUIDE, GUI, Ground Truth, XML Files.
1. INTRODUCTION
The annotation tool has been adapted from the software developed at University of Bonn,
Germany, as a general annotation tool. The objective of the project deals with the transformation of
the Tool from the ‘Annotation Tool’ to the ‘Traffic Annotation Tool’ which requires deep
understanding of the Annotation tool environment and the ways callbacks are behaving to make
changes in the tool. Objectives to be achieved are:
i. Addition of static textbox Content and Popup menu associated with it.
ii. Writing the Callback for the popup menu ‘content’ to make it work like other popup menus.
iii. Changes in the popup menu ‘Source’.
iv. Changes in the popup menu ‘Class Name’.

2. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
25
2. ANNOTATION TOOL
Image Annotations generally refer to highlighting features, details, or points of
interest within an image. Annotation Tool is a Matlab based image Annotation Tool developed at
University of Bonn for the Etrims image database and this particular Annotation Tool has been
modified to make it suitable to annotate images from the database (Traffic Signs) related to our
project.
We are working upon the Annotation tool version 1.7 as the latest version of the Annotation
tool which is 2.4 had some bugs indeed, the annotations are not being saved properly in the XML
files and the generated files had some errors as all class names were being shown up in the file.
The Annotation Tool originally employed LabelMe Matlab toolbox developed at the Massachusetts
Institute of Technology (MIT) for the LabelMe image database.Here is an example of Annotation
Tool interface where an image from the Etrims Image database is being annotated:
Fig 1: Screenshot of the Annotation Tool
2.1. SETUP
i. Download the Matlab toolbox LabelMe and add it to the Matlab search path, e.g.
• C:LabelMeToolbox
ii. Download the annotation tool and add it to the Matlab search path, e.g.
• C:AnnotationTool
iii. Create home folders for your image data and annotations e.g.
• C:AnnotationToolhome-images
• C:AnnotationToolhome-annotations
iv. An annotation tool comes along with three text files two of which should contain home folder
paths. Write the image data home directory and the annotations home directory in the
corresponding text files:
• C: AnnotationToolhome-images.txt
• C:AnnotationToolhome-annotations.txt
v. Now we create directories with your image datasets, e.g.:
• C:AnnotationToolhome-imagesimage-dataset1

3. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
• C:AnnotationToolhome-imagesimage-dataset2
26
vi. And copy your image data in this e.g.
• C:AnnotationToolhome-imagesimage-dataset1my-image.jpg
It is obligatory to create the image dataset subdirectory as this folder name will be stored with the
annotation.In Matlab, we can now run AnnotationTool.m after the first image is labelled, the
Annotation is stored in an XML file of the same name as the labelled image. The XML file with
annotation is saved in the annotations home directory of the same name as the image dataset
directory of the labelled image, e.g.
• C:AnnotationToolhome-annotationsimage-dataset1/my-image.xml
3. IMAGE ANNOTATION
3.1. LABELLING IMAGE DATA
Looking at the Data panel in the upper right corner of the (Fig 2).Opening an image dataset
folder by pressing the button Open Image Folder.Then selecting an image in the listbox underneath.
We may now specify the Image sourcein the Current image panel. In case the popup menu does not
offer a relevant option, we may specify an alternative source of the image by choosing the option
‘other’.
After pressing the annotate button on the New Annotation panel the tool will switch to
annotation mode where only image labelling using the mouse is allowed. Annotate by pressing the
left mouse button and clicking in the image area. Pressing the right mouse button will finish the
object labelling and will close the polygon (Fig. 2). If we press the Annotate button now, the
previous label will be erased and we may label the object again. Pressing the right mouse button
without having labelled anything will just cancel the annotation mode. The zoomfeature will ease the
annotation of smaller objects.
Fig 2: Annotating an image object from our database
3.2. OBJECT ANNOTATION
Having labelled an image we may now specify its class, degree of its occlusion,
representativeness and Content (modification) (Fig 3). It is possible to specify an alternative object
class name by choosing the option ‘other’…in the Object class pop up menu. The Annotation tool

4. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
comes along with three text files, one of which, called object-classes.txt contains a list of object class
names. We can simply extend or change the classes by simply editing the text file.
The next value to be specified is the annotation uncertainty: it is an estimate, in pixels, of
how precise the labelling is. A set of green and red concentric circles is drawn in the middle of the
new object label. A diameter of the red circle in pixels corresponds to the value chosen in the
Annotation Uncertainty popup menu.
And the last value to specify is the Content which has been added by me in the panel, where
the user can enter the value of the speed sign when it is being annotated and the same will appear in
the information about the annotation in the XML file generated. Having specified all the required
information we may add the annotation to the Current annotationslistbox by pressing the Add Object
button (Fig 4). A login that should specify the labeller and the place of annotation is required when
adding the first object. This information is added to the annotation of every labelled object.
(a) (b) (c)
Fig 3: (a) Specifying the Image Source, (B) Class of the Image Object, (C) Degree of Occlusion and
Content
After adding the new annotation to the pool of the Current Annotationwe have the possibility
of deleting it by pressing the Delete Last button.It is only the last added annotation that the labeller is
allowed to erase. Pressing the Save Annotationbutton will save all current annotations to a file.
(a) (b)
Fig 4: (a) Adding an annotated image object to current annotations, (b) A login specifying the
labeller and the place of annotation is required when adding the first object
27

5. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
4. ANNOTATION TOOL TRANSFORMATION
• When I added an additional Content Popup menu, as it would be useful for the tool to
have to annotate images from our database which consisted of Speed Limit Signals,
Guide was not generating a corresponding Callback in the Annotation Tool for the
Content Popup menu even after being saved with the same filename. Then after searching
over the internet, the problem was solved as there was a change to be made in the settings
of the Guide.
Fig 5: Changes to be made for Callback
After adding the desired uicontrol to the Guide environment, we need to go to toolsGUI
options (Fig 5). After this, we need to select the radio button for GenerateFIG-file and M-file
(Fig 6).
Fig 6: Selecting Radio button for both files
After the following changes are made, corresponding callbacks for the added uicontrols will
be generated in the .m file of the tool. Now changes can be made in the Callback related to the
uicontrol to make them function accordingly.
• The abling and disabling of the Content popup menu should be the same like the other popup
menus from the tool, meaning it should get activated when annotation is over as the right
28

6. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
mouse button is clicked. This was initially not working as there were errors in the Matlab
command window when the code was being run. It was due to static number of properties to
be activated or deactivated fixed to 20 which was prohibiting any addition done to the code
(Fig 7).
Fig 7
Then we set a global variable defines as ‘Handles.NUMCONTROLS’ and set its value to 21
and then we were able to make popup menu ‘Content’ enabled like other popup menus (Fig 8).
Fig 8
Therefore, in order to add any uicontrol to the Annotation Tool code we need to increase the
value for the ‘Handles.NUMCONTROLS’ which is a global variable we have defined at the starting
of the code and make corresponding changes in the code.
Following the above changes, corresponding changes have been made in the subsequent code
and Callback for the content to make it work like the other popup menus and get the value of
‘Content’ in the XML file which is generated after the annotation is saved (Fig 9).
Fig 9
29

7. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
30
5. TRAFFIC ANNOTATION TOOL
Several changes have been made in the Annotation Tool to make it ‘Traffic Annotation Tool’.
• Changes in the popup menu ‘SOURCE’.
Change in the source of the image can be made by opening the Property Inspector of
the popup menu ‘Source’and modifying the String section.
(a) (b)
Fig 10: (a) Annotation Tool (b) Traffic Annotation Tool
As we can see modification has been done in the popup menu ‘Source’ according to our
project (Fig 10).
• Changes in the popup menu ‘Class name’.
The Annotation Tool comes along with three text files, one of which, called ‘object-classes.
txt’ contains a list of object class names. We can make changes to the set of classes by
simply editing the text file.
Fig 11: Command given in the code to read the text file and load it into the popup menu
As we can see modification has been done in the popup menu ‘Class Name’ according to our
project (Fig 12).
(a) (b)
Fig 12: (a) Annotation Tool (b) Traffic Annotation Tool

8. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
• Adding an additional Static Textbox ‘Content’ and popup menu associated with it.
Static textbox ‘Content’ and corresponding popup menu can be added to the interface
when editable guide window for Annotation Tool is opened by the guide(AnnotationTool)
command in the command window (Fig 13).
(a) (b)
Fig 13: (a) Annotation Tool (b) Traffic Annotation Tool
• Writing the Callback for the popup menu ‘Content’ to make it work like other popup
31
menus.
Several additions to the code have to be done for adding the Content popup menu and
making it work like the other popup menus. This section has been explained in the later part of
the report (See Fig 9).
Therefore, all the changes have been made successfully and the Annotation Tool is
working correctly. The XML files generated for the annotations have the additional tag called
‘Content’ which specifies the value of the speed limit sign in addition to the Class name,
Occlusion, Representativeness and Annotation Uncertainty.
6. RESULTS
By the end of the project we have following Ground truth which is generated for an
annotation when it is saved and can be retrieved from the XML file generated.
• Image identifier
We can determine that from the XML file where it is stored as the filename.
• Position of the top left corner of the elements
We can determine that by having a look at the XML file generated as it contains the coordinates
the points plotted.
• Resolution(in pixels)
Image size (Pixel Dimension) as is in the axis where the image is loaded:
Width: 640 pixels
Height: 480 pixels
Therefore, the resolution of the annotation made can be determined by the axis and the
coordinates we have in the XML file.
• Type of Traffic Sign
We can determine the type of traffic sign we are annotating as it is loaded in the Class names
Popup Menu and that similar will be reflected in the XML file whether it is a speed limit sign,
danger sign, no parking sign, etc.

9. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
32
• Annotation Specifications
We have been able to determine Occlusion, Representativeness, Annotation Uncertainty and Content
for a particular traffic signal.
• Creation of an XML File.
Command used = ‘writexml’.It transforms a struct or struct array into an xml file.
Filename = ‘name of xml file’. V= struct variable containing fields of type char or numeric.
Writexml (filename, xml) will generate an .xml file.
All the results obtained can be seen in this example of an annotated image (3257.jpg) from
our database.
Fig 14: Image annotation by the Traffic Annotation Tool
All the ground truth generated for a particular image by the Traffic Annotation Tool can be
seen over the screen once the annotation is saved (Fig 14). The corresponding XML generated is for
the above image and all the results obtained are well depicted in the XML file and are highlighted.
annotation
filename3257.jpg/filename
folderimage-dataset2/folder
sourceImageENSISA_UHA/sourceImage
rectified0/rectified
viewTypeterrestrial/viewType
scalen/a/scale
imageWidthHeight640 480/imageWidthHeight
transformationMatrixn/a/transformationMatrix
object
namespeed limit signals/name
objectID73552514801738/objectID
occlusion60/occlusion
representativeness40/representativeness

10. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976 –
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue 9, September (2014), pp. 24-34 © IAEME
33
uncertaintyn/a/uncertainty
Content50/Content
deleted0/deleted
verified0/verified
date18-Oct-2013/date
sourceAnnotationShivank@UHA/sourceAnnotation
polygon
pt
x577.1029/x# position of the top left corner
y223.9076/y
/pt
pt
x577.8747/x
y264.8098/y
/pt
pt
x537.7443/x
y264.8098/y
/pt
pt
x537.7443/x
y224.6794/y
/pt
pt
x578.6464/x
y223.1359/y
/pt
/polygon
objectPartsn/a/objectParts
/object
/annotation
7. CONCLUSION
In this paper, Transformation of the Annotation Tool to the ‘Traffic Annotation Tool’ has
been discussed. The steps to make the transition have been discussed in detail citing all problems
encountered while transforming the code. The corresponding XML file generated contains all the
desired ground truth as our objective in the beginning of the project. Further changes and adoptions
can be made in the code for the Annotation Tool to redesign it according to need of the project. This
work can be further enhanced by designing an interface to incorporate the ground truth generated to
efficiently modify the speed of the car.
8. ACKNOWLEDGEMENT
It gives me immense pleasure to express my deepest sense of gratitude and sincere thanks to
my highly respected and esteemed Mentors Prof. Jean-Philippe Lauffenburger and Jeremie Daniel
(Ecole Nationale Supérieure d’ingénieurs Sud Alsace (ENSISA), MIPS-MIAM Laboratory) for their
valuable guidance, encouragement and help for completing this work.

11. International Journal of Electronics and Communication Engineering Technology (IJECET), ISSN 0976
6464(Print), ISSN 0976 – 6472(Online), Volume 5, Issue
REFERENCES
[1] Annotation Tool Documentation:
9, September (2014), pp. 24-34 © IAEME
[http://www.ipb.uni-bonn.de/html_pages_software/annotation
[2] @techreport{annotation-tool
Author = Kor{v c}, F. and Schneider, D.,
Title = Annotation Tool,
Number = TR-IGG-P-2007-
Month = June,
Year = 2007,
Institute = Dept. of Photogrammetry, University of Bonn,
URL = http://www.ipb.unibonn.de/html_pages_software/annotation
P-2007-01.pdf
[3] Bryan C. Russell, Antonio Torralba, Kevin P. Murphy,
‘LabelMe: a database and webbased tool for Image Annotation’
Computer Vision, 2008.
[4] For further information on Etrims image database:
[http://www.ipb.uni-bonn.de/projects/etrims_db/]
[5] Matlab tutorials can be viewed here:
[http://www.mathworks.com/products/matlab/videos.html]
[6] Detailed tutorial on Guide for beginners can be
[http://www.mathworks.com/videos/creating
[7] Abbass.A.Khorsheed, Hussein Chible, Giorgio Giacinto
Korc-TR-IGG-T.
nd De-Noising for Image Annotation
Electronics and Communication Engineering Technology (IJECET), Volume
2014, pp. 113 - 118, ISSN Print: 0976
BIOGRAPHY
–
MIT, USA
Journal of
Journal of
5, Issue 1,
Shivank Bhagotra
his Bachelors in Electronics and Communication Engineering from B.K. Birla
Institute of Engineering and Technology, Pilani. He has been a Research Intern
at MIPS-MIAM Laboratories, ENSISA, Mulhous
associated with IEEE for the past 3 years and currently acting as the IEEE
Mentor BKBIET Student Branch.
34
annotation-tool/
tool-2007,
-01,
annotation-tool/publ/Korc
William T. freeman
for International J
om/viewed on Math Works Website:
creating-a-gui-with-guide-68979.html]
and Ammar J. Fattah, “Automatic
or using Latent Semantic Analysis”, International
0976- 6464, ISSN Online: 0976 –6472.
was born in Jammu, India in 1993. He is currently pursuing
Mulhouse, e, France. He has been